feature: seed and validate testing harness#30
Merged
Conversation
Review Summary by QodoAdd seed-and-validate testing harness with end-to-end check validation
WalkthroughsDescription• Add comprehensive seed-and-validate Python orchestrator for end-to-end testing - Creates throwaway test database, seeds all pgFirstAid checks, validates firing - Patches thresholds for test-friendly sizes, runs live session threads - Supports both standard and managed database modes • Implement full structural seed SQL with 15+ check patterns - Tables, indexes, statistics, roles, locks, replication slots - Idempotent schema-based seeding with proper cleanup • Add pytest test suite validating seed logic and check classification - Tests connection params, threshold patching, report building, PSS state detection - Validates expected check sets match pgFirstAid.sql definitions • Integrate harness into CI workflows for managed database validation - Runs after integration tests in GitHub Actions matrix and managed-db workflows • Add design specification document and Python project configuration Diagramflowchart LR
A["seed_and_validate.py<br/>Orchestrator"] --> B["Create Test DB"]
B --> C["Install pgFirstAid<br/>with patched thresholds"]
C --> D["Run Static Seed<br/>01_seed_static_checks.sql"]
D --> E["Run PSS Seed<br/>02_seed_pg_stat_statements.sql"]
E --> F["Start Live Session<br/>Threads"]
F --> G["Wait 6 minutes<br/>for thresholds"]
G --> H["Run Validation<br/>pg_firstAid()"]
H --> I["Report Results<br/>PASS/FAIL/SKIP"]
I --> J["Drop Test DB<br/>Cleanup"]
File Changes1. testing/seed_and_validate.py
|
Code Review by Qodo
1. Managed view PSS crash
|
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
Design doc for the Python orchestration script that seeds all pgFirstAid health checks and validates each one fires in a throwaway test database.
…case pgfirstaid_pg_stat_statements_checks() now wraps all RETURN QUERY statements in a begin/exception block to gracefully handle ObjectNotInPrerequisiteState when the extension is installed but not in shared_preload_libraries. seed_and_validate.py gains is_pss_queryable() to detect this condition, tracks pss_extension_installed separately from pss_seeded, and re-verifies PSS accessibility before validation to guard against connection state changes during the 6-minute session wait. run_validation() skips the "Extension Missing" check when the extension is installed but not queryable.
randoneering
force-pushed
the
feature/load_testing
branch
from
bf91118 to
ec5e7fc
Compare
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Installs view_pgFirstAid_managed.sql and queries v_pgfirstAid instead of pg_firstaid() when --managed is passed. Allows validating the managed service path (no superuser-only queries) with the same seed data.
randoneering
force-pushed
the
feature/load_testing
branch
from
ec5e7fc to
56856c8
Compare
Owner
Author
|
Will create a new branch to fix runner. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Summary
The following PR includes a new testing harness that will seed a target database and validate health checks. For each new health check, we are asking that contributors add one test via pgTAP and through the seed and validate scripts. If you do not know how, that is totally fine! We can help you along the way.
Type of Change
Related Issues
Testing
PostgreSQL Version Compatibility
Has this code been tested against the following PostgreSQL versions?
Testing notes:
Managed Database Platforms
Has this code been deployed and tested on the following platforms?
Platform-specific notes:
Additional Notes
Greptile Summary
This PR introduces a seed-and-validate testing harness (
testing/seed_and_validate.py) that spins up a throwaway PostgreSQL database, seeds data and live sessions that trigger every pgFirstAid health check, then validates all expected check names appear in output. It also adds structured unit tests, per-check seed SQL scripts, and CI wiring for PG15–18 self-hosted and managed-DB targets.integration-pg-matrix.yml'spathstrigger omitstesting/seed_and_validate.pyandtesting/healthcheck_seed/**; future changes to the core harness won't trigger the integration workflow.04_session_blocked.sqlis empty despite being listed in the README as the "Session B" blocked-session script needed for manual testing.Confidence Score: 4/5
Safe to merge after fixing the CI path filter; remaining findings are cleanup/documentation gaps.
One P1 issue: the
pathstrigger inintegration-pg-matrix.ymlomits the new seed harness files, meaning future PRs touching those files won't trigger CI validation. The remaining findings (deadstopparameter, leaked role, empty session B script) are P2 quality/documentation issues that don't affect runtime correctness..github/workflows/integration-pg-matrix.yml(path filter),testing/healthcheck_seed/04_session_blocked.sql(empty file)Important Files Changed
stopparameter in_long_query_threadand missing role cleanup infinallyblock are minor issues.pathstrigger is missingtesting/seed_and_validate.pyandtesting/healthcheck_seed/**, so future changes to those files won't re-run CI.\gexecto generate workload patterns covering all PSS-dependent checks; guarded bypsqlavailability check in the Python harness.pg_firstAid()output by severity and check name.Sequence Diagram
sequenceDiagram participant M as main() participant A as admin_conn (postgres DB) participant T as test_conn (pgfirstaid_test DB) participant Th as Session Threads M->>A: connect_admin() M->>A: create_test_db() — DROP/CREATE pgfirstaid_test M->>T: connect_test() M->>T: install_function() — patch thresholds & install SQL M->>T: run_sql_file(01_seed_static_checks.sql) M->>T: seed_low_usage_index_scans() M->>T: run_psql_file(02_seed_pg_stat_statements.sql) M->>T: try_create_replication_slot() M->>Th: start blocker thread (holds UPDATE lock) M->>Th: start blocked thread (waits on lock) M->>Th: start idle-in-transaction thread M->>Th: start long-query thread (pg_sleep 700s) M->>A: _wait_for_active_query() — poll pg_stat_activity Note over M: sleep 6 minutes for 5-min session thresholds M->>T: run_validation() — SELECT from pg_firstAid() T-->>M: fired check names M->>M: build_report(fired, expected, skipped) Note over M: finally block M->>Th: stop_event.set() + sleep 2s M->>T: drop_replication_slot() M->>A: drop_test_db() — pg_terminate_backend + DROP DATABASEReviews (1): Last reviewed commit: "added seed/validation testing into workf..." | Re-trigger Greptile