ci: expand cassert nightly test coverage + smart failure notify#8646
Merged
Conversation
Add a nightly CI workflow that builds an --enable-cassert PostgreSQL on-the-fly via the existing pgenv machinery (dropping -DUSE_VALGRIND), rebuilds citus against it, and runs the full suites (regress, isolation, columnar, failure, pg-upgrade, citus-upgrade) to surface PG-core and Citus Assert() failures the PGDG-image PR pipeline cannot catch. - .github/actions/setup_cassert_pg: composite action that builds the cassert PG (cached on ~/.pgenv keyed by version + config hash), transforms the devcontainer pgenv config to drop USE_VALGRIND, and builds/installs citus against the resulting pg_config. Exports LD_LIBRARY_PATH to the pgenv libdir so psycopg finds libpq. - .github/workflows/nightly_cassert.yml: cron 0 3 * * * UTC plus workflow_dispatch. A single params job is the version SSOT seam (future the-process/versions.yml derivation point) feeding all matrices. Pinned to main majors 16.13/17.9/18.3. citus-upgrade uses the local tarball path (old v14.0.1 -> HEAD; CI="" to force the local branch). Opens/appends a dated nightly-cassert issue on failure. First runs are EXPECTED red across all majors as pre-existing asserts surface; that is the intended signal. pg19-support adapts this later to add the 19beta1 row and the 18->19 upgrade pair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove Linux-irrelevant CRLF strip, ineffective ulimit -c, and a redundant job-level permissions block already granted at workflow level. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address review: fail the build loudly if the pgenv config inherited from the devcontainer ever loses --enable-cassert, and add a back-reference comment in default.conf so editors know the nightly depends on it. Keeps the devcontainer config as the single source of truth while turning a silent asserts-off run into a hard failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
pgenv runs 'make world'/'install-world' for modern PG majors, which builds the PG documentation. That doc build invokes xmllint and xsltproc against the DocBook DTD/XSL; PG17+ hard-requires postgres-full.xml so setup fails at ~2 min without them (PG16's older doc Makefile no-ops gracefully, which is why only 17/18 broke in run 28804321920). Add docbook-xml, docbook-xsl, libxml2-utils and xsltproc to the apt list, mirroring .devcontainer/Dockerfile which builds this identical pgenv config successfully. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
pgenv runs 'make world'/'install-world' for modern PG majors, which builds the PG documentation. That doc build invokes xmllint and xsltproc against the DocBook DTD/XSL; PG17+ hard-requires postgres-full.xml so setup fails at ~2 min without them (PG16's older doc Makefile no-ops gracefully, which is why only 17/18 broke in run 28804321920). Add docbook-xml, docbook-xsl, libxml2-utils and xsltproc to the apt list, mirroring .devcontainer/Dockerfile which builds this identical pgenv config successfully. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…-cassert-nightly-workflow
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8646 +/- ##
==========================================
- Coverage 88.73% 88.73% -0.01%
==========================================
Files 288 288
Lines 64428 64428
Branches 8113 8114 +1
==========================================
- Hits 57170 57168 -2
Misses 4915 4915
- Partials 2343 2345 +2 🚀 New features to boost your workflow:
|
Fold da_DK locale setup from #8645 (columnar collation tests). Add generator group (check-query-generator, check-pytest), check-add-backup-node, three enterprise-isolation-logicalrep shards, and a sharded arbitrary-configs job. Notify job now dedups by failed-job-name signature: append to an existing open issue with the same failing set, otherwise open a new issue highlighting untracked failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ihalatci
changed the title
ihalatci
marked this pull request as ready for review
July 7, 2026 12:54
onurctirtir
approved these changes
Jul 14, 2026
ihalatci
enabled auto-merge (squash)
July 14, 2026 15:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the nightly full-cassert workflow (#8640). Expands test coverage and reworks failure notification.
.github/-only.Changes
setup_cassert_pgactionlocalesand runlocale-gen da_DK.UTF-8(columnar collation tests require theda_DKlocale). Folded from Fix citus_finish_citus_upgrade to always update last_upgrade_version #8645.nightly_cassert.yml— test coveragegeneratorgroup:check-query-generator+check-pytest(highest assert-fuzz value).regressgroup +=check-add-backup-node.isolationgroup +=check-enterprise-isolation-logicalrep-1/-2/-3.arbitrary-configsjob: config-fuzz matrix sharded 6 ways per major, mirroringbuild_and_test.yml's splitter (usespipenv runon the bare runner instead ofgosu circleci).nightly_cassert.yml— failure notificationnotifynow dedups by a signature of the sorted failed-job-name set (embedded as an HTML marker in the issue body).nightly-cassertissue → append a comment.Notes
arbitrary-configsshard count (N=6) is the primary runtime tuning knob.citus_finish_citus_upgradefix) are out of scope here — only itsaction.ymllocale change was folded.Kept as a draft.