[CRE] Mixed-env nightly full-matrix sweep + chain-specific variants - #23312
Open
prashantkumar1982 wants to merge 3 commits into
Open
[CRE] Mixed-env nightly full-matrix sweep + chain-specific variants#23312prashantkumar1982 wants to merge 3 commits into
prashantkumar1982 wants to merge 3 commits into
Conversation
product-security-plaid-production
Bot
requested review from
kalverra and
pavel-raykov
August 3, 2026 18:43
prashantkumar1982
marked this pull request as draft
August 3, 2026 18:43
Contributor
|
✅ No conflicts with other open PRs targeting |
Contributor
|
I see you updated files related to
|
prashantkumar1982
force-pushed
the
cre/mixed-env-nightly
branch
from
August 3, 2026 18:57
eed0791 to
45cc03a
Compare
prashantkumar1982
force-pushed
the
cre/mixed-env-nightly
branch
3 times, most recently
from
August 5, 2026 15:56
b2d5caa to
710cb27
Compare
Nightly full sweep that runs the entire CRE matrix under mixed-env, built on top of the dedicated cre-mixed-env-tests.yaml workflow. - cre-mixed-env-tests.yaml: add a full_matrix input. Off (per-PR default) it runs the base EVM set; on (nightly) it also runs the special-topology tests under their mixed-env variant. Matrix entries now carry per-test topology + config, and the render step / ctf-configs / TOPOLOGY_NAME use them. - cre-mixed-env-nightly.yaml: scheduled workflow calling cre-mixed-env-tests.yaml with full_matrix: true, comparing today's develop nightly against a previous one. - Special-topology mixed-env templates (2-2 split, verified via `topology show`): solana, aptos, stellar, sharded (preserves shard0 custom_ports), cache-test (preserves ModuleCache override), multi-gateway (preserves org override + both gateway DONs). - render-mixed-env.sh generalized to render any mixed-env-*-don.toml.tmpl. - gitignore covers rendered configs/mixed-env-*.toml; docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A scheduled sweep has no PR to block, so a failure was only visible via GitHub's default single-recipient email + the Actions tab. Add a notify-on-failure job (if: failure(), needs the sweep) that posts a rotating-light alert to the #cre-platform-operations Slack channel with a link to the run, mirroring the existing CRE Slack-notify pattern (slack-github-action + QA_SLACK_API_KEY). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-nightly
branch
from
August 5, 2026 16:42
710cb27 to
d90bf11
Compare
prashantkumar1982
marked this pull request as ready for review
August 5, 2026 16:57
Tofel
reviewed
Aug 6, 2026
| on: | ||
| schedule: | ||
| # After the nightly image build (docker-build.yml runs at 03:00 UTC). | ||
| - cron: "0 5 * * *" |
Contributor
There was a problem hiding this comment.
what do you think about triggering it with workflow_call from this workflow? that's what we do for all other nightly tests and it is triggered by Docker Build, which build the nightly image
Contributor
Author
There was a problem hiding this comment.
Good idea — that's a better fit than my cron. I checked: docker-build.yml only calls post-docker-build.yml on its schedule/workflow_dispatch runs (the 03:00 UTC nightly), after the core image is built — so wiring in there gives us the event-driven trigger for free and drops my fragile "runs sometime after 03:00" cron guess and its race with the image build.
One wrinkle specific to mixed-env: unlike the other nightly suites, it needs a baseline (a previous nightly) in addition to the just-built image, and post-docker-build only passes chainlink_core_image_tag + chainlink_version. So the adaptation is:
- switch
cre-mixed-env-nightly.yamlfromscheduletoworkflow_callwith those two inputs (keepingworkflow_dispatchfor manual runs), - use
chainlink_core_image_tagas the "today" side and resolve the baseline (a previous nightly) inside the workflow, and - add a
call-cre-mixed-env-nightlyjob topost-docker-build.yml.
I'll confirm the nightly tag scheme lines up for picking the baseline while I'm at it. Will push shortly — thanks!
Addresses review feedback (Tofel, r3729052542): trigger the nightly sweep the same way as the other nightly suites — via workflow_call from post-docker-build.yml, which docker-build.yml invokes on its scheduled/dispatch run right after the nightly core image is published. This drops the fragile "cron at 05:00, hoping the 03:00 image build finished" schedule and its race. - cre-mixed-env-nightly.yaml: replace the `schedule` trigger with `workflow_call` (chainlink_core_image_tag + chainlink_version), keep workflow_dispatch for manual runs. The just-built nightly is the "today" side; only the baseline (a previous nightly) is resolved internally (set-baseline; defaults to yesterday). - post-docker-build.yml: add a call-cre-mixed-env-nightly job passing the image tag and version, alongside the existing nightly test fan-out. - docs: note the post-docker-build trigger. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[CRE] Mixed-env nightly full-matrix sweep + chain-specific variants
Stacked on #23290. Extends mixed-env non-determinism coverage beyond the per-PR OCR3/DON2DON subset.
What
cre-mixed-env-nightly.yaml): runs the entire CRE matrix under mixed-env (mixed_env_all: true), comparing today's develop nightly image against a previous develop nightly — catching non-determinism introduced into develop between nightlies, across the whole suite. No image build (reuses cached nightlies).mixed_env_alltoggle incre-system-tests.yaml: appendsmixed-envto the default (EVM) topology list, and the chain-specific mixed-env variants to the Solana/Aptos/Stellar tests. Per-PR behavior is unchanged (toggle defaults tofalse).mixed-env-solana-don,mixed-env-aptos-don,mixed-env-stellar-don— a 2-2 PR/develop split of each chain topology.render-mixed-env.shnow renders anymixed-env-*-don.toml.tmpl; the workflow render step and image gating match anymixed-env*topology (startsWith).configs/mixed-env-*.toml; docs updated.Verified locally
topology show(Solana/Stellar multi-don 4+4+1, Aptos single-don 4+1).mixed_env_all: truegives Aptos→mixed-env-aptos, Solana→mixed-env-solana, base tests→mixed-env(default topology preserved alongside).Known follow-ups
topology generate --check-style guard would help).chainlink:nightly-<date>image exists; adjust if the nightly build slips.baseline_dateis overridable viaworkflow_dispatch.🤖 Generated with Claude Code