Skip to content

fix(ci): a failing seed exits 0 (#192); a tag inside describe.skip counts as live (#210) #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rubenvdlinde merged 1 commit into from
Aug 8, 2026
Merged

fix(ci): a failing seed exits 0 (#192); a tag inside describe.skip counts as live (#210) #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 32 additions & 3 deletions .github/workflows/quality-resolve-probe.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,34 @@ jobs:
fi
echo "OK — the lint fails when it should. Its clean pass above is a verdict."

seed-semantics:
name: "A failing seed fails the job"
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

# POSITIVE CONTROL, FIRST — same discipline as the probe below. The
# battery is run against the PRE-#192 step, reconstructed verbatim, and
# must reject it. A battery that passed the known-bad body would pass
# the fixed one for reasons unrelated to the fix, and its verdict on the
# real workflow would mean nothing.
- name: "Positive control — the known-bad seed step must be rejected"
run: python3 scripts/assert-seed-step-fails-loudly.py --positive-control

# THE MEASUREMENT. Every step named "Seed test data" is EXTRACTED from
# the workflow — env block and run body both — and EXECUTED under the
# same shell GitHub uses, against seeds that include a failing `&&`
# chain, a legitimate multi-stage chain, and the assignment-prefix +
# `|| true` shape launchpad actually ships.
#
# Running the shipped text rather than grepping it is what found the
# THIRD instance of the defect: the Newman job has its own "Seed test
# data" step reading `newman-seed-command`, and openbuild passes it
# `php occ app:disable openbuild && php occ app:enable openbuild`.
- name: "Exercise every 'Seed test data' step in quality.yml"
run: python3 scripts/assert-seed-step-fails-loudly.py .github/workflows/quality.yml

probe:
name: "quality.yml resolves (job count > 0)"
runs-on: ubuntu-latest
Expand Down Expand Up @@ -153,19 +181,20 @@ jobs:
guard:
name: "Shared-workflow guard"
runs-on: ubuntu-latest
needs: [static-limits, probe]
needs: [static-limits, seed-semantics, probe]
if: ${{ !cancelled() }}
steps:
- name: Assert both guards reached a verdict
env:
STATIC: ${{ needs.static-limits.result }}
SEED: ${{ needs.seed-semantics.result }}
PROBE: ${{ needs.probe.result }}
run: |
set -eu
echo "static-limits=${STATIC} probe=${PROBE}"
echo "static-limits=${STATIC} seed-semantics=${SEED} probe=${PROBE}"
# `skipped` is failed here on purpose. A guard that did not run is
# not a guard that passed.
for pair in "static-limits:${STATIC}" "probe:${PROBE}"; do
for pair in "static-limits:${STATIC}" "seed-semantics:${SEED}" "probe:${PROBE}"; do
name="${pair%%:*}"; result="${pair##*:}"
[ "${result}" = "success" ] || {
echo "::error::${name} did not succeed (result=${result}). A guard that \
Expand Down
107 changes: 96 additions & 11 deletions .github/workflows/quality.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1900,13 +1900,38 @@ jobs:
ADMIN_PASSWORD: admin
NC_ADMIN_USER: admin
NC_ADMIN_PASS: admin
# THE SEED ARRIVES AS DATA — see the long note on the Playwright
# job's seed step (#192). The comment that used to sit here said
# "Use eval to support complex commands with pipes or &&", and that
# was the assurance the defect hid behind: because the interpolation
# was UNQUOTED, `a && b` was split by THIS shell before eval saw it,
# eval ran only `a`, and bash's `set -e` exemption for `&&` lists
# made a failing `a` exit 0.
#
# This job is not hypothetical. openbuild ships
# `php occ app:disable openbuild && php occ app:enable openbuild`
# as its newman-seed-command — a failed disable meant the enable
# never ran and Newman tested whatever state was left, green.
SEED_COMMAND: ${{ inputs.newman-seed-command }}
run: |
cd server
echo "Running seed command: ${{ inputs.newman-seed-command }}"
# Run the seed command (e.g. maintenance:repair, custom script, etc.)
# Use eval to support complex commands with pipes or &&
eval ${{ inputs.newman-seed-command }}
echo "Seed command completed."
printf 'Running seed command: %s\n' "$SEED_COMMAND"
# eval is retained on purpose: fleet callers pass assignment
# prefixes (`SEED_SCOPE=register bash …`) and `||` lists. The fix is
# the QUOTING — the whole seed reaches eval as one unit, so eval's
# combined status is the step's.
eval "$SEED_COMMAND" && SEED_RC=0 || SEED_RC=$?
if [ "$SEED_RC" -ne 0 ]; then
# Diagnostics on STDERR — an `::error::` on stdout is swallowed by
# any caller that captures the step's output with `$(…)`.
{
echo "::error::Seed command failed with exit ${SEED_RC}: ${SEED_COMMAND}"
echo "Refusing to run Newman against a half-seeded instance — a full"
echo "collection tally from an unseeded server looks exactly like a real one."
} >&2
exit "$SEED_RC"
fi
echo "Seed command completed (exit 0)."

- name: Setup Node.js and Newman
uses: actions/setup-node@v4
Expand Down Expand Up @@ -2419,11 +2444,50 @@ jobs:
ADMIN_PASSWORD: admin
NC_ADMIN_USER: admin
NC_ADMIN_PASS: admin
# THE SEED ARRIVES THROUGH THE ENVIRONMENT, NOT THE SCRIPT BODY.
#
# `${{ }}` is a TEXTUAL substitution performed before bash ever sees
# the script, so interpolating the input into the body made the seed
# part of this step's own source. Two things followed (#192):
#
# 1. `eval ${{ … }}` was UNQUOTED, so a seed of `a && b && c` was
# parsed by THIS shell as `eval a` && `b` && `c`. eval received
# only the first stage. And bash's `set -e` exemption for `&&`
# lists ("except the command following the final &&") means a
# failing first stage aborts nothing and exits 0 — the tail
# never runs, the step goes green, and Playwright then tests a
# half-seeded instance. Measured on pipelinq run 30800304506.
# 2. `echo "Running seed command: ${{ … }}"` expanded the seed's
# own `$(…)` in THIS shell, before the seed ran. pipelinq's
# bundle-truncation control printed identical before/after byte
# counts by construction and could not have caught anything.
#
# Passing it as data fixes both, and also means a seed containing a
# quote can no longer reshape the surrounding script.
SEED_COMMAND: ${{ inputs.playwright-seed-command }}
run: |
cd server
echo "Running seed command: ${{ inputs.playwright-seed-command }}"
eval ${{ inputs.playwright-seed-command }}
echo "Seed command completed."
printf 'Running seed command: %s\n' "$SEED_COMMAND"
# `eval` IS still required. A survey of the fleet's callers found
# launchpad passing `OC_PASS=… php occ user:add … || true` — an
# assignment prefix and a `||` list. Running "$SEED_COMMAND" as a
# bare command would exec that whole string as one argv word.
# What matters is that the expansion is QUOTED, so the entire seed
# reaches eval as one unit and eval's combined status is the answer.
eval "$SEED_COMMAND" && SEED_RC=0 || SEED_RC=$?
if [ "$SEED_RC" -ne 0 ]; then
# Diagnostics on STDERR. An `::error::` written to stdout is eaten
# whole when a caller wraps the step's output in `$(…)`, leaving a
# bare non-zero exit with no stated cause.
{
echo "::error::Seed command failed with exit ${SEED_RC}: ${SEED_COMMAND}"
echo "Refusing to run Playwright against a half-seeded instance — a full,"
echo "credible-looking pass/fail tally from an unseeded server is worse than"
echo "no tally at all. Fix the seed, or unset playwright-seed-command."
} >&2
exit "$SEED_RC"
fi
echo "Seed command completed (exit 0)."

- name: Run Playwright tests
run: |
Expand Down Expand Up @@ -2936,11 +3000,32 @@ jobs:

- name: Seed test data
if: inputs.playwright-seed-command != ''
env:
# Same contract as the E2E job's seed step above — see the long note
# there. Short version (#192): interpolating the input into the
# script body made `eval ${{ … }}` unquoted, so an `a && b` seed was
# split by THIS shell, eval got only `a`, and bash's `set -e`
# exemption for `&&` lists let a failing first stage exit 0 while
# printing "Seed command completed."
SEED_COMMAND: ${{ inputs.playwright-seed-command }}
run: |
cd server
echo "Running seed command: ${{ inputs.playwright-seed-command }}"
eval ${{ inputs.playwright-seed-command }}
echo "Seed command completed."
printf 'Running seed command: %s\n' "$SEED_COMMAND"
# Quoted expansion: the whole seed reaches eval as one unit and
# eval's combined status is the step's. eval is retained because
# fleet callers pass assignment prefixes and `||` lists.
eval "$SEED_COMMAND" && SEED_RC=0 || SEED_RC=$?
if [ "$SEED_RC" -ne 0 ]; then
# Diagnostics on STDERR — an `::error::` on stdout is swallowed by
# any caller that captures the step's output with `$(…)`.
{
echo "::error::Seed command failed with exit ${SEED_RC}: ${SEED_COMMAND}"
echo "Refusing to capture documentation screenshots of a half-seeded"
echo "instance — the images would be committed and would look real."
} >&2
exit "$SEED_RC"
fi
echo "Seed command completed (exit 0)."

- name: Run docs-capture Playwright project
run: |
Expand Down
Loading
Loading