feat(action): single-test mode, partial-run fixes, and an end-to-end self-test - #1
Merged
Conversation
… action manifest)
…hs failed validation)
Single-test runs don't produce a JUnit report, but the upload step and junit-file output still keyed on the report path — so a stale testsprite-junit.xml (e.g. from a prior --all run in the same job) could be uploaded as this run's result and misattribute old data. Gate the upload on an empty test-id and emit an empty junit-file output in single-test mode. selftest v3-single-test now plants a stale report and asserts it is neither advertised nor uploaded.
…real-user-ref on main only The self-test caught a real defect: with allow-partial=true, a frontend-only V2 project dispatches nothing and the CLI exits 7 (incomplete), which the action's final `exit "$code"` propagated — reddening the job despite allow-partial. Under allow-partial (full-project runs) accept a skipped/deferred-only batch as GREEN, reddening only on a real test failure (failed>0). Also point the real-user-remote-ref self-test job at @main and gate it to main-only: a remote `uses:` resolves from the default branch, so pinning a stale feature-branch SHA tested old code and always failed. It now validates the real consumer shape against main after the change lands.
…crets Move the hardcoded dev endpoint (api.testsprite.com:11443) and the dev project/test ids out of the workflow file and into repo secrets (TESTSPRITE_ENDPOINT_URL / TESTSPRITE_V2_PROJECT / TESTSPRITE_V3_PROJECT / TESTSPRITE_V3_TEST_ID). Secrets are masked in Actions logs and absent from the YAML, so nothing dev-specific leaks if this repo is ever made public. The API keys were already secrets.
… assertions
Adversarial-review follow-ups:
- allow-partial now reds only on a REAL test failure. `failed` was read from the
CLI summary's `.failed` = total-passed-timedOut, which folds deferred / conflict
/ not-found (partials) into the count — so allow-partial would red a batch that
merely couldn't dispatch everything. Now `failed` = runs[].status in
{failed,blocked}, and allow-partial reds only on those.
- allow-partial that executed NOTHING (all skipped/deferred) now emits a loud
::warning instead of a silent green.
- Single-test verdict derives red/green from the run status (authoritative),
not the CLI exit code alone, so a blocked/timeout single test can't leave the
job green while the `failed` output is 1.
- New `skipped` action output.
- Self-test: v2-guard asserts skipped>0 (proves the guard fired, not that the
action broke for an unrelated reason); v2-allow-partial asserts skipped>0 as a
V2-skip-premise + account-drift/credit tripwire (skipped=0 => account may be V3,
executing + burning credits). Real-backend jobs restricted to main + ci/**.
- Dropped an internal ticket ref from a comment.
Collaborator
Author
Adversarial-review hardening (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the composite action's hardening + a real-runner self-test that exercises it end-to-end against the dev backend.
action.yml
test-idinput — run a single test by id instead of the whole project. Mutually exclusive withfilter; produces no JUnit (the CLI's JUnit is batch-only) and never uploads/advertises a stale report at the default path. The job still goes red/green on the verdict + emits a summary.allow-partialactually greens a skipped run — a frontend-only V2 project dispatches nothing and the CLI exits 7 (incomplete); the action used to propagate that viaexit "$code"and red the job despiteallow-partial: true. Now, underallow-partial(full-project runs) a skipped/deferred-only batch is GREEN, reddening only on a real test failure.mkdir -pthe report's parent dir so a nestedreport-file(e.g.results/junit.xml) doesn't fail.junit-fileoutput empty in single-test mode.selftest.yml (new)
Runs the action on a real GitHub runner against the dev backend — no credits (the V2 test project is frontend-only, so the V2 path skips execution):
v2-guard-blocks-partial— asserts the partial-run guard FAILS the job (FE skipped,allow-partial:false).v2-allow-partial— same skip, job stays GREEN.real-user-remote-ref— references the action remotely (@main, consumer shape); gated to main-only since a remoteuses:resolves from the default branch.v3-real-execution/v3-single-test(workflow_dispatch, manual — control credits) — real browser run producing a JUnit with cases; single-test-by-id mode + a stale-report-not-uploaded guard.Validation
Self-test green on this branch:
v2-guard-blocks-partial+v2-allow-partialboth pass; the v3/remote jobs are correctly skipped off-main / off-dispatch.Beta scope — not published to the Marketplace.