[bug-hunter] ReportFailedTests ignores invalid SERVERLESS/DRY_RUN values

[github-actions]

Impact

mage ReportFailedTests silently accepts invalid boolean values for SERVERLESS and DRY_RUN, then proceeds with default false values. This hides CI misconfiguration and causes incorrect behavior instead of failing fast with a clear error.

Reproduction Steps

1. Create and run this minimal repro script:

#!/usr/bin/env bash
set -euo pipefail
cd /home/runner/work/integrations/integrations
export PATH="$(go env GOPATH)/bin:$PATH"
if ! command -v mage >/dev/null 2>&1; then
  go install github.com/magefile/mage@v1.15.0
fi
mkdir -p /tmp/gh-aw/agent/empty-test-results
SERVERLESS=invalid DRY_RUN=invalid mage -v ReportFailedTests /tmp/gh-aw/agent/empty-test-results

2. Observe exit code and output.

Expected vs Actual

Expected: command fails with parse errors (for invalid boolean env values), e.g. failed to parse SERVERLESS value / failed to parse DRY_RUN value.

Actual: command succeeds (exit code 0) and continues:

Running target: ReportFailedTests
path:  /tmp/gh-aw/agent/empty-test-results

Failing Test

A minimal reproducible script is shown above. It deterministically reproduces the bug by setting invalid boolean values.

Evidence

• Buggy error checks compare the same variable to itself, so the condition is always false:
  • magefile.go:167 -> if err != err {
  • magefile.go:199 -> if err != err {
• Correct pattern exists nearby in same function:
  • magefile.go:190 -> if err != nil {
• Local reproduction result:
  • command: SERVERLESS=invalid DRY_RUN=invalid mage -v ReportFailedTests /tmp/gh-aw/agent/empty-test-results
  • observed exit code: 0
  • output contains no parse error.

---

What is this? | From workflow: Bug Hunter

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

• expires on Mar 24, 2026, 11:42 AM UTC

[strawgate]

/ai is there somewhere in CI where we've made this mistake and the tests arent running? can you assess?

[github-actions]

Good call — I audited CI for this exact pattern, and there are a few places where invalid boolean-like values can silently change behavior or skip test jobs.

Most relevant findings:

• magefile.go:167 and magefile.go:199 use if err != err after strconv.ParseBool(...) in ReportFailedTests, so parse errors are ignored (this is the bug in [bug-hunter] ReportFailedTests ignores invalid SERVERLESS/DRY_RUN values #17857).

• Buildkite step gating uses strict string checks like build.env('...') == "true" in .buildkite/pipeline.schedule-daily.yml (L29, L43, L58, L73, L87, L101, L113, L125, L134).

  • If any of those env vars are set to values like True, 1, yes, etc., the steps are skipped and tests won’t run.

• Shell-side boolean checks are also strict string compares:

  • .buildkite/scripts/common.sh:530 (SERVERLESS == "true")
  • .buildkite/scripts/build_packages.sh:118 and .buildkite/scripts/backport_branch.sh:282 (DRY_RUN == "true")
  • Invalid values don’t error; they fall through as false.

What I could verify about current scheduled CI config:

• catalog-info.yaml schedule env values are explicitly set to lowercase string "true" for TEST_PACKAGES_SERVERLESS, TEST_PACKAGES_9_BRANCH, etc. (L188-L202), so scheduler-defined runs look correct.

So: I don’t see a guaranteed current misconfiguration in-repo, but yes — there are multiple CI decision points with this same fragility where non-"true" values can cause tests to not run or run in the wrong mode.

If you want, I can open a follow-up PR to harden these checks (centralized strict bool validation + explicit error/annotation on invalid values).

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:[bug-hunter] ReportFailedTests ignores invalid SERVERLESS/DRY_RUN values #17857 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.