Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 10, 2025

This PR fixes a critical issue where failed BDD tests were incorrectly reporting as successful in the CI workflow, causing failed builds to appear as passing.

Problem

The staging test workflow in .github/workflows/on_call_staging_test.yaml was using a pipeline command:

docker compose -f docker-compose.yml -f docker-compose.ci.yml run test 2>&1 | tee bdd_output.txt
bdd_result=$?

In bash, pipelines return the exit code of the last command by default. This meant that $? captured the exit code of tee (which is always 0) instead of the docker compose run test command, causing failed BDD tests to be reported as successful.

For example, the issue showed this output:

...F...

Failures:

1) Scenario: All recent posts should have thumbnail images
   ✖ And each recent post should have a picture tag with an img element
       Error: function timed out, ensure the promise resolves within 5000 milliseconds

✅ BDD tests passed ← This was incorrect

Solution

Added set -o pipefail before the BDD test execution. This bash option ensures that pipelines return the exit code of the first failing command rather than the last command.

set -o pipefail  # Ensure pipeline returns exit code of first failing command
docker compose -f docker-compose.yml -f docker-compose.ci.yml run test 2>&1 | tee bdd_output.txt
bdd_result=$?

Now when BDD tests fail:

  • The docker compose run test command exits with code 1
  • The pipeline correctly returns exit code 1 (not 0 from tee)
  • The workflow properly fails the CI job

Testing

The fix has been thoroughly validated with multiple test scenarios:

  • ✅ Failed BDD tests now correctly fail the CI job (exit code 1)
  • ✅ Successful BDD tests continue to work as expected (exit code 0)
  • ✅ The change is minimal and surgical (only one line added)

Fixes #260.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: dannystaple <426859+dannystaple@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Failed CI Job results in "success" incorrectly Fix BDD test failure masking by adding pipefail to staging workflow Aug 10, 2025
@Copilot Copilot AI requested a review from dannystaple August 10, 2025 16:59
Copilot finished work on behalf of dannystaple August 10, 2025 16:59
Copy link
Contributor

github-actions bot commented Aug 10, 2025

🦙 MegaLinter status: ❌ ERROR

Descriptor Linter Files Fixed Errors Warnings Elapsed time
❌ ACTION actionlint 5 6 0 0.08s
✅ BASH bash-exec 1 0 0 0.0s
✅ BASH shellcheck 1 0 0 0.06s
⚠️ BASH shfmt 1 1 0 0.01s
❌ C cppcheck 8 8 0 0.32s
❌ CPP cppcheck 8 8 0 0.31s
✅ JSON npm-package-json-lint yes no no 0.45s
✅ JSON v8r 12 0 0 6.8s
❌ MARKDOWN markdown-link-check 738 213 0 41.1s
✅ REPOSITORY gitleaks yes no no 13.28s
✅ REPOSITORY git_diff yes no no 0.92s
❌ REPOSITORY grype yes 2 no 35.76s
✅ REPOSITORY secretlint yes no no 43.73s
✅ REPOSITORY syft yes no no 1.58s
✅ REPOSITORY trivy-sbom yes no no 3.5s
✅ REPOSITORY trufflehog yes no no 4.88s
✅ XML xmllint 1 0 0 189.68s
✅ YAML v8r 13 0 0 7.47s

See detailed report in MegaLinter reports

MegaLinter is graciously provided by OX Security

@dannystaple dannystaple marked this pull request as ready for review August 10, 2025 18:46
@dannystaple dannystaple merged commit 09e0a18 into master Aug 10, 2025
6 checks passed
@dannystaple dannystaple deleted the copilot/fix-260 branch August 10, 2025 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failed CI Job results in "success" incorrectly
2 participants