Skip to content

Failed CI Job results in "success" incorrectly #260

@dannystaple

Description

@dannystaple

The following was seen in a CI job:

> orionrobots.github.io@1.0.0 test:bdd
> cucumber-js tests/staging/features --require tests/staging/step_definitions/**/*.js

...F...

Failures:

1) Scenario: All recent posts should have thumbnail images # tests/staging/features/recent-posts-thumbnails.feature:3
   ✔ Given the Staging site is started # tests/staging/step_definitions/website_steps.js:28
   ✔ When the index page is visited # tests/staging/step_definitions/website_steps.js:46
   ✔ Then the recent posts list should be present # tests/staging/step_definitions/website_steps.js:80
   ✖ And each recent post should have a picture tag with an img element # tests/staging/step_definitions/website_steps.js:98
       Error: function timed out, ensure the promise resolves within 5000 milliseconds
           at Timeout.<anonymous> (/app/src/node_modules/@cucumber/cucumber/lib/time.js:64:20)
           at listOnTimeout (node:internal/timers:608:17)
           at process.processTimers (node:internal/timers:543:7)

2 scenarios (1 failed, 1 passed)
7 steps (1 failed, 6 passed)
0m09.600s (executing steps: 0m09.421s)

✅ BDD tests passed

This is clearly wrong and failed. Something is wrong with the BDD system if a timed out/failed tests is allowed to pretend to be a success.

Acceptance criteria

  • A failed test in BDD should result in a workflow job failure.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions