Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raise an exception if tests or changelog generation fails #294

Merged
merged 2 commits into from
Jan 3, 2024

Conversation

G-Rath
Copy link
Contributor

@G-Rath G-Rath commented Jan 2, 2024

I realised while working on my recent PRs that CI wasn't actually failing when bundle exec rake [test] reported errors - turns out that tests are being invoked by system which by default does not do any error handling; instead it sets $? to the exit code which can then be checked.

Ruby 2.6 added an exception option which when true causes system to raise an exception if the command exits with a non-zero code, making this very easy to address.

I've checked the codebase and confirmed these are the only uses of system.


Before submitting the PR make sure the following are checked:

  • The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
  • Wrote good commit messages.
  • Commit message starts with [Fix #issue-number] (if the related issue exists).
  • Feature branch is up-to-date with master (if not - rebase it).
  • Squashed related commits together.
  • Added tests.
  • Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
  • Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

@G-Rath G-Rath changed the title Raise an exception if tests fail, to ensure CI fails Raise an exception if tests or changelog generation fails Jan 2, 2024
@G-Rath
Copy link
Contributor Author

G-Rath commented Jan 2, 2024

Ok so it looks like the JRuby and "oldest supported RuboCop version" jobs are already failing which this change has revealed - I've added an env variable to control if test failures should actually be surfaced to use in those two jobs so that this PR can be landed as a dedicated change, and then those jobs can be addressed in their own dedicated PRs (since "mostly working CI" is better than "not working CI" and I have no context on either of the failures so expect they'll take more time to resolve).

I've kept this as a separate commit in case you want to do something different @koic, but if you're happy with this path I can squash it.

Also it looks like MultipleAssertionsTest#test_generates_a_todo_based_on_the_worst_violation is flakey - I've confirmed this locally: running multiple times, every so often I get a failure with an incorrect todo count; again something to resolve in a dedicated PR - for now just rerunning that job for this PR should get it passing.

Comment on lines +24 to +26
error_on_failure = ENV.fetch('ERROR_ON_TEST_FAILURE', 'true') != 'false'

system("bundle exec minitest-queue #{Dir.glob('test/**/*_test.rb').shelljoin}", exception: error_on_failure)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any problem with always setting it to true for exception option?

Suggested change
error_on_failure = ENV.fetch('ERROR_ON_TEST_FAILURE', 'true') != 'false'
system("bundle exec minitest-queue #{Dir.glob('test/**/*_test.rb').shelljoin}", exception: error_on_failure)
system("bundle exec minitest-queue #{Dir.glob('test/**/*_test.rb').shelljoin}", exception: true)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, two of the jobs are currently failing and I think fixing them should be done in dedicated PRs - once they're fixed, this can then be set to true.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, OK. Let's merge this first 👍

@koic koic merged commit f47bcbd into rubocop:master Jan 3, 2024
13 checks passed
@G-Rath G-Rath deleted the error-on-tests-failure branch January 3, 2024 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants