Allow assertionless tests to be reported #51625

fatkodima · 2024-04-21T18:53:16Z

Inspired by https://railsatscale.com/2024-01-25-catching-assertionless-tests/.

It is very easy to write a "false positive" broken test (as can be seen from the linked article) that actually tests nothing (or can become such in the future). A simple example:

def test_active
  active_users = User.active.to_a
  active_users.each do |user|
    assert user.active?
  end
end

We have some assertions used in the test case, so, looks good? But if we change the scope and it will now return 0 records, this test will still pass and none of the assertions will be run. For this test to be more robust, we need at least to test that active_users contains any records.

This PR now allows to easily detect such tests.

# config/environments/test.rb
config.active_support.assertionless_tests_behavior = :raise # also available :ignore and :log

With this configuration, assertionless tests we will now be marked as failed and not silently pass.

Currently, this feature is configured to :ignore such tests by default, but we can rollout it to eventually :log and then :raise in the future.

Some of such problematic tests were already fixed before, like in #48065.

As a follow up to this PR, I would like to enable :raise behavior for all the frameworks. Currently there are a few offences fixing which will make this PR bigger than is needed.

cc @nvasilevski @byroot (interested in your thoughts about this)

Earlopain · 2024-04-22T12:07:54Z

This seems pretty cool, I tried this out and it found a few cases. One annoyance when using :raise: the error location points to rails internals:

Failure:
ArtistsControllerTest#test_: An artists controller reverting an artist should work. [/usr/local/bundle/bundler/gems/rails-f6a1fc843207/activesupport/lib/active_support/testing/assertionless_tests.rb:33]:
Test is missing assertions

This makes it hard to find the actual test location when using something like minitest/spec or shoulda-context since most of the time you can't just search for the test name.

fatkodima · 2024-04-22T12:11:08Z

Thanks for the feedback and I am glad you found it helpful too! I will look into this problem more closely 👍 .

nvasilevski

Thank you!

I'm a little biased but I do find this change useful and I would love to see it in Rails ❤️

activesupport/lib/active_support/railtie.rb

nvasilevski · 2024-04-22T13:05:45Z

activesupport/lib/active_support/testing/assertionless_tests.rb

+        return if skipped? || error?
+
+        if assertions == 0
+          ActiveSupport::TestCase.assertionless_tests_behavior&.call


I think people find it helpful if we could pass name (if that's a correct getter for the test name) so we could include it in both the error message or the log message

Added a name and also a file/line combination to be easier detectable, as in the example for spec style given by @Earlopain.

This is pretty great. Minitest prints relative paths by default in recent versions though, I have created a patch that does that here as well. It uses a custom assertion class to control the location:

diff --git a/activesupport/lib/active_support/testing/assertionless_tests.rb b/activesupport/lib/active_support/testing/assertionless_tests.rb index 862b0f3d45..25d1d07c32 100644 --- a/activesupport/lib/active_support/testing/assertionless_tests.rb +++ b/activesupport/lib/active_support/testing/assertionless_tests.rb @@ -11,6 +11,15 @@ def self.prepended(klass) klass.assertionless_tests_behavior = :ignore end + class NoAssertions < Minitest::Assertion + attr_reader :location + + def initialize(message, test_location) + @location = test_location + super(message) + end + end + module ClassMethods def assertionless_tests_behavior=(behavior) @assertionless_tests_behavior = @@ -26,9 +35,9 @@ def assertionless_tests_behavior=(behavior) ActiveSupport::Logger.new($stderr) end - ->(message) { logger.warn(message) } + ->(message, test_location, test_name) { logger.warn("#{message}: `#{test_name}` #{test_location}") } when :raise - ->(message) { raise Minitest::Assertion, message } + ->(message, test_location, *) { raise NoAssertions.new(message, test_location) } else raise ArgumentError, "assertionless_tests_behavior must be one of :ignore, :log, or :raise." end @@ -42,8 +51,8 @@ def after_teardown if assertions == 0 && (behavior = ActiveSupport::TestCase.assertionless_tests_behavior) file, line = self.class.instance_method(name).source_location - message = "Test is missing assertions: `#{name}` #{file}:#{line}" - behavior.call(message) + test_location = "#{file}:#{line}" + behavior.call("Test is missing assertions", test_location, name) end end end diff --git a/activesupport/test/testing/assertionless_tests_test.rb b/activesupport/test/testing/assertionless_tests_test.rb index eb7b21285a..90cfbf99df 100644 --- a/activesupport/test/testing/assertionless_tests_test.rb +++ b/activesupport/test/testing/assertionless_tests_test.rb @@ -51,10 +51,12 @@ class AssertionlessTestsRaise < ActiveSupport::TestCase module AfterTeardown def after_teardown with_assertionless_tests_behavior(:raise) do - assert_raise(Minitest::Assertion, - match: /Test is missing assertions: `test_without_assertions` .+assertionless_tests_test\.rb:\d+/) do + e = assert_raise(Minitest::Assertion, match: "Test is missing assertions") do super end + report = Minitest::Result.from(self) + report.failures << e + assert_match(/\[test\/testing\/assertionless_tests_test.rb:\d+]/, report.to_s) end end end

As far as I can tell this is all done through public api. It's pretty much only the location method, which is documented. Sample output:

Failure: AssertionlessTestsTest::AssertionlessTestsRaise#test_without_assertions [test/testing/assertionless_tests_test.rb:7]: Test is missing assertions

Thanks for the suggestion but I am not sure these complications are needed and I am not comfortable overriding location method, even though it is a public method.

Let's see what others will say about this.

activesupport/test/testing/assertionless_tests_test.rb

byroot · 2024-04-23T10:11:03Z

@byroot (interested in your thoughts about this)

I'm generally sympathetic to the idea, I'll just have to take time to review this (this week hopefully), and also think about what the upgrade story need to be.

Also even though it's 3rd party project, we'd need to double check whether it breaks rspec-rails (not that it would be blocker, but if it breaks it would be nice of us to give them advance notice)

fatkodima · 2024-04-24T22:38:16Z

Also even though it's 3rd party project, we'd need to double check whether it breaks rspec-rails (not that it would be blocker, but if it breaks it would be nice of us to give them advance notice)

I did not get why this PR is relevant to rspec-rails?

byroot · 2024-04-25T06:48:48Z

I did not get why this PR is relevant to rspec-rails?

I'm not sure about this specific case. But IIRC rspec-rails uses ActiveSupport::TestCase under the hood to have access to various test helpers, and in the past similar changes broke it.

PS: I wanted to review this but for some weird reason GitHub isn't accessible from my hotel wifi... but I hope to have a bit of time this week end.

fatkodima · 2024-04-25T09:21:30Z

Made a simple testing with rspec-rails and this feature - assertionless tests are not reported of course, but there are no errors too, so works as expected.

byroot

Generally looks good, but we probably ought to have this disabled by default for upgrading apps.

So this mean adding the usual load_default 7.2 and new_framework_defaults, things.

byroot · 2024-04-26T10:09:19Z

activesupport/lib/active_support/testing/assertionless_tests.rb

+                if defined?(Rails.logger) && Rails.logger
+                  Rails.logger


Not a big fan of checking for Rails existence here. Might make more sense to introduce ActiveSupport.logger.

This is how it is done in another place too

rails/activesupport/lib/active_support/deprecation/behaviors.rb

Lines 25 to 33 in f04c59b

log: ->(message, callstack, deprecator) do

logger =

if defined?(Rails.logger) && Rails.logger

Rails.logger

else

require "active_support/logger"

ActiveSupport::Logger.new($stderr)

end

logger.warn message

Maybe introduce that change separately?

byroot · 2024-04-26T10:13:13Z

activesupport/lib/active_support/testing/assertionless_tests.rb

+            else
+              raise ArgumentError, "assertionless_tests_behavior must be one of :ignore, :log, or :raise."


Probably YAGNI, but we could accept any callable here.

Added.
I noticed, that .with helper is working only with objects, and not with modules. So something like

ActiveSupport::TestCase.with(assertionless_tests_behavior: :ignore, &block)

is not working. Was that intentional?

I just confirmed it does, I suspect that core_ext just instead loaded?

fatkodima · 2024-04-26T13:20:29Z

but we probably ought to have this disabled by default for upgrading apps.

So this mean adding the usual load_default 7.2 and new_framework_defaults, things.

It was disabled by default. I changed it to be :log for new apps and configured in the new_framework_defaults. Did I get it right?

byroot · 2024-04-26T13:27:15Z

activesupport/lib/active_support/testing/assertionless_tests.rb

+    # This is helpful in detecting broken tests that do not perform intended assertions.
+    module AssertionlessTests # :nodoc:
+      def self.prepended(klass)
+        klass.singleton_class.attr_reader :assertionless_tests_behavior


I think we should make this a class_attribute, so that you can change the behavior on a per child class basis. (don't forget not pass instance_accessor: false, instance_predicate: false)

Looks like class_attribute is not working with overridden setter method, as in our case? 🤔 Am I doing something wrong or there are other ways?

Yeah, you can't just override the setter, best is to use class_attirbute :_attr_name, then from you override set self._attr_name.

Thanks, that worked. Ready for review again.

byroot

Looks good. But the CI neeed some fixing.

ghiculescu · 2024-05-07T02:21:30Z

FYI for anyone who came across this and wondered where the config went: 6a6c7e6

zzak · 2024-05-19T01:50:56Z

I found a few of these in the railties suite, there maybe more elsewhere:

Test is missing assertions: `test_config.session_store_with_:active_record_store_with_activerecord-session_store_gem` test/application/configuration_test.rb:1763
Test is missing assertions: `test_playback_use` test/configuration/middleware_stack_proxy_test.rb:37
.Test is missing assertions: `test_playback_insert_after` test/configuration/middleware_stack_proxy_test.rb:27
.Test is missing assertions: `test_order` test/configuration/middleware_stack_proxy_test.rb:62
.Test is missing assertions: `test_playback_insert_before` test/configuration/middleware_stack_proxy_test.rb:17
.Test is missing assertions: `test_playback_move` test/configuration/middleware_stack_proxy_test.rb:52
.Test is missing assertions: `test_playback_move_after` test/configuration/middleware_stack_proxy_test.rb:57
.Test is missing assertions: `test_playback_delete` test/configuration/middleware_stack_proxy_test.rb:42
.Test is missing assertions: `test_playback_move_before` test/configuration/middleware_stack_proxy_test.rb:47
.Test is missing assertions: `test_playback_insert` test/configuration/middleware_stack_proxy_test.rb:22
.Test is missing assertions: `test_playback_swap` test/configuration/middleware_stack_proxy_test.rb:32
Test is missing assertions: `test_middleware_referenced_in_configuration` test/railties/engine_test.rb:656
Test is missing assertions: `test_models_without_table_do_not_panic_on_scope_definitions_when_loaded` test/application/loading_test.rb:82
Test is missing assertions: `test_loading_only_yml_fixtures` test/application/rake_test.rb:202
Test is missing assertions: `test_can_boot_with_an_unhealthy_database` test/application/initializers/frameworks_test.rb:219
Test is missing assertions: `test_ARGV_is_populated` test/application/generators_test.rb:180
Test is missing assertions: `test_tmp:clear_should_work_if_folder_missing` test/application/rake/tmp_test.rb:41

fatkodima · 2024-05-19T08:45:04Z

Interesting why these were not detected when originally the error was raised.

Not related, but I am also seeing Minitest::CI] Generating test report in JUnit XML format... in all the tests. Curious, how that is useful? As I don't see that generated files are stored somewhere or accessible somehow.

zzak · 2024-05-19T10:58:52Z

🤔

That message comes from the minitest-ci gem which we use here:

rails/tools/test_common.rb

Line 9 in 965b8c3

    
           Minitest::Ci.report_dir = File.join(__dir__, "../test-reports/#{ENV['BUILDKITE_JOB_ID']}")

We generate the JUnit test reports for every job and upload them to artifacts:

I don't think we're actually using that data to my knowledge, but in theory we could use it with Buildkite's Test Analytics features.

For example, we can use that data to find slow or flaky tests and track their performance overtime.

rails-bot bot added activesupport docs labels Apr 21, 2024

nvasilevski approved these changes Apr 22, 2024

View reviewed changes

fatkodima force-pushed the assertionless-tests branch from f6a1fc8 to 1fe871c Compare April 24, 2024 22:36

fatkodima force-pushed the assertionless-tests branch from 1fe871c to 39db54c Compare April 24, 2024 23:03

byroot reviewed Apr 26, 2024

View reviewed changes

fatkodima force-pushed the assertionless-tests branch from 39db54c to 3c74e16 Compare April 26, 2024 13:20

rails-bot bot added the railties label Apr 26, 2024

byroot reviewed Apr 26, 2024

View reviewed changes

fatkodima force-pushed the assertionless-tests branch from 3c74e16 to 73d6d52 Compare April 26, 2024 14:27

byroot approved these changes Apr 26, 2024

View reviewed changes

Allow assertionless tests to be reported

76966f9

fatkodima force-pushed the assertionless-tests branch from 73d6d52 to 76966f9 Compare April 26, 2024 15:24

byroot merged commit 664470b into rails:main Apr 26, 2024
4 checks passed

fatkodima deleted the assertionless-tests branch April 26, 2024 19:35

This was referenced Apr 29, 2024

Raise when a block is passed to ActiveRecord::Relation#with #51684

Merged

Enable raising for assertionless tests for internal framework tests #51693

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow assertionless tests to be reported #51625

Allow assertionless tests to be reported #51625

fatkodima commented Apr 21, 2024 •

edited

Loading

Earlopain commented Apr 22, 2024

fatkodima commented Apr 22, 2024

nvasilevski left a comment

nvasilevski Apr 22, 2024

fatkodima Apr 24, 2024

Earlopain Apr 25, 2024

fatkodima Apr 25, 2024

byroot commented Apr 23, 2024

fatkodima commented Apr 24, 2024 •

edited

Loading

byroot commented Apr 25, 2024

fatkodima commented Apr 25, 2024

byroot left a comment

byroot Apr 26, 2024

fatkodima Apr 26, 2024

byroot Apr 26, 2024

fatkodima Apr 26, 2024

byroot Apr 26, 2024

fatkodima commented Apr 26, 2024

byroot Apr 26, 2024

fatkodima Apr 26, 2024

byroot Apr 26, 2024

fatkodima Apr 26, 2024

byroot left a comment

ghiculescu commented May 7, 2024

zzak commented May 19, 2024

fatkodima commented May 19, 2024

zzak commented May 19, 2024

	log: ->(message, callstack, deprecator) do
	logger =
	if defined?(Rails.logger) && Rails.logger
	Rails.logger
	else
	require "active_support/logger"
	ActiveSupport::Logger.new($stderr)
	end
	logger.warn message

		else
		raise ArgumentError, "assertionless_tests_behavior must be one of :ignore, :log, or :raise."

Allow assertionless tests to be reported #51625

Allow assertionless tests to be reported #51625

Conversation

fatkodima commented Apr 21, 2024 • edited Loading

Earlopain commented Apr 22, 2024

fatkodima commented Apr 22, 2024

nvasilevski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byroot commented Apr 23, 2024

fatkodima commented Apr 24, 2024 • edited Loading

byroot commented Apr 25, 2024

fatkodima commented Apr 25, 2024

byroot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fatkodima commented Apr 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byroot left a comment

Choose a reason for hiding this comment

ghiculescu commented May 7, 2024

zzak commented May 19, 2024

fatkodima commented May 19, 2024

zzak commented May 19, 2024

fatkodima commented Apr 21, 2024 •

edited

Loading

fatkodima commented Apr 24, 2024 •

edited

Loading