Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

catch if we attempted to send metrics before connfiguring #400

Merged
merged 1 commit into from
Sep 21, 2020

Conversation

danmayer
Copy link
Owner

add a test that would have caught the view tracker race condition

How to test fully isolated Coverband Rails Initialization

Follow up on the previous race condition fix.

I found that I couldn't test the Coverband configuration in full isolation with Capybara... I still built off the forked tests which should help keep the test code isolated, but I ended up forking off an entire server and then testing against that server.

  • This is a bit complicated
  • I tried a few other paths, but wasn't able to find anything that would have caught the race condition bug
  • This should be a pretty re-usable pattern that we could write similar tests to verify coverage with multiple redises available.

Error Output

If I run this code against Coverband before the fix was committed this is how the test fails...

 bundle exec ruby test/forked/rails_view_tracker_stack_test.rb
D, [2020-09-19T16:04:40.642761 #28774] DEBUG -- : using default configuration
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
Run options: --seed 46770

# Running:

Traceback (most recent call last):
	36: from /Users/danmayer/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `<main>'
	35: from /Users/danmayer/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `eval'
	34: from /Users/danmayer/.rvm/gems/ruby-2.5.3/bin/rackup:23:in `<main>'
	33: from /Users/danmayer/.rvm/gems/ruby-2.5.3/bin/rackup:23:in `load'
	32: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/bin/rackup:4:in `<top (required)>'
	31: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/server.rb:148:in `start'
	30: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/server.rb:283:in `start'
	29: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/server.rb:354:in `wrapped_app'
	28: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/server.rb:219:in `app'
	27: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/server.rb:319:in `build_app_and_options_from_config'
	26: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/builder.rb:40:in `parse_file'
	25: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/builder.rb:49:in `new_from_string'
	24: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/builder.rb:49:in `eval'
	23: from /Users/danmayer/projects/coverband/test/rails5_dummy/config.ru:in `<main>'
	22: from /Users/danmayer/projects/coverband/test/rails5_dummy/config.ru:in `new'
	21: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/builder.rb:55:in `initialize'
	20: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/rack-2.0.6/lib/rack/builder.rb:55:in `instance_eval'
	19: from /Users/danmayer/projects/coverband/test/rails5_dummy/config.ru:1:in `block in <main>'
	18: from /Users/danmayer/projects/coverband/test/rails5_dummy/config.ru:1:in `require_relative'
	17: from /Users/danmayer/projects/coverband/test/rails5_dummy/config/environment.rb:2:in `<top (required)>'
	16: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/railties-5.2.2/lib/rails/application.rb:361:in `initialize!'
	15: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/railties-5.2.2/lib/rails/initializable.rb:60:in `run_initializers'
	14: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
	13: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
	12: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
	11: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:347:in `call'
	10: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:347:in `each'
	 9: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
	 8: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
	 7: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
	 6: from /Users/danmayer/.rvm/rubies/ruby-2.5.3/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
	 5: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/railties-5.2.2/lib/rails/initializable.rb:61:in `block in run_initializers'
	 4: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/railties-5.2.2/lib/rails/initializable.rb:32:in `run'
	 3: from /Users/danmayer/.rvm/gems/ruby-2.5.3/gems/railties-5.2.2/lib/rails/initializable.rb:32:in `instance_exec'
	 2: from /Users/danmayer/projects/coverband/lib/coverband/utils/railtie.rb:21:in `block in <class:Railtie>'
	 1: from /Users/danmayer/projects/coverband/lib/coverband/utils/railtie.rb:21:in `new'
/Users/danmayer/projects/coverband/lib/coverband/collectors/view_tracker.rb:23:in `initialize': Coverband: view tracker initialized before configuration! (RuntimeError)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 9999: Connection refused
F

Failure:
RailsWithoutConfigStackTest#test_check_view_tracker [test/forked/rails_view_tracker_stack_test.rb:18]:
Expected nil to be truthy.


bin/rails test test/forked/rails_view_tracker_stack_test.rb:16

add a test that would have caught the view tracker race conditiion
@danmayer danmayer requested a review from kbaum September 19, 2020 22:50
@@ -20,6 +20,7 @@ class ViewTracker

def initialize(options = {})
raise NotImplementedError, "View Tracker requires Rails 4 or greater" unless self.class.supported_version?
raise "Coverband: view tracker initialized before configuration!" if !Coverband.configured? && ENV["COVERBAND_TEST"] == "test"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we only want to raise this error for ENV["COVERBAND_TEST"] == "test"?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried about this raising in various scenarios like disabling coverband in RAILS env test or not having a redis in some CI build step, so I wanted to start with just something that let us test and verify the fix... I will keep looking if we can turn it on more extensively

shutdown_server
end

test "check view tracker" do
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. One thought here and probably not very important. Could use use capybara to hit a remote server here? I've done this before:

Capybara.run_server = false
Capybara.app_host = http://localhost:9999/

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, I hadn't thought of that, it would be nice to use capybara visit and asset helpers vs CURL.

I can give it a go

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obviously not so important but i had done it before in a project so figure would mention.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm tried several things, basically rack-test driver doesn't support it then hit various issues when trying to use selenium with run_server = false... seems promising but requires some tweaking

@danmayer danmayer merged commit 62179dc into master Sep 21, 2020
@danmayer danmayer deleted the background_check branch September 21, 2020 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants