Integrate with ActionDispatch::SystemTest #1813

fables-tales · 2017-05-05T13:47:39Z

TODO:

Implementation cleanup
ensure it works with infer_spec_type_from_file_location
docs
blog post with example
get some user testing in real rails apps

elsurudo · 2017-05-11T07:01:03Z

Great work! Any estimate on when this will be merged?

JonRowe · 2017-05-11T08:54:09Z

@elsurudo we'd gladly accept help on this, in the mean time our type: :feature specs work well with the new transactional setup and you can capybara on

This should JustWork™ in Rails 5.1+. I’m tracking rspec/rspec-rails#1813 so We can eventually integrate system specs (the now “official” Rails way) instead of feature specs.

fables-tales · 2017-06-16T20:14:15Z

This is now ready for user testing. Please let me know if you'd like to use this and give me some feedback

fables-tales · 2017-06-17T20:10:16Z

@JonRowe would you be able to provide review here?

derekprior · 2017-06-18T02:20:31Z

I tried this out on a Rails 5.1 project and got things mostly working. Here was my general process:

Update my rspec-rails dependency to point at this branch, and point other rspec-* deps at their master branch.
Remove database_cleaner gem and associated configuration! 🎊
Turned on config.use_transactional_fixtures because no more DatabaseCleaner! 🎆 🍾
Move all tests from spec/features to spec/system because we use infer_spec_type_from_file_location! (sorry (not sorry), Sam).
Add the following global configuration to maintain our previous use of Rack::Test for tests that don't require JS.

RSpec.configure do |config|
  config.before(:each, type: :system) do
    driven_by :rack_test
  end

  config.before(:each, type: :system, js: true) do
    driven_by :headless_chrome # a driver I define elsewhere
  end
end

Results

It kinda worked... I had a couple of problems.

We use an outermost feature keyword in our "feature" (now "system") specs with nested scenarios rather than describe and it. The use of feature seems to override both the location-inferred spec type and an explicitly set spec type and set the example group to feature. Doing a replace of feature with describe in spec/system fixed this issue.
I was not able to set the driver in a global before as specified above. Capybara would always use the Rails system test default. I was able to set the driver in line in the test.
System tests are configured to take a screenshot when a test fails. The screenshot is always written to tmp/screenshots/failures_.png which means you only get to review the last failure's screenshot. I've never used the Rails system tests, but judging by the dangling _ in the filename, I'm guessing there's supposed to be some sort of unique identifier there.

The first problem stems from the use of feature syntax. This is an old habit that maybe should just go away. I'm not sure how many people use it, but we use it extensively and exclusively at thoughtbot. I can't think of a good reason to keep it, but it's rather puzzling when one can't figure out why their global type: :system before blocks aren't running.

I also encountered what I consider to be an unfortunate, if minor, annoyance. I like my test output to be as clean as possible -- just the test status indicators with no output. System tests seem to muck this up quite a bit. See:

The first bit of output is from puma starting up, presumably when RSpec hits its first system test. I'm glad to see it doesn't log all requests, but it'd sure be awesome if we could suppress this output somehow.

Then you see my test failures. The screenshot writes a log in-line. The "RSpec-friendly" way to do this would be to include that output with the detailed failure message.

I'm guessing that these annoyances are mostly out of RSpec's control and have to do with how system tests are implemented in Rails. Is there something we could contribute back to Rails to make the output play nicer? It seems to be that if this outputs during minitest runs, it'd be equally annoying for folks like me? Maybe I'm just too much of a neat freak?

Poltergeist is no longer maintained Rails 5.1 introduces ActionDispatch::SystemTestCase which should allow our feature tests to be speed up when this is done: rspec/rspec-rails#1813

clupprich · 2017-06-22T10:49:59Z

I've followed the steps @derekprior describes in his comment above. Things work smoothly, besides the custom driven_by block:

RSpec.configure do |config|
  config.before(:each, type: :system) do
    driven_by :rack_test
  end

  config.before(:each, type: :system, js: true) do
    driven_by :chrome
  end
end

The driver isn't actually switched to rack_test or chrome for me, but the default selenium is used. Maybe that's because it's overwritten in this before block?

JonRowe

Nice to see how simple this is, few minor comments.

JonRowe · 2017-06-23T00:26:58Z

Gemfile

+  if /5(\.|-)1/ === RAILS_VERSION || "master" == RAILS_VERSION
+    gem 'capybara', '~> 2.13', :require => false
+  else
+    gem 'capybara', '~> 2.2.0', :require => false


Why doesn't bundler automatically work this out?

The pin to 2.2.0 prevents us from bumping further than the highest 2.2.x rails needs at least 2.13. However, we pin to the earliest version that we support, so this is why this is conditionaled like this.

JonRowe · 2017-06-23T00:27:25Z

Gemfile-rails-dependencies

 when /stable$/
  gem_list = %w[rails railties actionmailer actionpack activerecord activesupport]
  gem_list << 'activejob'  if version > '4-1-stable'
  gem_list << 'actionview' if version > '4-0-stable'
+  if RUBY_VERSION  >= "1.9.3"
+    gem_list << 'puma' if version > '5-0-stable'


5-0 or 5-1

JonRowe · 2017-06-23T00:29:23Z

example_app_generator/generate_app.rb

@@ -20,6 +20,7 @@
  gsub_file 'Gemfile', /^.*\bgem 'rails.*$/, ''
  gsub_file "Gemfile", /.*web-console.*/, ''
  gsub_file "Gemfile", /.*debugger.*/, ''
+  gsub_file "Gemfile", /.*puma.*/, ""


Will every generated app now need puma, shouldn't this be in a guard statement?

this deletes the generated puma requirement from the default rails generator, replacing it with ours.

JonRowe · 2017-06-23T01:33:58Z

features/system_specs/system_specs.feature

+    the default `driven_by(:selenium)` from Rails. If you want to override this
+    behaviour you can call `driven_by` manually in a test.
+
+


Excess white space, my old nemesis, we meet again.

JonRowe · 2017-06-23T01:34:17Z

features/system_specs/system_specs.feature

+    We encourage you to familiarse yourself with their documentation.
+
+    RSpec **does not** use your `ApplicationSystemTestCase` helper. Instead it uses
+    the default `driven_by(:selenium)` from Rails. If you want to override this


Might it be worth us overriding this and defaulting to rack test?

Strongly disagree, selenium is the default in Rails and so I think it's better to mirror that default.

Your call, just a suggestion

@derekprior

This is motivated by our usage of system test in RSpec. Puma lazily boots the first time a system test is used, but this causes some unfortunate output to appear in the middle of the user's green dots. An example of this can be seen in @derekprior's comment [here](rspec/rspec-rails#1813 (comment)). There are alternatives in RSpec where we attempt to intercept the puma boot and prevent the output from being made there, but that would involve some monkey patching. This seems like a cleaner solution.

JonRowe

I'm happy if you're happy @samphippen merge on 📗

abevoelker · 2017-06-25T16:27:49Z

features/system_specs/system_specs.feature

+
+    System specs are RSpec's wrapper around Rails' own
+    [system tests](http://guides.rubyonrails.org/testing.html#system-testing).
+    We encourage you to familiarse yourself with their documentation.


s/familiarse/familiarise/

A fresh Rails 5.1 app was generated with: ``` rails new . -d postgresql --skip-action-mailer --skip-spring --skip-listen --skip-coffee --skip-turbolinks --skip-test --skip-bundle --skip-keeps ``` Updated configuration files were added for the following: - Code Climate - EditorConfig - Rspec - Rubocop - Travis CI For now, the app will use Rspec feature specs instead of the new Rails 5.1 system tests pending resolution of this rspec/rspec-rails PR: rspec/rspec-rails#1813

Poltergeist is no longer maintained Rails 5.1 introduces ActionDispatch::SystemTestCase which should allow our feature tests to be speed up when this is done: rspec/rspec-rails#1813

moveson · 2017-10-19T08:14:41Z

I'm having trouble with selenium-driven tests not seeing records in the database. Consider the following:

RSpec.describe 'User logs in' do
  let!(:user) { create(:user, email: email, password: password, password_confirmation: password) }
  let(:email) { 'jane@example.com' }
  let(:password) { '12345678' }

  scenario 'with valid email and password' do
    visit new_user_session_path
    fill_in 'Email', with: email
    fill_in 'Password', with: password
    click_button 'Sign in'
    expect(page).to have_content('You are signed in.')
  end
end

This test passes when RSpec.configure is set to driven_by :rack_test, but it results in "Invalid email or password" and test failure when driven_by is set to :selenium or any other selenium-based driver.

Using Rails 5.1.4, RSpec 3.7.0, selenium-webdriver 3.6 and capybara 2.15.4, and config.use_transactional_fixtures = true.

twalpole · 2017-10-19T08:24:44Z

@moveson Prior to Rails 5.1 that would have been caused by running transactional testing, and not having database_cleaner installed. In Rails 5.1 however the DB connection is shared between threads in testing mode, thereby allowing transaction mode (and obviating the need for database_cleaner in most cases). Since you're using 5.1 your issue would tend to imply whatever server you're using with Capybara isn't running in process (so the DB connection can be shared). What server are you using? If puma, check the output produced for something like "clustered mode" and fix it to run in single mode.

moveson · 2017-10-19T08:45:09Z

@twalpole I'm using puma in dev and production. I don't see the familiar puma output on startup in my RSpec log. How can I tell what server is running in the test environment? In any case, to be safe, I changed my config/puma.rb to workers 1 and threads 1,1, but it did not fix the problem.

twalpole · 2017-10-19T08:53:48Z

@moveson You actually need workers 0 to ensure in process mode - there's a PR that was accepted into the next Rails 5.1 release to try and ensure it gets set that way - rails/rails@bc02a01 . You should be able to get the Puma output back by setting ActionDispatch::SystemTesting::Server.silence_puma = false - Unfortunately it was chosen to make silence the default in rspec-rails (unlike Rails), rather than letting experienced users opt into silence, which hides the issue with puma.

moveson · 2017-10-19T09:13:49Z

@twalpole Setting workers to 0 and threads to 0,1 did the trick. Thank you! Do you happen to know an elegant way to set these values in the test environment without affecting my settings in dev/prod?

twalpole · 2017-10-19T09:18:08Z

@moveson - You could monkey patch in the rails change I linked above (or run from the 5-1-stable branch), or configure a different config file for puma --- Puma reads it's config from a number of possible places, try config/puma/test.rb IIRC

moveson · 2017-10-19T09:30:31Z

Got it. Again, thanks!

aptituz · 2017-10-21T11:05:40Z

I'm currently having some trouble get this running with a dockerized application (while the same application works fine without docker involved).

In the docker case the tests are driven_by a selenium remote driver, with a chrome standalone docker container. So far the interaction between the two containers (one with the web application, one with the browser) works (as I can tell from the created screenshots), except for stubbing of authentication (the session is just not logged in via login_as).

I already tried adjusting puma settings in the docker container (worked fine for the non-docker-case), but the symptoms of the problem are quiet different (with more then one threads the authentication problem occurs in a number of cases only, in this case it fails on every try).

Anybody an idea? Could it be that the in other places suggested disabling of self.use_transactional_fixturesis still necessary in my use case and that I would need database cleaner?

twalpole · 2017-10-21T15:45:35Z

@aptituz You don't fully define your use case, nor what login_as you're referring to. The Warden provided login_as (Devise), Rails 5.1 DB connection sharing (transactional testing, no need for database_cleaner), and Capybara's request completion detection all depend on the tests and the application under test running in the same process (multiple threads, but 1 process).

If you're referring to that login_as and are trying to run your AUT in a docker instance but run the tests on your local host - then you lose all that, will have to log in manually by filling in the fields everytime (or write your own login shortcut that depends only on parameters passed in the request), will need to use database_cleaner, and will need to be very careful about any extra requests from your app which your tests don't specifically wait to finish.

If, instead, you are running the tests on the docker instance and letting Capybara start the AUT on the docker instance all in one process (but the browser is in a different docker instance) then you should be fine, and probably just have a configuration issue.

aptituz · 2017-10-21T16:18:38Z

@twalpole You are right. Thanks that you ask for clarification.

I'm referring to the Warden provided login_as method and only the browser is in a different docker instance. The application itself and the tests are in the same container, but ... well ... it could be that the tests ran against a previously started instance, I'll have to check that. According to your explanations this could be the culprit, so thanks: these hints are helpful to me.

aptituz · 2017-10-23T12:21:23Z

@twalpole

I just wanted to drop a note that I got it running eventually. So, thanks again for your help.

For some reasons I don't remember I had a Capybara. run_server = false line around. So yeah, the tests were indeed run again a different, accidentally running container. After removing that setting it was basically just a host! "http://#{ip}:#{Capybara.server_port}" in a before(:each) to let capybara actually use the AUT containers external address for the tests instead of 127.0.0.1 as per default.

Xosmond · 2017-11-23T19:57:54Z

I have a problem with system tests:

RSpec.describe "Authentification", type: :system, js: true do
  let(:valid_attributes) { attributes_for(:user) }
  before :each do
    @user = User.create!(valid_attributes)
    visit '/users/login'
  end
    it "Enable second factor auth" do
      sign_in_with(valid_attributes[:email], valid_attributes[:password])
      @user.update! direct_otp: "111111" # on js: true, the user is not updated on the browser database
      activate_second_factor(@user.role.name, @user.direct_otp)
      expect(page).to have_content 'Se ha activado correctamente el segundo factor de autentificación.'
    end
end

This example only works when javascript is disabled, when I add js: true to the test, I get an error because the direct_otp field its not correct because user is not updated on the browser database.

Any suggestion on why this is happening?
Because user is created via code and I can login but late, on the middle of the test, when I update the user, the changes are not shared.

Rails 5.1.4
Rspec-rails 3.7.2

twalpole · 2017-11-23T20:46:39Z

@Xosmond Assuming you're using Rails 5.1+, if the app code isn't seeing DB changes made in the test code it's usually that you have the app running in a separate process rather than separate thread. Turn off the silencing of puma that rspec-rails does by default (it's a bad default option for beginners trying to troubleshoot) and see if the output puma then generates tells you it's running in 'single mode' (as opposed to cluster mode). If it doesn't you need to check where/how your puma is getting it's config from and make sure it's set to run with 0 processes and some number >= 1 threads.

Xosmond · 2017-11-23T22:14:07Z

@twalpole I have turn off the silencing and I got:

Puma starting in single mode...
* Version 3.11.0 (ruby 2.3.5-p376), codename: Love Song
* Min threads: 0, max threads: 1
* Environment: test
* Listening on tcp://0.0.0.0:9887
Use Ctrl-C to stop

Thats why I can login with the user created via code, but the app is not seeing the later change (user.update!) done by the test code on the it macro.

But if I do that change on the before macro , it works:

RSpec.describe "Authentification", type: :system, js: true do
  let(:valid_attributes) { attributes_for(:user) }
  before :each do
    @user = User.create!(valid_attributes)
    @user.update! direct_otp: "111111"
    visit '/users/login'
  end
    it "Enable second factor auth" do
      sign_in_with(valid_attributes[:email], valid_attributes[:password])
      activate_second_factor(@user.role.name, @user.direct_otp)
      expect(page).to have_content 'Se ha activado correctamente el segundo factor de autentificación.'
    end
end

Something happens after the before macro.

twalpole · 2017-11-23T22:44:44Z

@Xosmond Ok, since it is running in single mode it's probably a timing issue with the way your tests are written (or possible you aren't actually running in transaction mode). This really isn't the right place for that discussion. You should post over on stack overflow, add a capybara tag to the question and provide details of exactly what your sign_in_with and activate_second_factor methods do.

Xosmond · 2017-11-24T00:21:42Z

@twalpole I'm using config.use_transactional_fixtures = true and I have tried to sleep 5 it and nothing works. Oh I will. Same, thanks.

Poltergeist is no longer maintained Rails 5.1 introduces ActionDispatch::SystemTestCase which should allow our feature tests to be speed up when this is done: rspec/rspec-rails#1813

fables-tales mentioned this pull request May 5, 2017

Rails 5.1 capybara integration #1808

Closed

dfaulken mentioned this pull request May 17, 2017

Move system tests over to RSpec umts/incidents#6

Closed

fables-tales force-pushed the integrate-with-system-test branch from 38d951e to 3bc329d Compare June 16, 2017 20:11

fables-tales force-pushed the integrate-with-system-test branch 4 times, most recently from 97eb91f to 6b68f6a Compare June 17, 2017 18:30

jcoyne mentioned this pull request Jun 20, 2017

Switch from Poltergeist to headless Chrome samvera/hyku#1255

Merged

JonRowe approved these changes Jun 23, 2017

View reviewed changes

fables-tales mentioned this pull request Jun 24, 2017

Add an option to silence puma in system tests. rails/rails#29561

Merged

JonRowe approved these changes Jun 24, 2017

View reviewed changes

abevoelker reviewed Jun 25, 2017

View reviewed changes

astronaut-wannabe mentioned this pull request Jul 4, 2017

Add config for capybara and feature specs crimethinc/website#378

Merged

This was referenced Oct 25, 2017

[Question] Skip ActionDispatch::SystemTest integration possible? #1895

Closed

Cannot kill Rspec when running feature tests. #1887

Closed

Running feature specs hang forever #1894

Closed

maurogeorge mentioned this pull request Oct 26, 2017

Remove the test folder yogodoshi/cansei#25

Open

This was referenced Mar 20, 2018

Bump rspec-rails, rails, fastly-rails and dotenv-rails rubytogether/rubytogether.org#178

Merged

Bump rspec-rails from 3.6.0 to 3.6.1 tnir/gitlabhq#102

Closed

Bump rspec-rails from 3.6.1 to 3.7.2 sumyapp/basecamp3_slack_notify#26

Merged

sebjacobs pushed a commit to futurelearn/rspec-rails that referenced this pull request Mar 15, 2019

Integrate with ActionDispatch::SystemTest (rspec#1813)

87f31e7

sebjacobs pushed a commit to futurelearn/rspec-rails that referenced this pull request Mar 15, 2019

Add a changelog entry for rspec#1813

3ff7cd5

rmacklin mentioned this pull request Sep 22, 2019

Issues with take_failed_screenshot after PR 36047 rails/rails#37255

Closed

		the default `driven_by(:selenium)` from Rails. If you want to override this
		behaviour you can call `driven_by` manually in a test.

Integrate with ActionDispatch::SystemTest #1813

Integrate with ActionDispatch::SystemTest #1813

Conversation

fables-tales commented May 5, 2017 • edited Loading

elsurudo commented May 11, 2017

JonRowe commented May 11, 2017

fables-tales commented Jun 16, 2017

fables-tales commented Jun 17, 2017

derekprior commented Jun 18, 2017 • edited Loading

Results

clupprich commented Jun 22, 2017

JonRowe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JonRowe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moveson commented Oct 19, 2017 • edited Loading

twalpole commented Oct 19, 2017 • edited Loading

moveson commented Oct 19, 2017

twalpole commented Oct 19, 2017 • edited Loading

moveson commented Oct 19, 2017

twalpole commented Oct 19, 2017 • edited Loading

moveson commented Oct 19, 2017

aptituz commented Oct 21, 2017

twalpole commented Oct 21, 2017 • edited Loading

aptituz commented Oct 21, 2017

aptituz commented Oct 23, 2017

Xosmond commented Nov 23, 2017 • edited Loading

twalpole commented Nov 23, 2017 • edited Loading

Xosmond commented Nov 23, 2017 • edited Loading

twalpole commented Nov 23, 2017

Xosmond commented Nov 24, 2017

fables-tales commented May 5, 2017 •

edited

Loading

derekprior commented Jun 18, 2017 •

edited

Loading

moveson commented Oct 19, 2017 •

edited

Loading

twalpole commented Oct 19, 2017 •

edited

Loading

twalpole commented Oct 19, 2017 •

edited

Loading

twalpole commented Oct 19, 2017 •

edited

Loading

twalpole commented Oct 21, 2017 •

edited

Loading

Xosmond commented Nov 23, 2017 •

edited

Loading

twalpole commented Nov 23, 2017 •

edited

Loading

Xosmond commented Nov 23, 2017 •

edited

Loading