WIP: Capybara Integration with Rails (AKA System Tests) #26703

Open
wants to merge 30 commits into
from
@eileencodes
Member
eileencodes commented Oct 4, 2016 edited

I'm really excited to open this initial PR for bringing system/acceptance test support to Rails through Capybara.

The goal of this PR is for Rails to take on all the setup that is needed in applications to allow Capybara and make system testing default in Rails applications.

Why is the name Rails::SystemTestCase?
I chose this name because this was the name DHH indicated he preferred in the Rails' Basecamp long ago. I'm totally open to changing the name but ultimately decided to put it in ActionPack with the Rails namespace so I could get onto writing code and stop worrying about the name 😉

The test framework has been moved it it's own gem under the Rails name and is now ActionSystemTest
Because the gem expects a Module and our test frameworks expect a Class for inheritance I've made the gem and module name ActionSystemTest and the class for inheriting test code from ActionSystemTestCase. Open to other names as well but the Module and Class names can't be the same else Ruby gets confused.

How do I add System tests to my application?
By default new applications include a base system test much like integration tests. Gemfiles generated for new applications will include Capybara and the Selenium driver.

Capybara's selenium driver requires some setup by the programmer, so I've added a layer between Capybara and Rails so that Rails' takes on that work called RailsSeleniumDriver. The default settings are as follows:

  • The server is puma (because this is Rails' default)
  • the browser is chrome (because FF is broken w/ Capybara and Selenium right now),
  • the default port is 28100, and
  • the default screen size is [1400, 1400].

The reason I have chosen Chrome as the default browser is because Firefox doesn't work out of the box. Selenium and the current version of Firefox don't play nicely together so I've set up the driver to default to Chrome, but it can easily be switched to using Firefox. I'd also like to provide support for Safari.

Instead of requiring the programmer to set up the port, browser, server, etc Rails handles that setup and frees up the programmer to work on writing their tests.

The Rails drivers allow the default configuration to be changed by initializing a new driver:

# config/environments/test.rb
config.system_testing.driver = SystemTesting::DriverAdapters::RailsSeleniumDriver.new(
  browser: :firefox
)

What if I don't want to use Selenium?
If the programmer wants to use one of Capybara's default drivers rather than the provided defaults in Rails Selenium configuration they can easily change that in the test environment. I've provided a shim in Rails so setting the Capybara driver is simple. I've named the class CapybaraDriver and it gives access to the 4 Capybara drivers: poltergeist, webrick, selenium (with no setup), and rack_test:

# config/environments/test.rb
config.system_testing.driver = :poltergeist

Each of the Capybara default drivers that requires a server defaults to Puma. Any of the settings can be changed by initializing a new CapbyaraDriver object:

# config/environments/test.rb
config.system_testing.driver = SystemTesting::DriverAdapters::CapybaraDriver.new(
  name: :poltergeist,
  server: :webrick
)

What if I'm making a new application and don't want system testing at all?
It's easy to skip system testing files in the app generator or the scaffold generator by running with the --skip-system-test flag.

What if I already use Capybara and don't want to use Rails?
Because Rails provides a specific test case name to inherit from you can easily just completely skip the Rails version of system testing and use Capybara directly.

I want to try it out!
I made a test app where you can try out system tests in Rails w/ Selenium! https://github.com/eileencodes/system_testing_app


Cool! What's next?

Below is a list of what is done and what's left to do. Once we sort out naming and other issues I'll finish up the items below.

What's done?

  • Generators: Scaffold, individual, and application
  • Basic Documentation
  • Testing adapter settings (this just tests that Rails provides options included, not that Capybara and friends work. The assumption is that actual behavior of Capybara and friends is tested by themselves, Rails tests it's framework for initializing Capybara)
  • Configurable driver adapters w/ RailsSeleniumDriver being the default.
  • Railtie for configuring system test settings
  • Support for screenshots
  • Changelog
  • Fix the railties tests 😁
  • Move to it's own gem under the rails name.
  • Rails Guides
  • Other custom but generic helpers we support at Basecamp that we'd like to port over, especially ActiveJob and ActionCable support
  • Testing the custom helpers and assertions

What's in progress?

  • Transactions aren't handled in Capybara the same as they are in Rails' other test so we need to handle that. Capybara suggests using the DatabaseCleaner gem but we only need a small portion of that code. I'm working on porting that over.

Future work

  • Support for Safari (last I checked this wasn't quite ready for prime time)

cc/ @dhh @georgeclaghorn

@eileencodes eileencodes added this to the 5.1.0 milestone Oct 4, 2016
@eileencodes eileencodes self-assigned this Oct 4, 2016
@prathamesh-sonpatki
Member

Transactions aren't handled in Capybara the same as they are in Rails' other test so we need to handle that. Capybara suggests using the DatabaseCleaner gem but we only need a small portion of that code.

@eileencodes We should also look for https://github.com/amatsuda/database_rewinder as an alternative to DatabaseCleaner.

@shakycode

This is excellent!

@alexcameron89

Hey @eileencodes, I ran through the documentation and commented on some grammar issues.

This PR looks great, and I'm excited for it to be a part of the default Test Suite!

actionpack/lib/system_test_case.rb
+
+module Rails
+ # System tests are similar to Integration tests in that they incorporate multiple
+ # controllers and actions, but can be used to similate a real user experience.
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

similate -> simulate

+ #
+ # By default Rails supports Capybara with the Selenium Driver. Rails provides
+ # configuration setup for using the selenium driver with Capybara.
+ # Additionally Rails can be used as a layer between Capybara and it's other
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

it's -> its

+ # The drivers Capybara supports are: +:rack_test+, +:selenium+, +:webkit+,
+ # and +:poltergeist+.
+ #
+ # Rails provides it's own defaults for Capybara with the Selenium driver
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

it's -> its

+ # directly.
+ #
+ # To set your system tests to use one of Capybara's default drivers add
+ # the following to yur Rails' configuration test environment:
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

"to use one of Capybara's default drivers add" -> "to use one of Capybara's default drivers, add"

line 18: yur -> your

+ # JavaScript testing and doesn't require a server.
+ #
+ # The +:poltergeist+ and +:webkit+ drivers are headless, but require some
+ # extra environment setup. Because the defalt server for Rails is Puma, each
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

defalt -> default

+ # of the Capybara drivers will default to using Puma. Changing the configuration
+ # to use Webrick is possible by initalizing a new driver object.
+ #
+ # The default settings for the <tt>CapybaraDrvier</tt> are:
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

CapybaraDrvier -> CapybaraDriver

+ # selenium-webdriver gem is required by this driver.
+ #
+ # The <tt>RailsSeleniumDriver</tt> is useful for real browser testing and
+ # support Chrome and Firefox.
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

support -> supports

+ # @screen_size=[ 1400, 1400 ]
+ # >
+ #
+ # The settings for the <tt>RailsSeleniumDriver</tt> can be changed in the
@alexcameron89
alexcameron89 Oct 4, 2016 Contributor

"in the Rails'" -> "in the Rails" OR "in Rail's"

@pixeltrix
Member

I assume that this follow's Capybara's opinionated policy of making it hard to access request/response objects and that we should use integration tests for those?

actionpack/lib/system_test_case.rb
+require "system_testing/test_helper"
+require "system_testing/driver_adapter"
+
+module Rails
@maclover7
maclover7 Oct 4, 2016 Member

If we're shipping this under the actionpack gem, should we use the ActionPack base module? 😬

@eileencodes
eileencodes Oct 4, 2016 edited Member

I explained why I chose this name in the message above and I don't want to move it around 100 times before we have a consensus on name.

@dhh
dhh Oct 4, 2016 Member

I actually don't see any dependencies for ActionPack in this patch, so I think we could consider making it a complete stand-alone gem.

@eileencodes
eileencodes Oct 4, 2016 Member

@dhh Rails::SystemTestCase inherits from ActionDispatch::IntegrationTest so we don't have to rewrite all the routing and url helpers handling.

@dhh
dhh Oct 4, 2016 Member

Ah, I missed that. Then going with ActionDispatch::SystemTest probably makes sense. But agree no sense in making any changes until we really know if we love that name!

@rafaelfranca
rafaelfranca Oct 29, 2016 Member

I like ActionDispatch::SystemTest

+ # Rails' configuration file.
+ def driver=(driver)
+ @driver = DriverAdapters.lookup(driver)
+ @driver.call
@maclover7
maclover7 Oct 4, 2016 Member

Should we be more explicit about what call does, in regard to setting up / registering the adapter? 😬

@kaspth
kaspth Oct 6, 2016 Member

Calling your personal @driver gets you a smooth getaway 😎

+ # | Selenium | Firefox | Yes |
+ # | Webkit | Headless w/ QtWebKit | Yes |
+ # | Poltergeist | Headless w/ PhantomJS | Yes |
+ module DriverAdapters
@maclover7
maclover7 Oct 4, 2016 Member

Does it make sense to provide a "base" class to inherit from, so people know which methods they need to implement when creating adapters? 😬

+ end
+
+ def supports_screenshots?
+ if @name == :rack_test
@fnando
fnando Oct 4, 2016 Contributor

Don't use a conditional to return a boolean.

def supports_screenshots?
  @name == :rack_test
end
@jonathanhefner
jonathanhefner Oct 5, 2016

I agree, but it would be @name != :rack_test. (However, it may have been written as is with the intention that more cases / logic would be added.)

@sgrif
Member
sgrif commented Oct 4, 2016

Is there a specific reason for defaulting to selenium instead of a headless driver such as poltergeist or capybara-webkit?

@dixpac
Contributor
dixpac commented Oct 5, 2016

I agree with @sgrif, IMHO in real world apps people will switch to headless driver right away, so maybe capybara-webkit is better default here 😄

@eileencodes
Member
eileencodes commented Oct 5, 2016 edited

@sgrif and @dixpac I chose the Selenium driver as default because the purpose of adding system testing to Rails is for the initial setup to be absolutely zero. Capybara does that through their default, Rack Test, but it's not really a useful demonstration of the merits of system testing since it doesn't support JavaScript testing.

The selenium driver is one that Rails can take on all of the setup without any extra requirements from the programmer. capybara-webkit requires environmental setup that Rails can't take on - installing the Qt libraries. poltergeist requires PhantomJS which does have a gem but I don't know much about whether it's stable. We use Selenium at Basecamp and for now I'm going to leave the default as-is.

If there's a compelling reason to change the default it's as simple as swapping out the pre-set default and ensuring that Rails takes on the setup for that.


@pixeltrix Since this PR doesn't change how Capybara works, but rather provides a layer between Capybara and Rails.Currently, no, the request object isn't defined, but Rails::SystemTestCase does inherit from ActionDispatch::IntegrationTest so perhaps that's something we can easily expose in the future 😄

@kaspth

Looks sweet! Especially looking forward to the DatabaseCleaner work around 😁

actionpack/lib/system_test_case.rb
+ # click_on 'New User'
+ #
+ # fill_in 'Name', with: 'Arya'
+ # click_on 'Create User'
@kaspth
kaspth Oct 6, 2016 Member

Should we do anything to make the Capybara helpers work better with Rails' integration helpers?

I haven't used Capybara much, but from what I can remember you can't call visit and then use assert_response :success for instance.

Pointing this out because the system test inherits from ActionDispatch::IntegrationTest so users might think our built in assertions would work.

@eileencodes
eileencodes Oct 14, 2016 Member

I think that's a great idea for the future, but right now there's still a ton left to do on this PR and I'm not exactly sure how to expose it just yet. I agree it would be great to add that in once we have this PR in master.

@kaspth
kaspth Oct 28, 2016 Member

Great, let's revisit some other time 👍

actionpack/lib/system_test_case.rb
+ # fill_in 'Name', with: 'Arya'
+ # click_on 'Create User'
+ #
+ # assert_text 'Arya'
@kaspth
kaspth Oct 6, 2016 edited Member

woof! 🐕

+ # Rails' configuration file.
+ def driver=(driver)
+ @driver = DriverAdapters.lookup(driver)
+ @driver.call
@kaspth
kaspth Oct 6, 2016 Member

Calling your personal @driver gets you a smooth getaway 😎

+ # | Rails' Selenium | Chrome | Yes |
+ # | Rack Test | No JS Support | No |
+ # | Selenium | Firefox | Yes |
+ # | Webkit | Headless w/ QtWebKit | Yes |
@kaspth
kaspth Oct 6, 2016 Member

I believe it's spelled WebKit.

+ #
+ # config.system_testing.driver = SystemTesting::DriverAdapters::CapybaraDriver.new(
+ # name: :webkit,
+ # puma: :webrick
@kaspth
kaspth Oct 6, 2016 Member

server: :webrick

+ # The <tt>RailsSeleniumDriver</tt> is useful for real browser testing and
+ # support Chrome and Firefox.
+ #
+ # By default Rails system testing will use the Rails' defaults with Capybara
@kaspth
kaspth Oct 6, 2016 Member

✂️ the in the Rails'

@kaspth
kaspth Oct 6, 2016 Member

Also find it a bit odd we say defaults twice here.

+
+ attr_reader :browser, :server, :port, :screen_size
+
+ def initialize(browser: :chrome, server: :puma, port: 28100, screen_size: [1400,1400]) # :nodoc:
@kaspth
kaspth Oct 6, 2016 Member

[ 1400, 1400 ]?

+ end
+
+ def register_webrick(app, port, host)
+ Rack::Handler::WEBrick.run(app, Host: host, Port: port, AccessLog: [], Logger: WEBrick::Log::new(nil, 0))
@kaspth
kaspth Oct 6, 2016 Member

WEBRick::Log.new?

@eileencodes
eileencodes Oct 14, 2016 Member

Whoops, that was left over from copying code from Capybara. Removed 👍

actionpack/lib/system_testing/railtie.rb
+require "system_test_case"
+
+module SystemTesting
+ # System Testing Railtie
@kaspth
kaspth Oct 6, 2016 Member

This comment doesn't really add much.

@eileencodes
eileencodes Oct 7, 2016 Member

This is how all the other Railtie files document their railtie class. AJ for example: https://github.com/rails/rails/blob/master/activejob/lib/active_job/railtie.rb#L5. I did forget to add the = System Testing Railtie but I've fixed that.

+ # Asserts that all of the provided selectors are present on the given page.
+ #
+ # assert_all_of_selectors('p', 'td')
+ def assert_all_of_selectors(*items)
@kaspth
kaspth Oct 6, 2016 Member

Ditto re double splat.

+ #
+ # assert_none_of_selectors('ul', 'ol')
+ def assert_none_of_selectors(*items)
+ options = items.extract_options!
@kaspth
kaspth Oct 6, 2016 Member

I think we can use a double splat here:

def assert_none_of_selectors(*items, **options)
  # ...
end
+ private
+ def type_for_selector(*items)
+ if items.first.is_a?(Symbol)
+ items.shift
@kaspth
kaspth Oct 6, 2016 edited Member

Why do we want to shift this arg? What kind of arguments can we see in this method?

+ def image_path
+ path = "tmp/screenshots/failures_#{method_name}.png"
+ page.save_screenshot(Rails.root.join(path))
+ path
@kaspth
kaspth Oct 6, 2016 edited Member

Find it a little odd that image_path both returns a path and saves an image. I mean, wouldn't the two calls to image_path in find_image save two more images?

+ assert_equal true, Rails::SystemTestCase.driver.supports_screenshots?
+
+ Rails::SystemTestCase.driver = :poltergeist
+ assert_equal true, Rails::SystemTestCase.driver.supports_screenshots?
@kaspth
kaspth Oct 6, 2016 Member

Why do we want this to return booleans over our generally preferred falsey values?

@eileencodes
eileencodes Oct 15, 2016 Member

This was an oversight, I've fixed this now.

+ # name: :webkit,
+ # server: :webrick
+ # )
+ class CapybaraDriver
@maclover7
maclover7 Oct 15, 2016 Member

[sorry, just reposting this comment since it got accidentally marked as "outdated" thanks to the great work being done here :)]

Does it make sense to provide a "base" class to inherit from, so people know which methods they need to implement when creating adapters? 😬

@eileencodes
eileencodes Oct 15, 2016 Member

Hey @maclover7 thanks. I know there are a lot of comments to address and getting around to them as I can. In the future, no need to re-comment.

I don't think adding a base class makes sense becasue the Rails provided adapter takes different arguments than the capybara provided adapters. It might make sense as this project evolves but I'll tackle that when that happens.

@@ -0,0 +1,31 @@
+require "system_testing/driver_adapters"
+
+module SystemTesting
@rafaelfranca
rafaelfranca Oct 29, 2016 Member

Should not this class be in one namespacing?

@rafaelfranca

It feels to me that this should be a new gem. It have its own namespace, its own railtie it can be skipped and I don't think we should add capybara as dependency of actionpack. I'd create a new gem, inside the rails repository and add it as dependency of Rails.

+ @port = port
+ end
+
+ def call
@rafaelfranca
rafaelfranca Oct 29, 2016 Member

Maybe call it start?

+ # config.system_testing.driver = SystemTesting::DriverAdapters::RailsSeleniumDriver.new(
+ # server: :webrick,
+ # port: 28100,
+ # screen_size: [ 800, 800 ]
@rafaelfranca
rafaelfranca Oct 29, 2016 Member

Our style guide doesn't put spaces inside array literals

@rafaelfranca
rafaelfranca Oct 29, 2016 Member

Actually it is the other way around. I'll make rubocop point that.

@rafaelfranca
rafaelfranca Oct 29, 2016 Member

There is no support for that in rubocop 😢 .

+
+ attr_reader :browser, :server, :port, :screen_size
+
+ def initialize(browser: :chrome, server: :puma, port: 28100, screen_size: [ 1400,1400 ]) # :nodoc:
@rafaelfranca
rafaelfranca Oct 29, 2016 Member

missing space after the , and remove the space inside the array literal

actionpack/lib/system_testing/railtie.rb
+ options.driver ||= Rails::SystemTestCase.default_driver
+
+ ActiveSupport.on_load(:system_testing) do
+ options.each { |k,v| send("#{k}=", v) }
@rafaelfranca
rafaelfranca Oct 29, 2016 Member

space after the ,

@eileencodes
Member

@rafaelfranca David and I talked about the naming etc this week and decided on creating a new gem inside Rails, so I'll be doing that soon. Thanks for the feedback!

+
+ s.license = "MIT"
+
+ s.author = ["Eileen Uchitell", "David Heinemeier Hansson"]
@kaspth
kaspth Nov 11, 2016 Member

Personally I'd throw an e on there, but I'll let you be the judge of how to spell your name 😄

@eileencodes
eileencodes Nov 11, 2016 Member

I noticed that, still working on a bunch of stuff. I'll let you know when I'm ready for more review.

@kaspth
kaspth Nov 11, 2016 Member

Sounds good 👍

+ # === Driver Features
+ #
+ # | | Default Browser | Supports Screenshots? |
+ # |-----------------|-----------------------|-----------------------|
@kaspth
kaspth Nov 11, 2016 Member

Nitpick: saw another table where we put spaces between the rows and pipes, e.g.: ---- | ----. Not sure if we have a preferred style.

@@ -0,0 +1,15 @@
+module ActionSystemTest
+ # Returns the version of the currently loaded Action Cable as a <tt>Gem::Version</tt>.
@kaspth
kaspth Nov 11, 2016 Member

Let's be cable cutters: Action System Test, I presume 😁

+require_relative "gem_version"
+
+module ActionSystemTest
+ # Returns the version of the currently loaded Action Cable as a <tt>Gem::Version</tt>
@kaspth
kaspth Nov 11, 2016 Member

Cable :)

railties/test/application/test_test.rb
puts "before: " + has_user_table.to_s
end
task :after_hook do
- has_user_table = ActiveRecord::Base.connection.table_exists?('users')
+ has_user_table = ActiveRecord::Base.connection.data_source_exists?('users')
@kaspth
kaspth Nov 11, 2016 Member

How is this change related to system testing?

+ # click_checkbox_label 'Admin', checked: true
+ def click_checkbox_label(name, checked: false)
+ field = find_checkbox(name, checked)
+ label = find_label_wrapper(field)
@twalpole
twalpole Nov 12, 2016 edited Contributor

Capybara provides a selector - https://github.com/jnicklas/capybara/blob/master/lib/capybara/selector.rb#L432 - for the label that wraps an input or where the id/for attributes match. so this could be find(:label, for: field)

+ end
+
+ private
+ def find_checkbox(name, checked)
@twalpole
twalpole Nov 12, 2016 Contributor

Capybara provides a :checkbox selector - https://github.com/jnicklas/capybara/blob/master/lib/capybara/selector.rb#L298 - which would limit this to only checkboxes if thats whats desired

@kaspth kaspth referenced this pull request Nov 22, 2016
Open

ActionCable testing #23211

+ take_failed_screenshot
+ Capybara.reset_sessions!
+ end
+end
@eileencodes
eileencodes Dec 15, 2016 Member

This file is generated when a new app is generated or a scaffold is generated and there is no system_test_helper file.

The reason for this file is there is a ton of other setup that Capybara can do and programmers need a place to put that code. Additionally some may not want to take_failed_screenshots or do other such work. This gives them a clear place to put anything and everything related to system tests or required for setup/teardown on every test.

@kaspth
kaspth Dec 15, 2016 Member

Great!

Should this call the Active Job helpers like setup :set_queue_adapter_to_async?

@eileencodes
eileencodes Dec 15, 2016 Member

I was torn about that. I'm thinking - since AJ is included by default and created by default maybe it should be included by default in the generated file. At first I was thinking that you may not want to use AJ, but if it's auto generated then it should probably be included.

@kaspth
kaspth Dec 20, 2016 Member

It does sound like baking it in is a good default, then people can just remove it themselves if it doesn't suit their needs.

+
+ s.license = "MIT"
+
+ s.author = ["Eileen Uchitelle", "David Heinemeier Hansson"]
@kaspth
kaspth Dec 15, 2016 Member

❤️

+ take_failed_screenshot
+ Capybara.reset_sessions!
+ end
+end
@kaspth
kaspth Dec 15, 2016 Member

Great!

Should this call the Active Job helpers like setup :set_queue_adapter_to_async?

eileencodes added some commits Aug 5, 2016
@eileencodes eileencodes Add generators and ability to run system tests
* Generates system test requirements with new Rails app
* Includes required default gems in Gemfile for Rails app
* Generates a single system test case
* Generates a system test case with scaffold
d6deb5c
@eileencodes eileencodes Add skeleton for Rails::SystemTestCase
This skelton is the bare minimum to get system tests to actually run in
an application. This of course doesn't yet actually run a test but it is
enough for `bin/rails test:system` to attempt to run files in
`test/system` that inherit from `Rails::SystemTestCase`.
8e8fdd0
@eileencodes eileencodes Add ability to run system tests via Capybara
Capybara defaults to Rack Test for it's driver and works out of the box
but this adds the headers and allows for future configurable adapters
for system testing.
4aa0667
@eileencodes eileencodes Add configurable selenium driver for capybara
This is not yet configurable but is the minimum required to make
Capybara work with the Selenium driver. A lot of this will change as the
tests get fleshed out and the initialization requirements will eventually
be configurable via the application.
a19dc19
@eileencodes eileencodes Add ForHelper's for system tests
These FormHelpers are selectors that aren't a capybara default but are
considered useful for Rails applications.
4c02944
@eileencodes eileencodes Add configuration option for driver adapter
This allows any application to change the driver adapter based on the
config settings in the test env.
46c1477
@eileencodes eileencodes Add test assertion helpers
Adds assertions that are not part of Capybara but may be useful to Rails
users writing system tests.
8808afd
@eileencodes eileencodes Inherit from ActionDispatch::IntegrationTest
Integration tests already handle all the fancy url mapping we need to do
so inherting from that allows us to not need to reinvent the wheel in
terms of loading up the route handling required to use `visit
users_path` over `visit /users`.
ae9d99b
@eileencodes eileencodes Refactor driver adapter getter/setter
This makes it easier to ask the system test what driver adapter it is
currently using, and makes it easier to change that setting when
necessary.
56f482f
@eileencodes eileencodes Refactor to not include `capybara/rails`
Rails itself is not a Rails application so instead of including
`capybara/rails` we should use the code in there to set up the test. The
only reason capybara needs to include capybara/rails in the first place
is because Rails didn't yet support it.
1e43a41
@eileencodes eileencodes Fix Railtie to pass class when setting options
This will clean up the railtie quite a bit, rather than passing a set of
hash keys call the new class directly like we do with ActiveJob.

Only call driver once when tests start rather than in every single test
setup. This is more performant, and the other way was creating
unnecessary calls.
fefd360
@eileencodes eileencodes Move SystemTesting::Base into SystemTestCase
There's no real benefit to the using the `Base` class here because
`SystemTestCase` is already a very small class.
f6657c3
@eileencodes eileencodes Add tests for system testing
* Adds test case test
* Adds driver adapter test
* Adds tests for capybara seleium driver (testing the settings not
actually opening the browser to test capybara w/ selenium because that
would so so so slow)
* Adds tests for rack test driver
* Adds tests for generators
24f2f58
@eileencodes eileencodes Add documentation for system tests
* Document Rails::SystemTestCase
* Document setting drivers with the configration options
* Document using the getter/setter for driver adapters
* Document the CapybaraRackTestDriver and defaults
* Document the CapybaraSeleniumDriver and defaults
* Document custom assertions provided by System Testing
* Document custom form helpers provided by System Testing
d3657cd
@eileencodes eileencodes Reconfigure how the drivers work
This removes the useless Rack Test Driver that Rails was providing and
moves to a shim like approach for default adapters.

If someone wants to use one of the default Capybara Drivers then we will
initialize a new `CapybaraDriver` that simply sets the default driver.

Rails though is much more opinionated than Capybara and to make system
testing a "works out of the box" framework in Rails we have the
`RailsSeleniumDriver`. This driver sets defaults that Rails deems
important for selenium testing. The purpose of this is to simply add a
test and it just works.
1e0882c
@eileencodes eileencodes Add support for screenshots
This change adds support, tests, and documentation for the screenshot
helper.

If taking screenshots is supported by the driver (for example Rack Test
doesn't support screenshots) then a screenshot will be taken if the test
fails.
5d37d6d
@eileencodes eileencodes Refactor so all drivers use Puma by default
Puma is the default webserver of Rails. Because of this it doesn't make
sense to run tests in Webkit if the default server is Puma.

Here I've refactored the webserver to be it's own standalone module so
it can be shared between Rails' selenium default driver and Capybara's
defaut drivers.
6da8e7e
@eileencodes eileencodes Appease Rubocop
Rubocop / code climate don't like single quotes and prefer doubles.
aa592fa

Fixed everything here!

eileencodes added some commits Nov 6, 2016
@eileencodes eileencodes Turn system testing into it's own gem and rename
Renames `Rails::SystemTestCase` to `ActionSystemTest` and moves it to a
gem under the Rails name.

We need to name the class `ActionSystemTestCase` because the gem expects
a module but tests themselves expect a class.

Adds MIT-LICENSE, CHANGELOG, and README for the future.
5fc06bd
@eileencodes eileencodes Don't load ActionSystemTest in production
By moving to the TestUnit Railtie, and doing the file requirement
inside the onload call we can avoid loading ActionSystemTest in
production and load it in the test env.

This is important for performance reasons - loading up unnecessary files
and object is expensive, especially when they should never be used in
production.
b29fd3c
@eileencodes eileencodes Refactor config settings to use generated file
Originally I had set up system testing to have one configuration option
to be set in the test environment. After thinking it over I think a
generated class on app creation would be best. The reason for this is
Capybara has a ton of configuration options that I'm sure some folks
want to use.

Thinking about how we handle screenshots, database transactions, and a
whole bunch of other settings it would be better for users to be able to
turn all of that on and off.

When an app or scaffold is generated a `test/system_test_helper.rb` test
helper will be generated as well. This will contain the class for tests
to inherit from `ActionSystemTestCase` which will inherit from
`ActionSystemTest::Base`. Here is where users can change the test
driver, remove the screenshot helper, and add their additional Capybara
configuration.
e5821ae
@eileencodes eileencodes Amend documentation
Many changes have been made since the beginning so documentation needed
a refresher.
ee9a73f
@eileencodes eileencodes Rename call to run
Call doesn't make as much sense here, we're really starting to run the
driver.
02d2105
@eileencodes eileencodes Set Webrick logger for system testing
If this is not set Webrick will log **everything** to STDOUT and
distract from the running tests. This will instead log to the log file.
This example was extracted from the Capybara source code.
1848b91
@eileencodes eileencodes Use 1 thread instead of 4 with Puma server for system tests 323e115
@eileencodes eileencodes Remove teardown code
Since I've moved the teardown code that contains the screenshot handling
to be generated when the application is generated this code was
interfering with the screenshot taking.

Because this runs before any app teardown code we would be resetting
sessions before taking the screenshot, resulting in a blank browser
window. The code to reset the sessions must come AFTER a screenshot has
been taken.
3efb705
@eileencodes eileencodes Cleanup Rails provided helpers
1. Update Rails provided assertions for selectors

This didn't quite work right and after debugging a bit I found that
there was an easier way to express what this was doing. Originally the
code was extracted from Basecamp 3's system tests.

These assertions assert that all of or none of the selectors on the page
match the provided items. It iterates through the provided items and
checks each one rather than requiring the test writer to assert each
selector manually in their test code.

This is a shortcut. Instead of writing:

```
assert_selector :avatar, 'Eileen'
assert_selector :avatar, 'Jeremy'
```

Users can now do

```
assert_all_of_selectors :avatar, 'Eileen', 'Jeremy'
```

2. Clean up screenshot helper

Updates documentation to be clearer and separates the concerns of saving
the image, setting the image path, and displaying the image.

3. Remove form helper

The form helper is pretty specific to Basecamp's needs and may not be
helpful outside of Rails.
bf4c399
@eileencodes eileencodes Add setup helper for Active Job for system tests
For system tests we want to set the queue to Async and then set it back
when we're done. These helper methods make it easy to change the adapter
with a single command in the system tests test helper generated with a
new app.
7542968
@eileencodes eileencodes Add guides for system testing
This adds the required guides for how to write and use system tests in
your application.
047c125
@eileencodes
Member

This is ready for review from @dhh. I've finished all the above work: added guides, got tests in Railties passing, rebased, cleaned up some commits, and moved everything to a new gem called ActionSystemTest.

ActionSystemTest has a Base class so that the test class can be ActionSystemTestCase which system tests will inherit from. The reason for this is because the gem name must be a module, but test's inherit from a class. ActionSystemTestCase, which all system tests inherit from is defined in the system_test_helper.rb generated with the application, scaffold or test.

This file is where anyone writing system tests would add additional Capybara configuration if desired, and where the setup/teardown requirements live. Specifically out of the box tests handle resetting Capybara sessions, taking screenshots of failed tests, and setting the ActiveJob queue adapter to async.

One note: I did not add DatabaseCleaner type code to Active Record. After discussing this with Aaron we found that Active Record was behaving incorrectly and not closing connections after opening them.

Fixing that makes DatabaseCleaner unnecessary because then Active Record would know how to rollback. I'm still working on finishing those changes up because there are some failures in AR, but I don't consider this a blocker for merging system tests because 1) this doesn't affect models that use fixtures and 2) database cleaner still works

+ include TestHelpers::ScreenshotHelper
+ include Capybara::DSL
+
+ Capybara.app = Rack::Builder.new do
@jnicklas
jnicklas Jan 2, 2017 Contributor

It's a bit confusing that this configuration, which is global configuration is set inside a module. It makes it look as though this config only applies to this module, when it in fact applies to everthing.

@dhh
dhh Jan 10, 2017 Member

Agree, but also, I think we should encapsulate this in a bootstrap method somewhere. It feels oddly free-floating in a helper to do this kind of global configuration. (I'm thinking something like register_browser_driver).

+ module TestHelpers
+ extend ActiveSupport::Autoload
+
+ autoload :ActiveJobSetup
@jnicklas
jnicklas Jan 2, 2017 Contributor

Isn't autoload generally discouraged? Maybe this was fixed in newer Ruby versions?

+ #
+ # assert_all_of_selectors(:avatar, 'Eileen', 'Jeremy')
+ # assert_all_of_selectors(:avatar, 'Eileen', 'Jeremy', visible: all)
+ def assert_all_of_selectors(selector, *items, **options)
@jnicklas
jnicklas Jan 2, 2017 Contributor

I think these helpers are quite useful, but I'd rather see them moved upstream to Capybara itself, otherwise we're creating a "Rails-dialect" of Capybara, which I think would be much harder for newer users to grasp.

@twalpole
twalpole Jan 2, 2017 edited Contributor

I've created a PR for adding these methods to Capybara - teamcapybara/capybara#1820 - the only real difference is that the :wait option will apply to the group as a whole rather than each individual selector check

+module ActionSystemTest
+ module TestHelpers
+ # Screenshot helper for system testing
+ module ScreenshotHelper
@jnicklas
jnicklas Jan 2, 2017 Contributor

I'm not really sure if the inclusion of this is warranted.

@dhh
dhh Jan 10, 2017 Member

I like it. I think it's exactly the kind of process improvement that makes the out-of-the-box configuration feels extra nice. We use those screenshots on every failure for BC3 to help diagnose.

@jnicklas
Contributor
jnicklas commented Jan 2, 2017 edited

Pinging @twalpole, the maintainer of Capybara, for his input on this as well.

I've had a look through this PR, and I have some comments:

I think it's not optimal to add a layer around Capybara drivers like this is doing. I understand that there is some desire from Rails' perspective to take ownership of the experience here, but this has a significant downside that we should be aware of: it requires every Capybara driver to also have a Rails-Capybara driver. It also just seems plain unnecessary to me, since Capybara already has a system for registering and managing drivers. If the desire is to make this look more Rails-native then simply building a thin shim around this system would be preferable, I think.

For example, supposing that Rails wants to change some defaults for the selenium driver, then Rails could just to something like this:

Capybara.register_driver :rails_selenium do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara.default_driver = :rails_selenium

And Rails could build a shim around Capybara.default_driver and Capybara.current_driver like this:

module ActionSystemTest
  def self.current_driver
    Capybara.current_driver
  end

  def self.current_driver=(name)
    Capybara.current_driver = name
  end
end

I think that maybe the purpose of this configuration is to be able to switch to something other than Capybara for these tests, but if that is the case, I'd like you to consider the following two issues: 1) Actually building an alternative driver would be a huge undertaking 2) This is precisely what Capybara already does and was in fact exactly what Capybara was created for, abstracting away the differences between multiple drivers transparently.

I think it'd be best to avoid adding a "Rails-dialect" of Capybara, where some methods come from Capybara, some from Rails, etc... This would make it much more difficult for users to discover where methods are coming from, and create confusion for people switching between ActionSystemTest and plain Capybara projects.

@jnicklas
Contributor
jnicklas commented Jan 2, 2017

Also, since my post was a bit critical I want to say thank you to @eileencodes for pushing forward on this and doing so much work to make this happen. Making it easier and more discoverable for Rails users to use Capybara/system/integration tests is really great.

@eileencodes
Member
@dhh dhh Merge branch 'master' into rails_system_tests
afb2bd1
@dhh

First pass of a review.

@@ -0,0 +1,67 @@
+# System tests are similar to Integration tests in that they incorporate multiple
+# controllers and actions, but can be used to simulate a real user experience.
@dhh
dhh Jan 10, 2017 Member

Let's get clearer on the value. System tests let you test JavaScript applications because they use a real browser as the testing agent. That's the difference and motivation. If you don't need to test JavaScript, don't bother with system tests, use integration tests.

@@ -0,0 +1,67 @@
+# System tests are similar to Integration tests in that they incorporate multiple
+# controllers and actions, but can be used to simulate a real user experience.
+# System tests are also known as Acceptance tests.
@dhh
dhh Jan 10, 2017 Member

Wouldn't confuse our vocabulary with talking about acceptance tests anywhere. There are many other names for system tests (including, confusingly enough, integration tests). Let's put forward a strong, consistent vocabulary.

+# <tt>ActionSystemTestCase</tt>. System tests use Capybara as a base and
+# allow you to configure the driver. The default driver is
+# <tt>RailsSeleniumDriver</tt> which provides a Capybara and the Selenium
+# Driver with no configuration. It's intended to work out of the box.
@dhh
dhh Jan 10, 2017 Member

I think we should downplay the RailsSeleniumDriver stuff. People shouldn't need to know the internals of this when first learning about it. RailsSeleniumDriver feels like essentially a private class that's just used below a set of configuration points.

+# require 'system_test_helper'
+#
+# class Users::CreateTest < ActionSystemTestCase
+# def adding_a_new_user
@dhh
dhh Jan 10, 2017 Member

Should use the string-based definition form, so test "adding a new user".

+# tests.
+#
+# class ActionSystemTestCase < ActionSystemTest::Base
+# ActionSystemTest.driver = :rack_test
@dhh
dhh Jan 10, 2017 Member

I don't think we have a compelling argument to ship with a Rack driver or highlight its availability. Since the only reason you'd suffer the slowdown of running through a browser for testing is to get the JavaScript tested too, I don't think it makes sense to promote a rack way that doesn't allow this.

I'd vote to nix this option. If there's a backdoor somewhere to do it by setting things by hand, fine. But it shouldn't be promoted and we should make no special accommodations for it.

+# class ActionSystemTestCase < ActionSystemTest::Base
+# ActionSystemTest.driver = ActionSystemTest::DriverAdapters::RailsSeleniumDriver.new(
+# browser: :firefox
+# )
@dhh
dhh Jan 10, 2017 Member

Thinking about the two-way configuration path we could use:

class ActionSystemTestCase < ActionSystemTest::Base
  served_by :puma, port: 3000
  accessed_with :firefox, headless: true
end
+# )
+# end
+#
+# A list of supported adapters can be found in DriverAdapters.
@dhh
dhh Jan 10, 2017 Member

Let's make the drivers an implementational detail, only expose the configuration options.

+end
+```
+
+First we visit the +users_path+. From there we are going to use Action System
@dhh
dhh Jan 10, 2017 Member

We should be using _url's here, rather than _path's, since we're simulating full browser requests.

+button. Lastly, we assert that the text on the Users show page is what we
+expected, which in this case is "Arya".
+
+For more helpers and how to write Capybara tests visit Capybara's README.
@dhh
dhh Jan 10, 2017 Member

Need at least a link to the write spot in the Capybara README that talks about the helpers.

+end
+```
+
+Capybara itself provides 4 drivers: RackTest, Selenium, Webkit, and Poltergeist.
@dhh
dhh Jan 10, 2017 Member

I don't think the name/type of driver is actually important to dwell on for these system tests. What we care about is what these drivers provide: Firefox, Chrome, Safari?, and headless/not driving. Let's bring the documentation up to talk about these areas configuration points and downplay the fact of whatever underlying driver is supplying what.

+
+```ruby
+class ActionSystemTestCase < ActionSystemTest::Base
+ ActionSystemTest.driver = ActionSystemTest::DriverAdapters::CapybaraDriver.new(
@dhh
dhh Jan 10, 2017 Member

There's something odd about this way of configuring the driver. Since ActionSystemTest is a global, it's weird to set properties on it within the scope of a ActionSystemTestCase class. Like, is there some hook that automatically rolls this driver setting back when this case is done running? Or are we just generally setting a global that'll be in place until we change it? If the latter, it should be set outside the scope of the class definition.

But I think even better would be to extract these settings to a higher level, so it's something like:

class ActionSystemTestCase < ActionSystemTest::Base
  driven_by :poltergeist, server: :webkit, port: 3000
end
@dhh
dhh Jan 10, 2017 Member

As I reflected further on the whole poltergeist vs selenium vs whatever, I realized that this configuration point is still too low level. We shouldn't be caring about whether its poltergeist or whatever. We should be caring about which browser we're simulating, whether it's headless or not, and then perhaps the port we're hitting against. Need more work there.

@dhh
dhh Jan 10, 2017 Member

Further reflection: I think we should split these configuration points into two separate things: 1) What server are we providing, like puma on 3000. 2) What driver type are we using, so that's which browser we're simulating and whether we're headless or not.

+ # | Rack Test | No JS Support | No |
+ # | Selenium | Firefox | Yes |
+ # | WebKit | Headless w/ Qt | Yes |
+ # | Poltergeist | Headless w/ PhantomJS | Yes |
@dhh
dhh Jan 10, 2017 Member

Can we just pick WebKit or Poltergeist? If they offer the same browser compatibility, we don't need both.

+ # ActionSystemTest.driver = ActionSystemTest::DriverAdapters::CapybaraDriver.new(
+ # name: :webkit,
+ # server: :webrick
+ # )
@dhh
dhh Jan 10, 2017 Member

Don't think we have any reason to advocate for a webrick server any place in the documentation. Unless I'm missing something that webrick gives us that puma does not? And if it does, we should document that reason.

+
+module ActionSystemTest
+ module DriverAdapters
+ module WebServer # :nodoc:
@dhh
dhh Jan 10, 2017 Member

Let's turn this from a concern into a real class. It relies on shared state where it's getting mixed in and it has its own private methods. Too much entanglement to be a good candidate for a concern, imo.

+ include TestHelpers::ScreenshotHelper
+ include Capybara::DSL
+
+ Capybara.app = Rack::Builder.new do
@dhh
dhh Jan 10, 2017 Member

Agree, but also, I think we should encapsulate this in a bootstrap method somewhere. It feels oddly free-floating in a helper to do this kind of global configuration. (I'm thinking something like register_browser_driver).

+ #
+ # This method should be used in the test setup code so that jobs run
+ # when the test runs because the test can't wait to check for pass/fail
+ # if it depends on running jobs.
@dhh
dhh Jan 10, 2017 Member

I'm not sure I understand this. Are we doing this to make sure that people don't run this with a synchronous runner? Async is already the Rails default, right? I don't think we need to do this work at all then?

+ #
+ # This method should be used in the test teardown code so that the
+ # queue adapter is always set back to the original queue adapter.
+ def reset_queue_adapter_to_original
@dhh
dhh Jan 10, 2017 Member

Who are we setting this adapter back for? Is it if you run your system tests together in the same process as other tests? Reasonable to ponder whether that's a good idea in general.

+module ActionSystemTest
+ module TestHelpers
+ # Screenshot helper for system testing
+ module ScreenshotHelper
@dhh
dhh Jan 10, 2017 Member

I like it. I think it's exactly the kind of process improvement that makes the out-of-the-box configuration feels extra nice. We use those screenshots on every failure for BC3 to help diagnose.

```
The `helpers`, `mailers`, and `models` directories are meant to hold tests for view helpers, mailers, and models, respectively. The `controllers` directory is meant to hold tests for controllers, routes, and views. The `integration` directory is meant to hold tests for interactions between controllers.
+The system test directory holds system tests, also known as acceptance tests.
+System tests inherit from Capybara and perform in browser tests for your
+application.
@dhh
dhh Jan 10, 2017 Member

Let's nail the value proposition clearer here. You do system tests when you want to test with JavaScript and through a real browser. The other stuff is not important in comparison.

+```
+
+Here the test is inheriting from `ActionSystemTestCase`. This allows for no-setup
+Capybara integration with Selenium Webdriver.
@dhh
dhh Jan 10, 2017 Member

What other options are there for inheritance?

+```ruby
+ActionSystemTest.driver.supports_screenshots?
+=> true
+```
@dhh
dhh Jan 10, 2017 Member

Good example of stuff we can nix when Rack is ejected.

+queue adapter back to the adapter your application has set for other environments.
+
+This is helpful for ensuring that jobs run async so that the test's that rely
+on job code are correctly tested at the time they run.
@dhh
dhh Jan 10, 2017 Member

Can nix all this when we commit to async.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment