New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Capybara assertion support to Controller, Integration, Mailer, and View tests #43361
base: main
Are you sure you want to change the base?
Add Capybara assertion support to Controller, Integration, Mailer, and View tests #43361
Conversation
241a378
to
3825ad0
Compare
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Hey, just want to say this looks like a desirable PR. |
@seanpdoyle maybe the lack of context is the reason for the lack of further discussion on this PR. It's such a useful implementation... and it seems to me you've made requested modifications and done the benchmarking. Would it make sense to abandon this newer PR, and consolidate these changes into the older PR branch, so reviewers and core committers have the reminders of the work and review that has already gone into this? |
@seanpdoyle awesome work, +1-ing that it would be great if this was reconsidered. Oh, and thanks again for all your fantastic contributions to the community! |
909ac38
to
a1ce763
Compare
a1ce763
to
3599b88
Compare
thanks @p8, @cpjmcquillan and @seanpdoyle ! |
6a3b28e
to
10d5797
Compare
Unfortunately, #41291 was closed due to inactivity, and as a result, the context for this discussion must be split between two pull requests. Due to that split, I need to ping @eileencodes @rafaelfranca @zzak, since you were all involved in the original discussion. Given the concerns about the performance impacts of the original changeset, I've re-structured the proposal to a point where it's entirely opt-in. Is this approach more viable than the original? |
I think so, but I also think is more confusing. Why would someone chose this new list of assertions over the existing ones? Rails is opinionated, and having two set of assertions for the same kind of tests seems that we don't have an opinion in the matter. I still don't understand why we wouldn't improve rails-dom-testing. If the capybara assertions were not slower I'd even say we could just stop using rails-dom-testing, but that doesn't seems to be the case since those capybara assertions are almost 2 times slower in the worse case. |
@rafaelfranca thank you for jumping back into this conversation! I'm sorry for the long pause and the change in venue.
Capybara's built-in selectors are more robust and flexible than what # <button>Submit</button>
assert_button "Submit"
assert page.find(:button, "Submit")
# <button type="reset">Reset</button>
assert_button type: "reset"
assert page.find(:button, type: "reset")
# <button disabled>Submit</button>
assert_no_button "Submit"
page.find(:button, "Submit") #=> raises!
assert_button "Submit", disabled: true
assert page.find(:button, disabled: true)
# <button>Submit</button>
# <button type="button">Cancel</button>
assert_button { _1.text.include?("Submit") && _1["type"] != "button" }
# <button>Submit</button>
assert_select "button", "Submit"
# <button type="reset">Reset</button>
assert_select 'button[type="reset"]'
# <button disabled>Submit</button>
assert_no_select "button:not([disabled])", "Submit"
assert_select "button[disabled]", "Submit" Some selectors can be mapped directly one-to-one from Capybara to # <input type="submit" value="Submit">
# <input type="button" value="Submit">
# <button>Submit</button>
# <button type="button">Submit</button>
assert_button "Submit", count: 4
assert_button "Submit", type: "button", count: 2 Calls to Another feature of Capybara's selectors that would be challenging to recreate in # <label for="a_text_field">A text field</label>
# <input id="a_text_field" value="an initial value">
assert_field "A text field", with: "an initial value" The accessibility benefits alone feel valuable enough to warrant a migration. Personally, I find support for asserting that form controls are properly labelled without the need for JavaScript-enabled System Tests very compelling.
I think we're in agreement here. My perspective is that since most applications already depend on Capybara for driving their System Tests, integrating those predictable and robust selectors into their Rack Test driven integration tests could halve their maintenance burden. Choosing Capybara over
This is an unfortunate concern. My original proposal considered replacing |
To me, given all options capybara helpers provide over our own helpers I can't see why Rails should not provide that as default, other than the performance concerns. We spent a considerable amount of time trying to make But, if we can make Capybara helpers as fast as the ones we have today, I don't see a reason why we should not do that. I totally agree that if not too hard, we should spend time optimizing Capybara instead of reimplementing it. |
I'm not sure this is a true statement. I can see why it seems that way, but any apps upgrading wouldn't be forced to add Capybara unless they wanted system tests and any application that's generated as just an API skips system tests. It's also pretty easy to remove. So while we can say a fair amount of applications already depend on Capybara it's much less of a requirement than replacing rails-dom-testing with Capybara would be. We'd be making a hard requirement on Capybara for default integration testing, whereas right now it's only a hard requirement for system testing.
I think this is a big concern of mine as well, since I worked on making integration tests faster so they were in line with functional/controller tests. I wouldn't be supportive of undoing that work. The other side of it is that currently we control rails-dom-testing but we have no control over how and when Capybara fixes bugs or adds features. I'm sure that it's well maintained but it is a complex codebase and would make it harder for the core team to have influence into how it affects the framework and testing environments. There is a non-zero chance it would increase the maintenance burden for us. This is fine for system testing because there aren't many other alternatives we could plug and play into Rails, and system testing isn't your default go to when testing the majority of your framework (mainly because system testing is pretty slow and should be mainly used for testing how Rails and JS interact). I can tell you from first hand experience that since I'm not that familiar with Capybara it's been difficult for me to decide when to merge changes into Rails that "fix" system tests. I'm often relying on the PR author to have done due diligence into the right fix because I don't know Capybara well enough. TL;DR: My main concerns for moving to Capybara from rails-dom-testing are maintenance burden and performance. Improving Capybara perf gets rid of one my concerns. |
Thank you two for your perspectives. While I'm personally interested in exploring ways to optimize Capybara, I'm curious about the viability of this Pull Request in its current form. The Capybara assertions are opt-in, since the This changeset isn't replacing or deprecating This diff is the least disruptive version of these changes I could imagine. If the team isn't interested in accepting them, could we instead determine a more public interface so that a third-party gem could extend |
I'm up for that. While I can see this PR as a good compromise it goes against the opinionated nature of Rails. We are including 2 solutions inside the framework, when we usually are very opinionated on what should be used. An API to allow people to build on top of our system gives people to flexibility to chose, while keeping our opinion. |
cda3a52
to
cbc01c7
Compare
fa8c2a6
to
c571a1f
Compare
@rafaelfranca @eileencodes I've reconsidered the initial proposal implemented by e428beb. In its place, c571a1f introduces a I've updated the PR's description to encompass the changes, and I've included changes to Guides documentation. Since performance impacts were a concern, I've included some rough benchmarks in the PR description. Swapping Is this approach an improvement on the initial proposal? |
c571a1f
to
f79147c
Compare
819cdad
to
0566707
Compare
ce1ba4a
to
f84055a
Compare
f84055a
to
2055f66
Compare
9f061c7
to
6cd5267
Compare
6cd5267
to
f208914
Compare
f208914
to
49991ec
Compare
Also introduce `#assert_text_part` and `#assert_html_part` convenience methods: ```ruby test "asserts text parts" do mail = MyMailer.welcome("Hello, world") assert_part mail, :text do |part| assert_includes part.body.raw_source, "Hello, world" end assert_text_part mail do |text| assert_includes text, "Hello, world" end end test "asserts html parts" do mail = MyMailer.welcome("Hello, world") assert_part mail, :html do |part| assert_includes part.body.raw_source, "Hello, world" end assert_html_part mail do |html| assert_select html, "p", "Hello, world" end end ```
This is a re-submission of rails#41291. Both `ActionDispatch::IntegrationTest` and Capybara in `:rack_test` mode are capable of exercising Rails applications that serve HTML over HTTP. Capybara itself provides a [wide-range of selectors][selectors], which can be [extended even further][capybara_accessible_selectors]. While Capybara's JavaScript far exceeds `ActionDispatch::IntegrationTest`' HTML-only support, their overlapping capabilities makes Capybara an interesting candidate to supplant `ActionDispatch::IntegrationTest`'s assertions. For example, Capybara provides a `:button` selector, which can be invoked by `assert_selector :button, "Button Text"` or `assert_button "Button Text"`. Additionally, Capybara's `button` selector supports resolution based on ARIA attributes like `aria-label` and `[role="button"]`. The assertions provided by `rails-dom-testing` do not. It's possible to recreate those assertion characteristics in `rails-dom-testing`, Rails itself, or a consumer application. However, given the fact that `capybara` is already a Rails testing dependency (through System Test support), and the fact that they're both capable of coordinating with `Rack::Test`, there is an opportunity to unify them. [selectors]: https://github.com/teamcapybara/capybara/tree/84acc29d5ff807507fe57aafcf7f9b2acdb89fe2/lib/capybara/selector/definition [capybara_accessible_selectors]: https://github.com/citizensadvice/capybara_accessible_selectors/tree/d61971c609e3b019df6dc0ea0c9ce11433f3d0f7#documentation Action Dispatch: Use `build_rack_mock_session` in test --- This commit introduces the `ActionDispatch::Assertions::CapybaraAssertions` module to override the `ActionDispatch::Integration::Session#_mock_session` implementation by substituting the `Rack::MockSession.new` call with a memoized delgation to the an already constructed `Rack::MockSession` instance created by the `RackTest`-driven `Capybara::Session`. This commit includes a new test case that extends the `Session` instance to integrate with `Capybara`.`` Alternatives --- This is currently possible, without any implementation changes. Applications could create their own versions of `ActionDispatch::Assertions::CapybaraAssertions`by overriding `_mock_session`. That feels like a blatant violation of the public API. Having said that, if a third-party package were to provide this kind of Capybara integration, it would have to re-open the class and target the private `_mock_session` method for overriding. Similarly, [open_session][]'s documentation mentions the pattern of using `#extend` on the block's `Session` instance. However, I'm not sure if there's a way to promote the block argument to serve globally as the overridded `#integration_session` that methods are delegated to. [Rack::Test::Methods]: https://github.com/rack/rack-test/blob/v1.1.0/lib/rack/test/methods.rb#L29-L31 [Capybara::RackTest::Browser]: https://github.com/teamcapybara/capybara/blob/3.35.3/lib/capybara/rack_test/browser.rb#L128-L131 [open_session]: https://github.com/rails/rails/blob/main/actionpack/lib/action_dispatch/testing/integration.rb#L377-L386
…d View tests This commit introduces framework-specific configuration values: * `config.action_controller.assertions` for `ActionController::TestCase` * `config.action_dispatch.assertions` for `ActionDispatch::IntegrationTest` * `config.action_mailer.assertions` for `ActionMailer::TestCase` * `config.action_view.assertions` for `ActionView::TestCase` When set to `:capybara`, those tests include framework-scoped `CapybaraAssertions` modules that transitively include `Capybara::Minitest::Assertions`, and define the required `#page` method to parse the HTML into a `Capybara::Node::Simple` instance. When set to `:rails_dom_testing`, those tests include a framework-scoped `RailsDomTestingAssetrions` module to preserve the existing behavior. They all default to `:rails_dom_testing`.
49991ec
to
c4be01c
Compare
Motivation / Background
This is a re-submission of #41291.
Capybara provides a wide-range of selectors, which
can be extended even further.
While Capybara is capable of exercising browser-based, JavaScript-enabled pages, its assertions and selectors excel in JavaScript-free static HTML environments.
HTML-only assertions make Capybara a compelling alternative to the
rails-dom-testing
assertions utilized byActionController::TestCase
,ActionDispatch::IntegrationTest
, andActionView::TestCase
.For example, Capybara provides a
:button
selector, which can beinvoked by
assert_selector :button, "Button Text"
orassert_button "Button Text"
. Additionally, Capybara'sbutton
selector supportsresolution based on ARIA attributes like
aria-label
and[role="button"]
. The assertions provided byrails-dom-testing
donot.
It's possible to recreate those assertion characteristics gem-side in
rails-dom-testing
, internally Rails-side, or application-side.Given the fact that
capybara
is already a Rails testing dependency(through System Tests), there is an opportunity re-use that dependency across all HTML test cases.
Detail
This commit introduces framework-specific configuration values:
config.action_controller.html_assertions
forActionController::TestCase
config.action_dispatch.html_assertions
forActionDispatch::IntegrationTest
config.action_mailer.html_assertions
forActionMailer::TestCase
config.action_view.html_assertions
forActionView::TestCase
When set to
:capybara
, those tests include framework-scopedCapybaraAssertions
modules that transitively includeCapybara::Minitest::Assertions
, and define the required#page
method to parse the HTML into aCapybara::Node::Simple
instance.When set to
:rails_dom_testing
, those tests include a framework-scopedRailsDomTestingAssetrions
module to preserve the existing behavior.They all default to
:rails_dom_testing
.Additional Information
I've forked
eileencodes/integration_performance_test
and updated it to work with this branch:eileencodes/integration_performance_test@master...seanpdoyle:rails/action-dispatch-integration-capybara
The results show that asserting with
Capybara::Minitest::Assertions
is roughly 1.11x slower thanRails::Dom::Testing::Assertions
Checklist
Before submitting the PR make sure the following are checked:
[Fix #issue-number]