Added Rails 3.1 asset pipeline support

[JangoSteve]

I added support for the Rails 3.1 asset pipeline to save_and_open_page.

It's pretty straight forward. Currently, the method rewrites local asset paths to absolute filesystem paths in the rendered HTML. So, I basically just added a check to see if ::Rails is defined, if the version is 3.1 or greater, and if the asset pipeline is enabled. If so, it loads the asset manifest file to grab the asset fingerprints for each file, and then rewrites the fingerprints into the rendered HTML.

I have not yet extensively tested this, or had a chance to think about what a test case would look like, but it works very well with my app.

[JangoSteve]

Oh, btw, this relates to #485.

[tracedwax]

Works great - thanks very much for this. Css displays perfectly, some pictures don't display as normal but it's more than adequate for save_and_open_page.

[JangoSteve]

@tracedwax, are you able to figure out why some images aren't displaying as normal? They should all work. Maybe check your manifest.yml file to make sure they're getting included in there, and then make sure the image src matches in the generated page. It may be that those images aren't configured properly in the asset pipeline.

[jfi]

Thanks for this! I get the following error (Ruby 1.8.7, Rails 3.1.1)

can't convert nil into String

/Users/James/.bundler/ruby/1.8/capybara-16837b71299f/lib/capybara/util/save_and_open_page.rb:57 :in `join'

[dom1nga]

JangoSteve, plz replace in lib/capybara/util/save_and_open_page.rb:58 "Dir.exists?()" to ".File.directory?()" for compatibility with ruby 1.8.7
is there an easier way to solve this problem?
with selenium javascript_driver styles work correctly when browsing. fall only when viewing the page. Does it make sense to compile assets only for single using save_and_open_page method?

[JangoSteve]

Hey @dom1nga good point about 1.8.7, I guess I've been using 1.9.2 for so long I forgot that was a new thing.

I was wondering the same thing when I started working on a fix. Trust me, requiring an asset precompile was not my first choice.

The issue is that, with the asset pipeline enabled, assets are now part of the app; javascripts, stylesheets, and images are no longer static files that bypass the rails stack, they actually go through the rails process and get interpreted at runtime just like views.

The only way we could get them to render in the save_and_open_page template would be to rewrite all asset paths to point to the localhost domain/port spawned by capybara when running the test suite, so that the asset requests will get properly piped through sprockets allowing the assets to be compiled.

However, that would require the localhost app to still be running after our save_and_open_page tmp page has rendered to the browser (since that's when the asset requests would actually start). By this time, our server instance is probably already gone. I'm assuming this is the reason save_and_open_page even creates a tmp cached static file in the first place instead of just opening the current host/port URL in a new window (but I may be wrong).

This was easy before, because all assets were just rendered as static files from the public directory anyway, so there was no difference if you loaded them from the dev server path or from a file page. And so save_and_open_page simply rewrote relative domain paths to be absolute file paths. This is no longer possible since those static files don't necessarily exist anymore.

The only other option would be to make save_and_open_page compile the assets and then cache all of them to the same tmp location as the html page. In fact, that's basically the "solution" mentioned in #485, except that it ignored images and javascripts. However, this results in save_and_open_page taking a long time to run and creating a ton of tmp files (one for each javascript, stylesheet, and image on the page).

Taking all fo these considerations into account, this was the best solution I could come up with.

[JangoSteve]

@jfi, if you open RAILS_ENV=test rails console, what are the values of:

> ::Rails.application.config.assets.enabled
> ::Rails.application.config.assets.manifest

The only time I know of that the latter would be nil would be if the former was false. But if if assets.enabled is false, then that line shouldn't be called. Definitely weird.

[JangoSteve]

@tracedwax, I've encountered a few times where certain images or stylesheets weren't being rendered properly, and it was because I had forgotten to actually add those files to the asset pipeline (meaning they weren't being rendered properly in production either, only I hadn't known that because they were fine in dev).

[jnicklas]

Sorry to take so long to weigh in on this. I am never going to accept a patch which does anything Rails specific, even if you put it in a conditional. I honestly think this is a lot of overhead, for very little gain. If you desperately need this, temporarily switch to selenium instead.

[JangoSteve]

@jnicklas, am I correct in the reasoning for why save_and_open_page copies the page and changes all of its asset references, rather than just simply opening the page it's currently on as-is?

[jnicklas]

Pretty much. Copying all the assets seems like overkill. I think we shouldn't overengineer this, there are better alternatives when this kind of precision is required.

[JangoSteve]

Copying all the assets seems like overkill.

Agreed. That's why my patch here does not copy any assets.

I think we shouldn't overengineer this, there are better alternatives when this kind of precision is required.

Any suggestions? The problem is that, in Rails 3.1 and later, assets are no longer static files, they are compiled as a part of the asset pipeline. That means either:

a) They are not precompiled, and thus, rely on the Sprockets middleware running with the rails server.

b) They are compiled, in which case, the filenames are different, and the rewritten paths from save_and_open_page will be wrong.

Either way, it doesn't work.

Capybara's solution to the page requiring the server to render was to copy the html over to a tmp directory. I didn't want to use that same solution for the assets that require the server, because that would require copying all the assets over too (and like you said, that would seem overkill).

So, instead, this patch just rewrites all the asset paths to the precompiled asset paths, instead of blindly rewriting the paths to the /public directory.

There may be a way we could do this without requiring Rails-specific code, though. I'm going to try something else, and if it works, I'll open a new ticket. In the meantime, everyone who needs to use save_and_open_page with Rails 3.1 is still welcome to use my fork.

[dom1nga]

@JangoSteve +1

[jnicklas]

I just don't see this as a big issue, and I disagree that it needs fixing at all. Switch to selenium, use pry and you'll have a debug environment which is 100 times better than save_and_open_page. And even if you use it as is, you can still probably see most of the relevant markup just fine, even if CSS and JS are stripped.

[dom1nga]

@jnicklas, better to open a page, than every time to think about whether enough seconds set for sleep. Why watch "cartoon" of, for example, twenty episodes, if you want to watch only twenty-first? Let's all give up the use of webkit. More cartoons! More selenium!

[JangoSteve]

@jnicklas, that's the point, you can't see most of the relevant markup just fine. Everyone on my team has stated that save_and_open_page became useless in Rails 3.1 (and I've heard it from a lot of people outside of my team as well), and it made simple debugging (short of switching to selenium/pry) very difficult. Trust me, I wouldn't be spending my time on it if it wasn't a big issue.

[baldwindavid]

The fork from @JangoSteve has worked well for me and I will continue to use it if that is the best option. That being said, I would just as soon not see Capybara be fragmented into multiple branches if @JangoSteve is willing and able to work towards a solution compatible with the vision and constraints of the project. @jnicklas - Since this does not seem to be a core concern for you and already depends on launchy, are you thinking that save_and_open_page should be pulled out into a separate gem?

[JangoSteve]

So um, can everyone having problems with this try something? After spending half the day combing the source for sprockets and sprockets-rails, I think I may have found a simpler solution.

Add this to config/environments/test.rb:

config.assets.digest = false

Then run:

RAILS_ENV=test bundle exec rake assets:precompile

By default rake assets:precompile runs in the production mode, not development as you'd expect, and production usually has digest enabled, so you only get digested filenames when compiled into your public directory. However, if you explicitly run the precompile task in an environment which has digest=false, it will actually create two sets of assets, one with the digested filenames, and one with undigested filenames.

And with that, capybara's default path rewriting will match the undigested filenames and things will work.

If you want to get rid of all of these files when you're done debugging, you can run:

bundle exec rake assets:clear

If you call save_and_open_page and notice some CSS, or JS missing, it's probably because those weren't included in your sprockets load paths. Be sure to either require them in your application.[cs|js], or add them to config.assets.precompile and/or config.assets.paths. If this was the problem, it would have become apparent in production when your assets didn't show up there either.

Perhaps we could add something to the readme or something that tells people to run precompile in test mode before using save_and_open_page with Rails 3.1.

[pietere]

Hey @JangoSteve, your last tip (precompiling) did the trick for me, and I think it's the cleanest way to do so! Thanks!

[JangoSteve]

@pietere, the only problem is that, since it's creating precompiled assets without the digest, your development environment will actually serve those instead of the live ones in your app directory, which can really mess things up when developing. So, in order to use save_and_open_page, you must precompile, do your thing, and then always clear the assets before going back to developing.

But I agree, it's the cleanest way to do so (just not the most convenient).

[pietere]

@JangoSteve, yes I realised that after posting my message here. I will try to find a workaround for this to automate this clean up, and post it here if I did find anything good to share

[pietere]

@JangoSteve I think I have a quite clean solution

• we tell the test environment to use the precompiled assets, but to use another folder to server the assets from. Add this in the file config/environments/test.rb

config.assets.digest = false
config.assets.prefix = "/cucumber_test_assets"

• we want to automatically precompile assets when we run cucumber, so we add this at the bottom of the hooks.rb file from cucumber:

%x[bundle exec rake assets:precompile]

Now, each time that we run cucumber, the assets are precompiled in a folder called public/cucumber_test_assets, and development environment is not affected, so no need to clean up those assets.

• For completeness, I added this to my .gitignore

public/cucumber_test_assets/

[ghost]

This works fine with simple asset directory structure ( application.css ), but I am working on a multi-layout app, and I my application.css is not used ... rather I have 2 css assets files referenced in my layouts :
application_admin.css
application_users.css

each one requiring different css files .... it works fine in dev / prod ..
I guess it's a matter of doing a correct recompilation .... ? %x[bundle exec rake assets:precompile] should be modified ?

[ghost]

found it : adding other files .... config.assets.precompile += ['admin.js', 'admin.css', 'swfObject.js']

[Nerian]

Combining @pietere solution with https://github.com/dnagir/guard-rails-assets works like a charm :)

[maxim]

Why not release a gem named capybara-rails that adds the rails-specific logic, just like many other gems out there? cc @JangoSteve

[JangoSteve]

@maxim, the biggest factor is that the original solution requires re-writing parts of #rewrite_css_and_image_references, since capybara doesn't provide event hooks within the method. This means that an external gem would have to track capybara and merge the method with any changes capybara makes in core, which usually just results in an outdated gem.

But if one of the newer solutions can solve the problem without rewriting the internal methods, then I'd say yes, creating a separate gem would be a good approach.

[maxim]

@JangoSteve I'm sure @jnicklas wouldn't mind adding an extension point where one is useful. On the other hand, I don't see a problem overriding a method in capybara as long as dependencies are correctly specified. We can just add gem "capybara-rails" to Gemfile, which depends on its compatible version of capybara, and remove gem "capybara" itself. Then hopefully update capybara-rails often enough until @jnicklas has a chance to add a cleaner way to change that stuff, like injecting a proc.

Regarding the feature being Rails-specific — this pull request would be, but a generic extension point would not. Rails is nothing more than a very popular type of Rack app that has some special handling of assets. I see it as a good enough reason to add an extension point to inject "special handling of assets". There are rack apps in the wild that depend on it.

Whether or not save_and_open_page a best way to debug is a separate conversation. Feature exists, I love using it, many others do. It's quicker for me than to switch to a terminal and "sense" my surroundings with pry, or setup selenium.

[JangoSteve]

@maxim Agreed on all points.

By the way, IIRC, when I originally wrote this, I actually was going to just extend #rewrite_css_and_image_references to call e.g. a #post_rewrite method which could just be defined to return true in core capybara, and then could be overwritten in a gem or app. The reason I didn't is because, it would have required me to separate out this new functionality from capybara's core functionality, which would then require us to have two separate regexes that loop through all the asset directory entries, which would have been twice as slow.

But maybe that's not a bad tradeoff to gain the extensibility and modularity, especially considering this is a debug function that's not going to be used inside loops or in production apps.

[sdhull]

FWIW, 👍 for gem "capybara-rails"

[gaffo]

try the capybara-screenshot gem. Works well.

[JeanMertz]

Was this ever further developed? I don't see a (working) capybara-rails gem, and I too miss this feature. Sure capybara-screenshot is nice and all, but I like to inspect my elements, plus, the screenshots only work when using webkit, which I only use for my Javascript tests (due to it being slower than basic Rack::Test.

Also, unfortunately @JangoSteve 's solution doesn't seem to work for me. I can see the assets being pre-compiled in in the public/assets (or test_assets) directory. And I can see the reference to them in the compiled html file:

<link href="/assets/application.css" media="all" rel="stylesheet" type="text/css">

But the css simply isn't loaded. Perhaps, I need the full path in there for it to work, because the base-path is probably incorrect when opening the html page with Capybara?

EDIT

My huch was correct, the following does work:

<link href="../../public/assets/application.css" media="all" rel="stylesheet" type="text/css">

But I only tested this using the Chrome inspector and editing the path manually. Now I need to find a way to make this change permanent. Although I don't understand why others don't have this issue.

[sdhull]

@JeanMertz I don't believe it has been developed. I might put in some work on this tonight, however.

[JeanMertz]

@sdhull Thank you for your input. As long as the above asset-precompile option works (hint: for me it doesn't, see my above updated post), I guess the need is a bit less relevant. Still, a gem would make it easier for most.

[sdhull]

@JeanMertz rake assets:precompile doesn't work for you? Or when you save_and_open_page, it doesn't have CSS & JS?

[JeanMertz]

@sdhull precompiling works, but the path to the precompiled assets is "wrong". It would not be wrong, if the page was to be opened using a web-server, which root path would be set to /public, but Capybara opens the page like this for me:

file://localhost/Users/Jean/Projects/test_app/tmp/capybara/capybara-20130209082823237840783.html

And then the assets are linked like this:

<link href="/assets/application.css" media="all" rel="stylesheet" type="text/css">

Obviously, that isn't going to work, because there is nothing setting the root path to file://localhost/Users/Jean/Projects/test_app/public. In my situation, the root path is actually file://localhost/Users/Jean/Projects/test_app/tmp/capybara/, so thats why (manually) prepending ../../public worked for me, because I change the root path to the correct location.

So again, not sure why others have gotten this to work, but I don't see how this would work without a change to the root path? (I am not using the fork in this PR, but it was my understanding that using the asset precompiling solution doesn't require a fork, correct?)

[sdhull]

Ahhh, yes ok that makes sense. Not sure what the recommended solution is at this point. I'm running capybara 1.1.4 which rewrites asset paths to file paths like (in OSX):

<link href="/Users/sdhull/path/to/app/public/assets/application.css" media="all" rel="stylesheet" type="text/css">

You must be running a more recent version of Capybara, in which this capability has been removed. I guess perhaps the recommended solution is to switch to Selenium driver and throw a debugger in your example to pause execution so you can inspect & debug your page in the browser window that pops up during execution.

Also, FWIW, I prefer debugging in Chrome, so we have this in our spec_helper.rb file (this defaults selenium to use Chrome, prereq is that you do brew install chromedriver first assuming you use Homebrew):

Capybara.register_driver :selenium do |app|
  Capybara::Selenium::Driver.new(app, :browser => :chrome)
end unless ENV["CHROME"] == "false"

[JeanMertz]

Ah, right, that makes sense. I was actually debugging by looking at the Capybara method documentation but only now do I realize that documentation is for version 0.3.9 (!!) which still had this method in.

One question: I was considering doing something like that, but I feel like I might lose a lot of performance using any other driver than Rack::Test for simple non-javascript tests. Is this hunch correct? If so, I might simply overwrite the default Capybara behavior and add back the path rewriting.

[sdhull]

If you choose to embark on that journey of adding back asset rewriting, you should consider going whole-hog and adding asset server support. Maybe via a capybara-rails gem? 😺

Of course, I'm working on adding it as a new PR, following the guidelines outlined by @jnicklas in #609 (after my brief but furious outrage about the asset situation in save_and_open_page).

And if my PR doesn't get merged, then I figured I'd release it as a gem that monkey-patches it in.

[JeanMertz]

That seems like a great solution. I'll be sure to chime in if you have anything to go with (just ping me at Github). For now, I'll use a dirty method overwrite to at least get me on the road again.

Finally, about the slower test suite when swapping Rack::Test for non-javascript tests, is this something you concur with? It seems your team uses Selenium even for non-javascript tests, this feels like it might slow you down a lot during development.

[sdhull]

Nope, we use Rack::Test for non-js specs. Though for now our test suite is small enough that it probably doesn't matter too much. But of course, save_and_open_page still loads assets for us (as long as we've precompiled them) ;)

Also worth noting is that for non-js specs, assets really don't matter that much for us anyway. If a spec fails, it's generally because something wasn't output on the page (or was output when it wasn't supposed to be), in which case assets don't matter much.

Only final annoyance for us is that we like to grab screenshots using save_and_open_page (after setting up some crazy states), but old asset rewriting only worked for like 99% of stuff so some stuff was still off.

We were previously trying to use capybara-webkit for js specs, but we're using some js that isn't supported by that driver (not sure what exactly), so these specs only pass with Selenium anyway.

[JeanMertz]

Thank you for your input.

Did you check out jonleighton/poltergeist? Maybe that would solve your problems, it seems like a nice alternative to capybara-webkit, including screenshot support.

[sdhull]

Holy crap, page.driver.debug sounds awesome! I'll have to give this a try. Thanks for the suggestion 😄

[JeanMertz]

For future reference, here's the method I added back from earlier Capybara versions:

module Capybara
  class Session

    def save_page(path=nil)
      body = rewrite_css_and_image_references(html)
      path ||= "capybara-#{Time.new.strftime("%Y%m%d%H%M%S")}#{rand(10**10)}.html"
      path = File.expand_path(path, Capybara.save_and_open_page_path) if Capybara.save_and_open_page_path

      FileUtils.mkdir_p(File.dirname(path))

      File.open(path,'w') { |f| f.write(body) }
      path
    end

    private

    def rewrite_css_and_image_references(response_html) # :nodoc:
      return response_html unless Capybara.asset_root
      directories = Dir.new(Capybara.asset_root).entries.inject([]) do |list, name|
        list << name if File.directory?(name) and not name.to_s =~ /^\./
        list
      end
      response_html.gsub(/("|')\/(#{directories.join('|')})/, '\1' + Capybara.asset_root.to_s + '/\2')
    end

  end
end

Simply add it in some file like features/support/capybara_assets_patch.rb.

This overwrites the default save_page method (I tried using super, but couldn't get it to work). Here's the diff:

 def save_page(path=nil)
+  body = rewrite_css_and_image_references(html)
   path ||= "capybara-#{Time.new.strftime("%Y%m%d%H%M%S")}#{rand(10**10)}.html"
   path = File.expand_path(path, Capybara.save_and_open_page_path) if Capybara.save_and_open_page_path

   FileUtils.mkdir_p(File.dirname(path))

   File.open(path,'w') { |f| f.write(body) }
   path
 end

[sdhull]

@JeanMertz Woot! Check me out: #958

[joxxoxo]

@JeanMertz You cannot use super, cuz it's not inheritance. But there's another option - alias_method_chain. Something like:

   def save_page_with_assets_patch(...)
   end

   alias_method_chain :save_page, :assets_patch

Or without active_support:

  alias_method :foo_without_feature, :foo
  alias_method :foo, :foo_with_feature