Extending asset pipeline with custom pre-processors #2211

Closed
igrigorik opened this Issue Jul 23, 2011 · 39 comments

2 participants

@igrigorik

Perhaps this already exists, in which case someone can point me to an appropriate example...

When you're working with Google's Closure JS libraries, there is the following workflow: first you reference the base.js file in your header, and then in your JS you can just call goog.require('name_of_package') and Closure will automatically figure out which files to pull and serve from its source. This is all great, except that if you leave it at that then you'll likely end up serving 10+ JS files to satisfy all the deps - and, after all, this is what asset pipeline is supposed to fix!

For a quick intro to closure to see this pattern, check out: http://code.google.com/closure/library/docs/gettingstarted.html

<script src="closure-library/closure/goog/base.js"></script>
<script>
  goog.require('goog.dom');
</script>
<script>
  var newHeader = goog.dom.createDom('h1');
</script>

To address the above issue, Closure ships with a calcdeps.py which basically runs a regex on all your files looking for goog.require's, assembles a dependency tree and then concats all the right files into a single output file. For sake of cleanliness, it would't be hard to rewrite the above in Ruby, but the point is: this step needs to run prior to the compiler within the asset pipeline.

# scan hello.js for requires and output to hello-calc.js
closure-library/closure/bin/calcdeps.py -i hello.js -p closure-library/-o script > hello-calc.js

If you currently try changing the JS compiler from :uglifier to :closure and you create a closure "Hello World", you'll find that the compiler will error out and report that it can't find any of the imports. First, calcdeps.py needs to run, then the compiler can do its job and compress the file.

Question: What's the best way to tackle this within the current asset pipeline? I guess one way around this is to use the precompile stage, but this obviously breaks in dev mode where you'll be forced to rerun calcdeps everytime you update your javascript.


FWIW: Found this rake script trying to solve this problem, but its not really what we're after here: http://code.google.com/p/webos-goodies/source/browse/trunk/cms/rails/lib/tasks/closure.rake?r=551 -

calcdeps.py: http://code.google.com/p/closure-library/source/browse/trunk/closure/bin/calcdeps.py?r=134

@josh josh was assigned Jul 23, 2011
@igrigorik

plovr seems to one way that this problem has been solved by existing closure users: http://www.plovr.com/docs.html

It runs a java server which does the calcdeps stuff on the fly and abstracts all of the closure libraries (plovr ships with closure inside). For production, they suggest just piping its output to a new file:

java -jar plovr.jar build config.js > sample-compiled.js

Effectively, we need plovr, minus the java, and inside of Rails. :-)

@josh
Ruby on Rails member

Sprocket's directive parser is a completely modular. This makes it easy to change the syntax of //= require foo comments

https://github.com/sstephenson/sprockets/blob/master/lib/sprockets/directive_processor.rb

For the closure, google is basically using goog.require('goog.dom'); instead of our magic comments. But its basically the same principle. So I wrote a custom processor that looks for those lines and marks them as dependencies.

https://gist.github.com/1101923

If you're actually interested in this, you might consider making it into a real sprocket-closure library.

@josh josh closed this Jul 23, 2011
@igrigorik

Josh, trying to make this work.. Q: what's the proper way to register the new Tilt template handler with Rails 3.1? Trying to connect closure-compiler with this under prod mode, so far, no luck.

@josh
Ruby on Rails member

YourApp::Application.assets or Rails.application.assets should be a Sprockets::Environment instance. You can call register_preprocessor on that. Not sure if theres a way to do this directly off the config object.

@igrigorik

Hmm, that's exactly what I was trying to do.. doesn't seem to pick it up. Right now in my production config I have:

config.assets.register_preprocessor 'application/javascript', ClosureDependenciesProcessor

Loads up fine, but doesn't appear to be actually invoked.

Trying to access App::Application.assets / Rails.application.assets results in nil's for both.

@josh
Ruby on Rails member

config.assets is a different object. Its some Rails magic object hash.

Try doing it in an initializer file. config/initializers/closure.rb

@igrigorik

Same deal. This is what I have as my initializer:

require 'klosure-tilt'

p [:rails, Rails.application]
p [:app, Klosure::Application.assets]

# Klosure::Application.assets.register_preprocessor 'application/javascript', ClosureDependenciesProcessor
Rails.application.assets.register_preprocessor 'application/javascript', ClosureDependenciesProcessor

Klosure::Application.assets returns nil, and Rails.application doesn't appear to have the Sprockets env (only a railtie which references it).

@josh
Ruby on Rails member

this railtie stuff is bullshit. looks like you need an after_initialize hook.

config.after_initialize do |app|
  app.assets.register_preprocessor
end
@igrigorik

Heh, slowly but surely, moving in the right direction.. I think.

require 'klosure-tilt'

Rails.application.config.after_initialize do |app|
  # app.assets.unregister_processor 'application/javascript', Sprockets::DirectiveProcessor
  app.assets.register_preprocessor 'application/javascript', ClosureDependenciesProcessor
end

That bombs at startup with:

Exiting
/Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/sprockets-2.0.0.beta.12/lib/sprockets/index.rb:60:in `expire_index!': can't modify immutable index (TypeError)
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/sprockets-2.0.0.beta.12/lib/sprockets/processing.rb:91:in `register_preprocessor'
    from /Users/igrigorik/Desktop/klosure/config/initializers/closure.rb:5:in `block in <top (required)>'

@josh
Ruby on Rails member

sigh. I'm going to see about getting Rails.application.assets to exist on boot (before after_initialize).

@josh
Ruby on Rails member

kk, try this 1edfadf

@igrigorik

Hopefully not doing somethign silly here, but getting the following:

/git/rails/activesupport/lib/active_support/core_ext/module/aliasing.rb:31:in `alias_method': undefined method `asset_environment' for class `Sprockets::Railtie' (NameError)
    from /git/rails/activesupport/lib/active_support/core_ext/module/aliasing.rb:31:in `alias_method_chain'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/bundler/gems/sass-rails-233cf837197f/lib/sass/rails/monkey_patches.rb:7:in `included'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/bundler/gems/sass-rails-233cf837197f/lib/sass/rails/monkey_patches.rb:23:in `include'
@josh
Ruby on Rails member

fucking mess - https://github.com/rails/sass-rails/blob/master/lib/sass/rails/monkey_patches.rb

Disable sass-rails for now. Thats going to have to be fixed.

@igrigorik

Back to the same immutable index exception:

/Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/sprockets-2.0.0.beta.12/lib/sprockets/index.rb:60:in `expire_index!': can't modify immutable index (TypeError)
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/sprockets-2.0.0.beta.12/lib/sprockets/processing.rb:91:in `register_preprocessor'
    from /Users/igrigorik/Desktop/klosure/config/initializers/closure.rb:5:in `block in <top (required)>'
    from /git/rails/activesupport/lib/active_support/lazy_load_hooks.rb:34:in `call'
    from /git/rails/activesupport/lib/active_support/lazy_load_hooks.rb:34:in `execute_hook'
    from /git/rails/activesupport/lib/active_support/lazy_load_hooks.rb:43:in `block in run_load_hooks'
    from /git/rails/activesupport/lib/active_support/lazy_load_hooks.rb:42:in `each'
    from /git/rails/activesupport/lib/active_support/lazy_load_hooks.rb:42:in `run_load_hooks'
    from /git/rails/railties/lib/rails/application/finisher.rb:56:in `block in <module:Finisher>'
    from /git/rails/railties/lib/rails/initializable.rb:25:in `instance_exec'
    from /git/rails/railties/lib/rails/initializable.rb:25:in `run'
    from /git/rails/railties/lib/rails/initializable.rb:50:in `block in run_initializers'
    from /git/rails/railties/lib/rails/initializable.rb:49:in `each'
    from /git/rails/railties/lib/rails/initializable.rb:49:in `run_initializers'
    from /git/rails/railties/lib/rails/application.rb:92:in `initialize!'
    from /git/rails/railties/lib/rails/railtie/configurable.rb:30:in `method_missing'
    from /Users/igrigorik/Desktop/klosure/config/environment.rb:5:in `<top (required)>'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:236:in `require'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:236:in `block in require'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:222:in `block in load_dependency'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:616:in `new_constants_in'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:222:in `load_dependency'
    from /git/rails/activesupport/lib/active_support/dependencies.rb:236:in `require'
    from /Users/igrigorik/Desktop/klosure/config.ru:4:in `block in <main>'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/builder.rb:51:in `instance_eval'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/builder.rb:51:in `initialize'
    from /Users/igrigorik/Desktop/klosure/config.ru:1:in `new'
    from /Users/igrigorik/Desktop/klosure/config.ru:1:in `<main>'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/builder.rb:40:in `eval'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/builder.rb:40:in `parse_file'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/server.rb:200:in `app'
    from /git/rails/railties/lib/rails/commands/server.rb:46:in `app'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/server.rb:301:in `wrapped_app'
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/gems/rack-1.3.2/lib/rack/server.rb:252:in `start'
    from /git/rails/railties/lib/rails/commands/server.rb:70:in `start'
    from /git/rails/railties/lib/rails/commands.rb:54:in `block in <top (required)>'
    from /git/rails/railties/lib/rails/commands.rb:49:in `tap'
    from /git/rails/railties/lib/rails/commands.rb:49:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'
@josh
Ruby on Rails member

Don't use the after_initialize hook anymore.

# config/initializers/closure.rb
Rails.application.assets.register_preprocessor 'application/javascript', ClosureDependenciesProcessor
@igrigorik

Ok, now I'm back to square one - I think. Question on the gist: https://gist.github.com/1101923 -- does that actually render something for you? In order to get it to work on this side I had to add an extra javascript script tag to pull in the base.js file.

(and I have a feeling that whatever is the difference there is also why I can't get it to work in rails)

@josh
Ruby on Rails member

I bet all your closure-library code is outside your asset path. Either add it to app/assets or add /path/to/closure-library/closure to your config.assets.path.

@igrigorik

Alright, making progress. Can bring up "hello world" both in dev and production. Here's the basic summary:

https://gist.github.com/9a72dc99931429c4f8f8

A few outstanding questions that are not resolved in my mind:

  • When I run in prod, with :closure as js compressor (and set to true), I would expect the output to be minified.. but that's not the case. Hmm?
  • When we register_preprocessor it replaces the native DirectiveProcessor, correct? In which case, the ClosureDepsProcessor would need to be extended to support patterns/files other than native closure lib. Does that sound roughly right?
@josh
Ruby on Rails member
  1. The closure compressor should work. You do need this gem, but otherwise that sounds like a Rails bug. Also try RAILS_ENV=production rake assets:precompile.

  2. No, they stack. Quick explanation, preprocessors + extension processors + postprocessors run on each file. DirectiveProcessor is a preprocessor so it runs and reads any "magic comments" and remembers them. Then any extension processors like .coffee or .erb run. Last, we have this SafetyColons filter that makes sure the last line is a semicolon for safe concatenation. register_preprocessor adds an additional processor. So DirectiveProcessor then ClosureDependenciesProcessor runs. It should be fine running both on all your js.

@igrigorik

RAILS_ENV=production rake assets:precompile throws an error on a test file, but here's the question.. I'm looking at the precompile rake task, and if I understand it correctly, it simply grabs all the assets paths and runs them through the compressor? Isn't this kind of backwards? First we need to figure out the deps, then generate the single file, and then run the compiler / compressor? (Must be missing something here)

You're right on the Processors - got it working, not sure why it didn't fire for me yesterday!

@igrigorik

Orthogonal, but related.. It would be nice to be able to provide compile flags to the compiler (right now it just uses the defaults):

https://github.com/rails/rails/blob/master/actionpack/lib/sprockets/railtie.rb#L74
https://github.com/documentcloud/closure-compiler/blob/master/lib/closure/compiler.rb#L13

@josh
Ruby on Rails member

precompile only compiles application.js and application.css by default. Sprockets will look up application.js run the processors, gather the dependencies, process them as well, then run the compressor on the entire concatenation.

You could actually do this by hand with Rails.application.assets["application.js"].to_s, then write that to a file.

If you want other compiler flags, don't use the symbol shorthand. Just do js_compressor = Closure::Compiler.new({...})

@igrigorik

As far as I can see, the compiler is not getting invoked -- where is it actually being called within Rails?

Also, if I run the rake precompile task, even though my application.js only specifically requires the goog stuff, I'm seeing goog/, jquery/, bin/, closure-library/ showing up in my public/assets - that doesn't seem right?

My application.js is // = require welcome.js and welcome.js is:

goog.require('goog.dom');

function sayHi() {
  var newHeader = goog.dom.createDom('h1', {'style': 'background-color:#EEE'}, 'Hello world!');
  goog.dom.appendChild(document.body, newHeader);
}

ig

@josh
Ruby on Rails member

I'm suspecting it isn't being assigned.

https://github.com/rails/rails/blob/master/actionpack/lib/sprockets/railtie.rb#L52

Images and other files are copied over too. A better setup would be:

app/assets/application.js
app/assets/welcome.js
vendor/closure-library/
config.assets.paths << "vendor/closure-library/closure"

Is this just a dummy app you are setting up? Maybe you could create git repo and I can pull it down and help you test it.

@igrigorik

Josh, I must be missing something super obvious here.. Shared the repo with you @ https://github.com/igrigorik/klosure

Doing a precompile appears to import jquery, and also skips the compile on application.js.

@josh
Ruby on Rails member

Pushed some cleanups to that repo.

Think I know whats wrong with your compressor. Clear out tmp/cache and try this again:

$ RAILS_ENV=production bundle exec rake assets:precompile
@igrigorik

aha! it worked - thanks josh!

Ok, so that's a gotcha.. I run my server in dev mode, it throws a bunch of stuff into tmp/cache, and then even if I run assets:precompile it skips right over that and declares its job done? That doesn't feel right, and something tells me I'm not the only one that would get burned by this behavior? Or am I simply overlooking something here?

@josh
Ruby on Rails member

tmp/cache is setup for faster recompiles == faster deploys

@igrigorik

Sure, I can appreciate that, but in this case it seems to break my expectations. If the tmp/cache cached development javascript, and my production javascript is subject to different rules (like an extra compile stage), then it seems that when I boot the server in production mode, the JS should be recompiled.

Should tmp/cache itself be namespaced with an environment? tmp/cache/dev vs tmp/cache/prod?

@josh
Ruby on Rails member

What we could actually do its sprinkle Rails.env into the sprockets hash key. That'd be other way to namespace them.

https://github.com/sstephenson/sprockets/blob/master/lib/sprockets/digest.rb#L23-44

env.version = "#{Rails.env}-1.0"

@igrigorik

Hmm, wait, I think I can answer that.. So the problem I ran into is that at some point my compile stage really didn't work, but it still compiled the js with the md5 slug at the end, and hence subsequent requests always resulted in non-compiled version. Correct?

Perhaps the actual gotcha here is: when I ran rake assets:clean I assumed that I would be started with a clean slate, except.. that's not the case, I also needed to run rake tmp:clear. How about running tmp:clear as part of assets:clean?

@igrigorik

+1 for adding Rails.env to the key. In fact, I think if you do both then you'll solve this problem once and for all: the tmp:clear approach would still fall down if dev environment was set to compile the JS (hence generate same fingerprint).

@josh
Ruby on Rails member

Clearing out tmp/cache seems like another good thing todo.

@igrigorik

@josh: slowly but surely getting there. Extracted the initializer stuff into a gem / railtie: https://github.com/igrigorik/closure-sprockets

Boots up and works great in dev mode, but blows up when I try to start the server in production mode:

=> Ctrl-C to shutdown server
Exiting
/Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/bundler/gems/sprockets-fc31a6b42d51/lib/sprockets/index.rb:60:in `expire_index!': can't modify immutable index (TypeError)
    from /Users/igrigorik/.rvm/gems/ruby-1.9.2-p180/bundler/gems/sprockets-fc31a6b42d51/lib/sprockets/trail.rb:39:in `append_path'
    from /git/closure-sprockets/lib/closure-sprockets/railtie.rb:7:in `block in <class:Railtie>'

@josh
Ruby on Rails member

Can't use config.after_initialize, thats too late.

  1. require 'sprockets/railtie' at the top of the file. This ensures the sprockets hooks get installed first.
  2. Use a normal initializer block

Example: https://github.com/rails/sass-rails/blob/master/lib/sass/rails/railtie.rb

@igrigorik

Awesome, works great.. took a bit longer than I expected, but great to have it working!

In terms of things remaining to be done:

  • make assets:clean invoke tmp:clear
  • make digests use Rails.env

The sass issue I ran into earlier seems to be fine now, albeit probably still needs to be revisited. Anything else I'm forgetting?

@josh
Ruby on Rails member

That sass issue is fixed in git but a new sass-rails gem hasn't been released.

Please do make issues for those 2 items. Even better pull requests if you can can.

@igrigorik

Pull request: #2448

@masatomo masatomo added a commit to masatomo/i18n-js that referenced this issue Jun 19, 2014
@masatomo masatomo Fixes an issue which doesn't update assets cache with rails 4
What happend:
1. Rails.application.assets.register_preprocessor always raises
TypeError: can't modify immutable index
2. The error is rescued so looks it's done. But context.depend_on is not
registered so locale file changes doesn't trigger expiring cache

I did some research and found
rails/rails#2211 (comment)
and simply removed ActiveSupport.on_load and everything is working well.
07b2757
@masatomo masatomo added a commit to masatomo/i18n-js that referenced this issue Jun 19, 2014
@masatomo masatomo Fixes an issue which doesn't update assets cache with rails 4
What was happening:
1. Rails.application.assets.register_preprocessor always raises
TypeError: can't modify immutable index
2. The error is rescued so looks it's done. But context.depend_on is not
registered so locale file changes doesn't trigger expiring cache

I did some research and found
rails/rails#2211 (comment)
and simply removed ActiveSupport.on_load and everything is working well.
a899898
@masatomo masatomo added a commit to masatomo/i18n-js that referenced this issue Jun 19, 2014
@masatomo masatomo Fixes an issue which doesn't update assets cache with rails 4
What was happening:
1. Rails.application.assets.register_preprocessor was always raising
TypeError: can't modify immutable index.
2. The error was rescued so looks it's done. But context.depend_on is
not registered so locale files changes didn't expires cache.

I found
rails/rails#2211 (comment)
and simply removed ActiveSupport.on_load (and the rescue) and everything
is working well.
abcd3ec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment