3.1 rc5, sprockets 2.0.0.beta.13: Setting config.assets.compress = true causes rails/sprockets to ignore js file include order

[ncri]

Without asset compression sprockets keeps the load order of my js files as specified in the manifest. However, when setting config.assets.compress = true the load order is ignored.

[spastorino]

@josh ?

[spastorino]

@josh ?

[josh]

We're going to need some more info.

[josh]

We're going to need some more info.

[ncri]

Okay, well, what info? ;-)

Well here is my manifest file:

// This is a manifest file that'll be compiled into including all the files listed below.
// Add new JavaScript/Coffee code in separate files in this directory and they'll automatically
// be included in the compiled file accessible from http://example.com/assets/application.js
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// the compiled file. require_tree .
//
//= require prototype
//= require prototype_ujs
//= require effects
//= require dragdrop
//= require controls
//= require main/lowpro
... and some more js files...
//= require_self

This load order is preserved when asset compression is off. But when on, the files seem to be compiled in alphabetical order, so i get lots of js errors.

[ncri]

Okay, well, what info? ;-)

Well here is my manifest file:

// This is a manifest file that'll be compiled into including all the files listed below.
// Add new JavaScript/Coffee code in separate files in this directory and they'll automatically
// be included in the compiled file accessible from http://example.com/assets/application.js
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// the compiled file. require_tree .
//
//= require prototype
//= require prototype_ujs
//= require effects
//= require dragdrop
//= require controls
//= require main/lowpro
... and some more js files...
//= require_self

This load order is preserved when asset compression is off. But when on, the files seem to be compiled in alphabetical order, so i get lots of js errors.

[ncri]

Hi there, any updates on this?
Thanks, Nico

[ncri]

Hi there, any updates on this?
Thanks, Nico

[spastorino]

Hey @ncri can you push an app to github that shows the issue? that would be helpful, thanks!!

[spastorino]

Hey @ncri can you push an app to github that shows the issue? that would be helpful, thanks!!

[elstudio]

I can confirm this is still an issue for 3.1 RC6 -- it breaks prototype and UJS/RJS for my app.

What can I get you guys to help diagnose this?

Would it help to see application.js source, and then compiled with and without asset compression?

[elstudio]

I can confirm this is still an issue for 3.1 RC6 -- it breaks prototype and UJS/RJS for my app.

What can I get you guys to help diagnose this?

Would it help to see application.js source, and then compiled with and without asset compression?

[josh]

#2580 is going to solve it. Please test this new approach.

[josh]

#2580 is going to solve it. Please test this new approach.

[ncri]

Thanks, I don't quite get it though what that ticket #2580 is about. My issue is not about offline or on the fly compiling, it's about the order sprockets appends my js files together when I switch on asset compression, which is preserved without compression. Sprockets is supposed to keep the ordering defined by the requires in the manifest as far as I know, so this is its intended use. So it seems like a bug to me. But maybe it's in sprockets?

[ncri]

Thanks, I don't quite get it though what that ticket #2580 is about. My issue is not about offline or on the fly compiling, it's about the order sprockets appends my js files together when I switch on asset compression, which is preserved without compression. Sprockets is supposed to keep the ordering defined by the requires in the manifest as far as I know, so this is its intended use. So it seems like a bug to me. But maybe it's in sprockets?

[ncri]

@josh, your comment seems to refer to another issue rather then this one: #2569

[ncri]

@josh, your comment seems to refer to another issue rather then this one: #2569

[ncri]

Could someone please reopen this? I believe it is not fixed by #2580 as that just adresses precompilation. But correct me if I'm wrong.

[ncri]

Could someone please reopen this? I believe it is not fixed by #2580 as that just adresses precompilation. But correct me if I'm wrong.

[ncri]

I can make an example app to illustrate the issue if it helps.

[ncri]

I can make an example app to illustrate the issue if it helps.

[ncri]

Oops, has been reopen already, a page refresh could help. :) thx

[ncri]

Oops, has been reopen already, a page refresh could help. :) thx

[guilleiguaran]

This still failing with Rails 3.1.0.rc6 and Sprockets 2.0.0.beta.14?

[guilleiguaran]

This still failing with Rails 3.1.0.rc6 and Sprockets 2.0.0.beta.14?

[ncri]

Yep.

[ncri]

Yep.

[spastorino]

@elstudio @ncri can you post an app which shows the issue?, thanks.

[spastorino]

@elstudio @ncri can you post an app which shows the issue?, thanks.

[elstudio]

Okay, here's that test app. It's a stoplight.

Source: https://github.com/elstudio/stoplight-rjs-test

Production on Heroku: http://blazing-mist-772.herokuapp.com/

When run in Development (without JS asset compression), the stoplight changes colors when you click on a color name. It uses RJS to do this.

In production, though (both locally and on Heroku), the UJS/RJS fails -- you'll see page loads, for one thing. And of course the colors don't change. If I turn off asset compression in production the UJS/RJS works fine.

This is a problem for Sprockets 2.0.0.beta.14 with both Rails 3.1.0.rc6 and today's Rails 3-1-0 branch, 283f5.

Let me know what I can do to help!

@spastorino, @ncri, @guilleiguaran

[elstudio]

Okay, here's that test app. It's a stoplight.

Source: https://github.com/elstudio/stoplight-rjs-test

Production on Heroku: http://blazing-mist-772.herokuapp.com/

When run in Development (without JS asset compression), the stoplight changes colors when you click on a color name. It uses RJS to do this.

In production, though (both locally and on Heroku), the UJS/RJS fails -- you'll see page loads, for one thing. And of course the colors don't change. If I turn off asset compression in production the UJS/RJS works fine.

This is a problem for Sprockets 2.0.0.beta.14 with both Rails 3.1.0.rc6 and today's Rails 3-1-0 branch, 283f5.

Let me know what I can do to help!

@spastorino, @ncri, @guilleiguaran

[ncri]

thanks @elstudio !

[ncri]

thanks @elstudio !

[spastorino]

Is this RJS only related? can I see an example without RJS?

[spastorino]

Is this RJS only related? can I see an example without RJS?

[spastorino]

@ncri your app was using RJS too?

[spastorino]

@ncri your app was using RJS too?

[ncri]

@spastorino Yes, but the issue is really about the asset pipeline's file compilation order. And that is completely independent of RJS.

[ncri]

@spastorino Yes, but the issue is really about the asset pipeline's file compilation order. And that is completely independent of RJS.

[spastorino]

@ncri I'd like to take a look to an app without RJS, if you can build one as an example please do it

[spastorino]

@ncri I'd like to take a look to an app without RJS, if you can build one as an example please do it

[guilleiguaran]

@ncri @elstudio the assets are concatened in alphabetical order when compression is on? I'm trying to reproduce it (without RJS) and I couldn't:

application.js:

//= require b
//= require a

b.js:

var a = 1;

a.js:

a = a + 1;

With config.assets.compress = false, I'm getting:

var a = 1;
a = a + 1;

and with config.assets.compress = true I'm getting:

var a=1;a=a+1;

I will check now if the use of external assets (ex. jquery_ujs) is changing this order

[guilleiguaran]

@ncri @elstudio the assets are concatened in alphabetical order when compression is on? I'm trying to reproduce it (without RJS) and I couldn't:

application.js:

//= require b
//= require a

b.js:

var a = 1;

a.js:

a = a + 1;

With config.assets.compress = false, I'm getting:

var a = 1;
a = a + 1;

and with config.assets.compress = true I'm getting:

var a=1;a=a+1;

I will check now if the use of external assets (ex. jquery_ujs) is changing this order

[guilleiguaran]

Just found the issue, it is caused by Uglifier (UglifyJS?) doing wrong compression with your files.

I modified the app created by @elstudio to use Closure as compressor and now it works fine

Sorry but there aren't nothing that we can do in Rails or Sprockets to solve it. You will need to contact to the authors of Uglifier/UglifyJS to get this fixed and use alternative compressors as temporary workaround

[guilleiguaran]

Just found the issue, it is caused by Uglifier (UglifyJS?) doing wrong compression with your files.

I modified the app created by @elstudio to use Closure as compressor and now it works fine

Sorry but there aren't nothing that we can do in Rails or Sprockets to solve it. You will need to contact to the authors of Uglifier/UglifyJS to get this fixed and use alternative compressors as temporary workaround

[ncri]

@guilleiguaran Great you found the issue. Just wondering if uglifier is the default compressor? As the issue appears when setting config.assets.compress = true even without specifiying a compressor.

[ncri]

@guilleiguaran Great you found the issue. Just wondering if uglifier is the default compressor? As the issue appears when setting config.assets.compress = true even without specifiying a compressor.

[guilleiguaran]

Yes, uglifier is the default compressor: https://github.com/rails/rails/blob/master/actionpack/lib/sprockets/railtie.rb#L52

[guilleiguaran]

Yes, uglifier is the default compressor: https://github.com/rails/rails/blob/master/actionpack/lib/sprockets/railtie.rb#L52

[ncri]

Okay, I can confirm my app works fine with closure. Just wonder if it would be better to set rails to a default that works properly? I don't know why uglifier was chosen as a default, I assume there are good reasons for it. However, if it's currently broken it might make sense to replace it by a more robust default for now.

[ncri]

Okay, I can confirm my app works fine with closure. Just wonder if it would be better to set rails to a default that works properly? I don't know why uglifier was chosen as a default, I assume there are good reasons for it. However, if it's currently broken it might make sense to replace it by a more robust default for now.

[guilleiguaran]

@lautis, can you check this, please?

[guilleiguaran]

@lautis, can you check this, please?

[lautis]

This seems to happen also when using raw UglifyJS without any Ruby involved. To be more specific, there seems to be some issues with name mangling feature of UglifyJS as config.assets.js_compressor = Uglifier.new(:mangle => false) works but results in larger file size.

[lautis]

This seems to happen also when using raw UglifyJS without any Ruby involved. To be more specific, there seems to be some issues with name mangling feature of UglifyJS as config.assets.js_compressor = Uglifier.new(:mangle => false) works but results in larger file size.

[elstudio]

I can confirm that using closure-compressor rather than uglifier also fixes Prototype UJS and RJS in a production app (http://tally4.com).

Might closure-compressor be a better default for new Rails 3.1 apps?

Thanks @guilleiguaran and @lautis for the detective work!

[elstudio]

I can confirm that using closure-compressor rather than uglifier also fixes Prototype UJS and RJS in a production app (http://tally4.com).

Might closure-compressor be a better default for new Rails 3.1 apps?

Thanks @guilleiguaran and @lautis for the detective work!

[lautis]

Closure Compiler is definitely better engineered than UglifyJS, but there are caveats:

• it requires Java
• Closure Compiler imposes restrictions on the JS you're allowed to write[1]
• compared to UglifyJS it's slow[2]

[1] http://code.google.com/closure/compiler/docs/limitations.html
[2] http://www.peterbe.com/plog/comparing-google-closure-with-uglifyjs

[lautis]

Closure Compiler is definitely better engineered than UglifyJS, but there are caveats:

• it requires Java
• Closure Compiler imposes restrictions on the JS you're allowed to write[1]
• compared to UglifyJS it's slow[2]

[1] http://code.google.com/closure/compiler/docs/limitations.html
[2] http://www.peterbe.com/plog/comparing-google-closure-with-uglifyjs

[ncri]

Well, just read around a bit and Uglifier does seem to be a very good choice. See for example:

http://badassjs.com/post/971960912/uglifyjs-a-fast-new-javascript-compressor-for-node-js
http://blog.foxxtrot.net/2010/12/a-comparison-of-javascript-compressors.html

However, if this is issue is not fixed, it's a no go for me.

Has anyone compared file sizes of uglifier generated js with mangle = false vs. mangle = true?
If it's negligible, using config.assets.js_compressor = Uglifier.new(:mangle => false) could be a good option for now (looks hackish though :)

[ncri]

Well, just read around a bit and Uglifier does seem to be a very good choice. See for example:

http://badassjs.com/post/971960912/uglifyjs-a-fast-new-javascript-compressor-for-node-js
http://blog.foxxtrot.net/2010/12/a-comparison-of-javascript-compressors.html

However, if this is issue is not fixed, it's a no go for me.

Has anyone compared file sizes of uglifier generated js with mangle = false vs. mangle = true?
If it's negligible, using config.assets.js_compressor = Uglifier.new(:mangle => false) could be a good option for now (looks hackish though :)

[guilleiguaran]

@elstudio @ncri I'm not sure if is a good idea change the standard compressor now since 3.1.0 will be released very soon and the others supported compressors (YUI and Closure) require Java.

On the next version of Rails we will have plugins (ex uglifier-rails, closure-rails, ...) to set default compressors and we will be able to change the compressor only changing the plugin in the Gemfile. Also we will be able to add custom compressors easily.

[guilleiguaran]

@elstudio @ncri I'm not sure if is a good idea change the standard compressor now since 3.1.0 will be released very soon and the others supported compressors (YUI and Closure) require Java.

On the next version of Rails we will have plugins (ex uglifier-rails, closure-rails, ...) to set default compressors and we will be able to change the compressor only changing the plugin in the Gemfile. Also we will be able to add custom compressors easily.

[NZKoz]

We could switch our own uglifier settings to turn mangle off. Has anyone been able to reproduce this with 'raw uglifier' and report it up stream?

[NZKoz]

We could switch our own uglifier settings to turn mangle off. Has anyone been able to reproduce this with 'raw uglifier' and report it up stream?

[lautis]

The problem exists with raw UglifyJS.

PrototypeJS tries to get argument names for JS functions using toString. Superclasses work by having a special argument name $super. Unfortunately, the name mangling feature of UglifyJS does break this and replaces $super with a to save bytes.

Update: this can be easily given by adding a exception to Uglifier, i.e. config.assets.js_comperssor = Uglifier.new(:except => ["$super"]). I'll add this as default in Uglifier.

[lautis]

The problem exists with raw UglifyJS.

PrototypeJS tries to get argument names for JS functions using toString. Superclasses work by having a special argument name $super. Unfortunately, the name mangling feature of UglifyJS does break this and replaces $super with a to save bytes.

Update: this can be easily given by adding a exception to Uglifier, i.e. config.assets.js_comperssor = Uglifier.new(:except => ["$super"]). I'll add this as default in Uglifier.

[guilleiguaran]

@NZKoz, @lautis is the author of uglifier and he confirmed that isn't a problem in uglifier neither, we must report the problem to UglifyJS authors.

I've send a Pull Request (#2721)

Update: Closed Pull Request in favor of @lautis solution

[guilleiguaran]

@NZKoz, @lautis is the author of uglifier and he confirmed that isn't a problem in uglifier neither, we must report the problem to UglifyJS authors.

I've send a Pull Request (#2721)

Update: Closed Pull Request in favor of @lautis solution

[guilleiguaran]

@lautis can you release soon a new version of Uglifier? (Rails 3.1 will be released very soon)

Upgrade to a new version of uglifier with the new default looks like the best solution for me.

[guilleiguaran]

@lautis can you release soon a new version of Uglifier? (Rails 3.1 will be released very soon)

Upgrade to a new version of uglifier with the new default looks like the best solution for me.

[lautis]

Uglifier 1.0.2 has been released, it should fix this issue (the test app seems to work)

[lautis]

Uglifier 1.0.2 has been released, it should fix this issue (the test app seems to work)

[guilleiguaran]

@lautis thanks for your help, I just updated the app to use last version of uglifier and is working fine: http://blooming-autumn-295.herokuapp.com/

@elstudio @ncri upgrade uglifier, test again and let us know

@josh @spastorino @NZKoz this can be closed

[guilleiguaran]

@lautis thanks for your help, I just updated the app to use last version of uglifier and is working fine: http://blooming-autumn-295.herokuapp.com/

@elstudio @ncri upgrade uglifier, test again and let us know

@josh @spastorino @NZKoz this can be closed

[NZKoz]

Wicked

[NZKoz]

Wicked

[elstudio]

Wonderful!

Works great for Tally4 -- Uglifier 1.0.2 makes compression work like a charm for PrototypeJS.

Thanks for the fix, @lautis! And the diagnosis outreach, @guilleiguaran!

[elstudio]

Wonderful!

Works great for Tally4 -- Uglifier 1.0.2 makes compression work like a charm for PrototypeJS.

Thanks for the fix, @lautis! And the diagnosis outreach, @guilleiguaran!

[ncri]

@lautis and @guilleiguaran: Yep, thanks a lot guys!

[ncri]

@lautis and @guilleiguaran: Yep, thanks a lot guys!

[rhulse]

In my upgraded app the order of JS files is not as per the manifest.

// Gem jquery-rails
//= require jquery.min
//= require jquery-ui.min
//= require jquery_ujs
//= require admin/jquery-slug
//= require admin/jquery.inline_label.min
//= require admin/chefs
//= require admin/recipe
//= require admin/tag
//= require admin/categories
//= require admin/presenter
//= require admin/status_changes
//= require admin/galleries
//= require admin/date_picker
//= require admin/pictures
//= require admin/pjax
//= require admin/core
//= require admin/jquery.fileupload
//= require admin/jquery.fileupload-ui
//= require admin/presenter_selector
//= require admin/programme_selector

But the order is (as far as I could track through the minified code):

admin/chefs
admin/recipe
admin/categories
admin/status_changes
admin/galleries
admin/date_picker
admin/pictures
admin/presenter
admin/programme
jquery.min
etc

It does not seem to be alphabetical or length-based ordering, but appears to be consistent with each compile operation.

This is Rails 3.1.0 and Uglifier 1.0.2

cc @guilleiguaran @NZKoz

[rhulse]

In my upgraded app the order of JS files is not as per the manifest.

// Gem jquery-rails
//= require jquery.min
//= require jquery-ui.min
//= require jquery_ujs
//= require admin/jquery-slug
//= require admin/jquery.inline_label.min
//= require admin/chefs
//= require admin/recipe
//= require admin/tag
//= require admin/categories
//= require admin/presenter
//= require admin/status_changes
//= require admin/galleries
//= require admin/date_picker
//= require admin/pictures
//= require admin/pjax
//= require admin/core
//= require admin/jquery.fileupload
//= require admin/jquery.fileupload-ui
//= require admin/presenter_selector
//= require admin/programme_selector

But the order is (as far as I could track through the minified code):

admin/chefs
admin/recipe
admin/categories
admin/status_changes
admin/galleries
admin/date_picker
admin/pictures
admin/presenter
admin/programme
jquery.min
etc

It does not seem to be alphabetical or length-based ordering, but appears to be consistent with each compile operation.

This is Rails 3.1.0 and Uglifier 1.0.2

cc @guilleiguaran @NZKoz

[guilleiguaran]

@rhulse can you report this issue in Sprockets issue tracker?

[guilleiguaran]

@rhulse can you report this issue in Sprockets issue tracker?

[rhulse]

Sure.

[rhulse]

Sure.

[rhulse]

Response: https://github.com/sstephenson/sprockets/issues/194

This is apparently uglifier, which according to the above thread is fixed.

Trying to narrow this down (by turning compression off to rule out sprockets) didn't work (see #3010)

[rhulse]

Response: https://github.com/sstephenson/sprockets/issues/194

This is apparently uglifier, which according to the above thread is fixed.

Trying to narrow this down (by turning compression off to rule out sprockets) didn't work (see #3010)

[spastorino]

@lautis can you check this ^^^ ? please

[spastorino]

@lautis can you check this ^^^ ? please

[rhulse]

FWIW, just tried swapping in Closure, and the order in the compiled file is now correct.

[rhulse]

FWIW, just tried swapping in Closure, and the order in the compiled file is now correct.

[lautis]

The original bug report here is misleading and you might be slightly confused about the nature of JavaScript. The ordering was never fixed. In fact, it was not even the issue.

The source files are "shuffled" because UglifyJS does what your browser would do anyway: it hoists function definitions and variable declarations on top of the binding scope. As a result, it may seem that some files are moved on the top of the minified file.

function() {
  function b() {}
  // some code here
  function a() {}

}

is equivalent to

function() {
  function a() {}
  function b() {}
  // some code here
}

There are some quite wonky ways to shoot yourself in the foot with these. For a better description, I recommend that you read http://www.adequatelygood.com/2010/2/JavaScript-Scoping-and-Hoisting.

I'm not intimately familiar with the reasoning behind why this is done in UglifyJS. Maybe it allows few bytes to be saved, perhaps there are some performance reasons or it could just be easier to write. Doesn't really matter, it's a safe thing to do.

@rhulse Is there actually any issues with the minified code?

[lautis]

The original bug report here is misleading and you might be slightly confused about the nature of JavaScript. The ordering was never fixed. In fact, it was not even the issue.

The source files are "shuffled" because UglifyJS does what your browser would do anyway: it hoists function definitions and variable declarations on top of the binding scope. As a result, it may seem that some files are moved on the top of the minified file.

function() {
  function b() {}
  // some code here
  function a() {}

}

is equivalent to

function() {
  function a() {}
  function b() {}
  // some code here
}

There are some quite wonky ways to shoot yourself in the foot with these. For a better description, I recommend that you read http://www.adequatelygood.com/2010/2/JavaScript-Scoping-and-Hoisting.

I'm not intimately familiar with the reasoning behind why this is done in UglifyJS. Maybe it allows few bytes to be saved, perhaps there are some performance reasons or it could just be easier to write. Doesn't really matter, it's a safe thing to do.

@rhulse Is there actually any issues with the minified code?

[rhulse]

Thanks @lautis for taking the time to explain. It makes perfect sense.

Yes, part of the code stopped working correctly - a jquery UI list sort. It did work, but was adding dozens of extra elements each time anything was moved. Interestingly the code works fine if I use Closure instead. I didn't write most of the code, but if you think it might be an actual bug, I can do some digging around.

[rhulse]

Thanks @lautis for taking the time to explain. It makes perfect sense.

Yes, part of the code stopped working correctly - a jquery UI list sort. It did work, but was adding dozens of extra elements each time anything was moved. Interestingly the code works fine if I use Closure instead. I didn't write most of the code, but if you think it might be an actual bug, I can do some digging around.

[lautis]

It does sound like a bug in UglifyJS/Uglifier. If you can debug further and perhaps generate a minimal failing example, it'd help. Issues should probably be reported to UglifyJS.

Some pointers:

• using eval and function.toString() likely lead to unexpected behavior when minified (though Closure is usually more anal about these)
• try less aggressive options, e.g. config.assets.js_compressor = Uglifier.new(:mangle => false, :squeeze => false)

[lautis]

It does sound like a bug in UglifyJS/Uglifier. If you can debug further and perhaps generate a minimal failing example, it'd help. Issues should probably be reported to UglifyJS.

Some pointers:

• using eval and function.toString() likely lead to unexpected behavior when minified (though Closure is usually more anal about these)
• try less aggressive options, e.g. config.assets.js_compressor = Uglifier.new(:mangle => false, :squeeze => false)