Allow assets to be fetched without processing, in order to generate a digest of their sources #367

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
6 participants
@ndbroadbent

This feature lets us determine if an asset's sources have changed. If not, then we don't need to recompile and recompress that asset.

If :process => false is passed as an option to find_asset, the only processors that are run are Sprockets Directives and ERB. Other processors, such as coffeescript and compression, are removed from the chain. This lets us quickly concatenate the raw dependencies and calculate the digest of the result. We can compare this digest on subsequent runs to determine if the file has changed. I've added a new UnprocessedAsset class, which is basically just ProcessedAsset minus processors. UnprocessedAsset and ProcessedAsset both inherit from AssetWithDependencies.

I haven't touched the Sprockets manifest generation code, so please let me know if you want me to update that to include these source digests.

This PR is a dependency for the changes at sprockets-rails PR #21, which speeds up the Rails asset pipeline for unchanged assets.

Thanks for your time!

@ndbroadbent ndbroadbent referenced this pull request in capistrano/capistrano Sep 27, 2012

Closed

built-in rails asset compile: skip if no changes to assets? #227

Allow assets to be concatenated unprocessed to generate a digest of t…
…heir sources.

This lets us determine if an asset's sources have changed.
If not, then we don't need to recompile and recompress that asset.
If :process => false is passed as an option, the only processors
that are run are Sprockets Directives and ERB.
Other processors, such as coffeescript and compression, are removed.
@fbernier

This comment has been minimized.

Show comment Hide comment
@fbernier

fbernier Oct 1, 2012

+1. Just fell on a bug in our production environment where find_asset would throw an error with uglifier not being found when looking for a .js file. this pull request would allow us to use find_asset('myasset.js', :process => false) with more speed and no error.

fbernier commented Oct 1, 2012

+1. Just fell on a bug in our production environment where find_asset would throw an error with uglifier not being found when looking for a .js file. this pull request would allow us to use find_asset('myasset.js', :process => false) with more speed and no error.

@ndbroadbent

This comment has been minimized.

Show comment Hide comment
@ndbroadbent

ndbroadbent Oct 1, 2012

That's a very interesting use case, thanks for mentioning it! I'm glad this feature could have uses outside of just speeding up asset compilation. Hope someone will be able to review it soon :)

That's a very interesting use case, thanks for mentioning it! I'm glad this feature could have uses outside of just speeding up asset compilation. Hope someone will be able to review it soon :)

@ndbroadbent

This comment has been minimized.

Show comment Hide comment
@ndbroadbent

ndbroadbent Oct 1, 2012

@fbernier, have you tested that this pull request does get rid of your uglifier error? If uglifier is still being referenced too early, I would be happy to look at adding some code to handle the exception if :process is false.

@fbernier, have you tested that this pull request does get rid of your uglifier error? If uglifier is still being referenced too early, I would be happy to look at adding some code to handle the exception if :process is false.

@ndbroadbent ndbroadbent closed this Oct 1, 2012

@ndbroadbent ndbroadbent reopened this Oct 1, 2012

@fbernier

This comment has been minimized.

Show comment Hide comment
@fbernier

fbernier Oct 1, 2012

No, I have not tested it =/. I assumed it would fix my problem, but it may need some more tweaking as you mention. I Will try to test it out quite soon, but for now I have used workarounds since I could not wait for this to be merged in.

fbernier commented Oct 1, 2012

No, I have not tested it =/. I assumed it would fix my problem, but it may need some more tweaking as you mention. I Will try to test it out quite soon, but for now I have used workarounds since I could not wait for this to be merged in.

@neersighted

This comment has been minimized.

Show comment Hide comment
@neersighted

neersighted Oct 1, 2012

👍

👍

@josh

This comment has been minimized.

Show comment Hide comment
@josh

josh Oct 10, 2012

Contributor

This feature lets us determine if an asset's sources have changed. If not, then we don't need to recompile and recompress that asset.

Is already a core feature of sprockets.

All the ProcessedAsset stuff is on the way out. The future source map path is going to kill all the different asset types.

So this isn't the direction things are moving.

Contributor

josh commented Oct 10, 2012

This feature lets us determine if an asset's sources have changed. If not, then we don't need to recompile and recompress that asset.

Is already a core feature of sprockets.

All the ProcessedAsset stuff is on the way out. The future source map path is going to kill all the different asset types.

So this isn't the direction things are moving.

@josh josh closed this Oct 10, 2012

@guilleiguaran

This comment has been minimized.

Show comment Hide comment
@guilleiguaran

guilleiguaran Oct 10, 2012

This feature lets us determine if an asset's sources have changed. If not, then we don't need to recompile and recompress that asset.

I could be wrong, but I think this can be achieved without change anything in sprockets and just skipping entirely undigested assets compiling (done already in josh's sprockets-rails, I'm porting it in main sprockets-rails) and setting the assets cache property to avoid sharing same assets cache between different environments (done in rails master), right?

This feature lets us determine if an asset's sources have changed. If not, then we don't need to recompile and recompress that asset.

I could be wrong, but I think this can be achieved without change anything in sprockets and just skipping entirely undigested assets compiling (done already in josh's sprockets-rails, I'm porting it in main sprockets-rails) and setting the assets cache property to avoid sharing same assets cache between different environments (done in rails master), right?

@ndbroadbent

This comment has been minimized.

Show comment Hide comment
@ndbroadbent

ndbroadbent Oct 10, 2012

@josh, I agree, your source maps feature looks awesome! I noticed that before, and would have definitely used it if it was merged in, so I'm looking forward to that. My sprockets-rails pull request will only need some minor changes to work with source maps.

I don't think this is already a core feature of sprockets, unless you were talking about your source maps work. As far as I can tell, there was no way to build a list of asset dependencies without processing and compressing the final asset. Please let me know if there's a way to do that already, and I'll update the sprockets-rails pull request.

@guilleiguaran - I agree that skipping non-digest assets would be better, and I don't really mind either way. I'll probably just add some images to public/ when I need to have a permanent link for email newsletters and logos.
I'm not sure what you mean about not sharing the assets cache between environments. Will that solve the problem of not recompiling assets that haven't changed?

@josh, I agree, your source maps feature looks awesome! I noticed that before, and would have definitely used it if it was merged in, so I'm looking forward to that. My sprockets-rails pull request will only need some minor changes to work with source maps.

I don't think this is already a core feature of sprockets, unless you were talking about your source maps work. As far as I can tell, there was no way to build a list of asset dependencies without processing and compressing the final asset. Please let me know if there's a way to do that already, and I'll update the sprockets-rails pull request.

@guilleiguaran - I agree that skipping non-digest assets would be better, and I don't really mind either way. I'll probably just add some images to public/ when I need to have a permanent link for email newsletters and logos.
I'm not sure what you mean about not sharing the assets cache between environments. Will that solve the problem of not recompiling assets that haven't changed?

@josh

This comment has been minimized.

Show comment Hide comment
@josh

josh Oct 10, 2012

Contributor

So its clear, Sprockets 3.x (with source maps) will have NO "non-digest" support at all. The server will only route digested paths. Even dev mode will generate digests.

I'll probably just add some images to public/ when I need to have a permanent link for email newsletters and logos.

YES, do this. app/assets for dynamic stuff and public/ for purely static non-managed assets.

Contributor

josh commented Oct 10, 2012

So its clear, Sprockets 3.x (with source maps) will have NO "non-digest" support at all. The server will only route digested paths. Even dev mode will generate digests.

I'll probably just add some images to public/ when I need to have a permanent link for email newsletters and logos.

YES, do this. app/assets for dynamic stuff and public/ for purely static non-managed assets.

@ndbroadbent

This comment has been minimized.

Show comment Hide comment
@ndbroadbent

ndbroadbent Oct 12, 2012

@guilleiguaran - Thanks for the mention in your blog post :) Now I understand what you mean about sharing the same assets cache between different environments. It looks like the cache will detect changes in dependencies by checking the modification times of files, but that will miss a few cases where recompiling is required.

There are cases where asset processors handle their own internal 'require directives', so sprockets will only see one cached asset even if that asset imports other files. It won't be able to detect when imported files have changed, so it won't recompile the asset. An important example is @import statements in SASS and Less assets. I'm handling this in my gem by recursively replacing those imports with the contents of the imported files.

I'm also running the ERB processor every time, because you can do things like process a directory of icons into CSS with data-uri background images. If you add new icons to the directory, the 'source digest' of the asset would change, but the actual asset file would remain unmodified, so it won't be recompiled. Most assets won't use ERB, and it's pretty fast most of the time, so I think it's important to handle those cases.

What about improving the caching in Sprockets to handle these cases by using source digests instead of modification times? Then, the development environment would also get the speed improvements.

@guilleiguaran - Thanks for the mention in your blog post :) Now I understand what you mean about sharing the same assets cache between different environments. It looks like the cache will detect changes in dependencies by checking the modification times of files, but that will miss a few cases where recompiling is required.

There are cases where asset processors handle their own internal 'require directives', so sprockets will only see one cached asset even if that asset imports other files. It won't be able to detect when imported files have changed, so it won't recompile the asset. An important example is @import statements in SASS and Less assets. I'm handling this in my gem by recursively replacing those imports with the contents of the imported files.

I'm also running the ERB processor every time, because you can do things like process a directory of icons into CSS with data-uri background images. If you add new icons to the directory, the 'source digest' of the asset would change, but the actual asset file would remain unmodified, so it won't be recompiled. Most assets won't use ERB, and it's pretty fast most of the time, so I think it's important to handle those cases.

What about improving the caching in Sprockets to handle these cases by using source digests instead of modification times? Then, the development environment would also get the speed improvements.

@jrochkind

This comment has been minimized.

Show comment Hide comment
@jrochkind

jrochkind Oct 12, 2012

Taking non-digest-named asset support out of sprockets-rails is going to be a huge pain for some use cases -- presumably those same use cases that had the non-digest-named version in there in the first place, no?

I'm not sure what you mean by 'static assets' -- any asset can change with future versions of the app/gem. One reason to put assets in the pipeline is to get digesting, sure. But another reason is to get the compilation -- sass, coffeescript, minification, building a single asset file from multiple source files. Sometimes you need compilation but need to reveal the lastest version of the asset with a non-digested-name. My cases involve javascript "widgets" that need to be referenced by third-parties at a static location. My case is probably not common, but there must be some reason they were there in the first place, right? What were the use cases that justified the non-digested-name output in the first place, and are they no longer valid/common?

If I have an asset that is compiled (sass, coffeescript, etc), but does need to be available at a persistent location -- you're saying every time it changes, I should move a copy of it myself over to ./public and check it into source repo? Why is this a good idea?

Taking non-digest-named asset support out of sprockets-rails is going to be a huge pain for some use cases -- presumably those same use cases that had the non-digest-named version in there in the first place, no?

I'm not sure what you mean by 'static assets' -- any asset can change with future versions of the app/gem. One reason to put assets in the pipeline is to get digesting, sure. But another reason is to get the compilation -- sass, coffeescript, minification, building a single asset file from multiple source files. Sometimes you need compilation but need to reveal the lastest version of the asset with a non-digested-name. My cases involve javascript "widgets" that need to be referenced by third-parties at a static location. My case is probably not common, but there must be some reason they were there in the first place, right? What were the use cases that justified the non-digested-name output in the first place, and are they no longer valid/common?

If I have an asset that is compiled (sass, coffeescript, etc), but does need to be available at a persistent location -- you're saying every time it changes, I should move a copy of it myself over to ./public and check it into source repo? Why is this a good idea?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment