Skip to content
This repository

Make rake assets:precompile:nondigest optional #5379

Closed
wants to merge 8 commits into from
David Rice

(for applications with fully digested assets)

  • Improves speed of precompile significantly.
  • Therefore improving speed of deployments OOTB on Heroku etc.
  • assets:precompile:nondigest task still runs in Rake and warning issued
  • as Rake task still runs, rake task enhancements still possible
  • config.assets.nondigest_enabled setting disregarded if not present (backwards compatibility)
David Rice

This is actually in reference to a ticket I raised 5 months ago that is still open #3419

All of our applications on Rails 3 are fully using digested assets. With this patch it allows us to shave some time off of every deployment!

José Valim
Owner
David Rice

@josevalim I know, there are separate tasks already. My motives for this pull request are not to workaround heroku limitations, but instead good defaults.

Most PaaS services I've seen are focusing on integration with rake assets:precompile as the default.

Sure on some platforms it means simply just changing a rake task that gets executed. On Heroku for example, I'd need to configure all of those apps with custom build packs (not so simple).

So, it would be great that this is just handled by a configuration setting in Rails so the only task need run is rake assets:precompile in all scenarios.

This is also great for extensions to the asset pipeline for example asset_sync where we are now relying on integrating with certain tasks in the pipeline compilation because of how Kernal.exec is being used.

In my opinion having one way of compiling assets makes things a lot simpler (to the end users). As it doesn't matter what my configuration is (wether I use digests or nondigest assets) my deployment processes are still the same.

Brian Ewing

+1 for good defaults

José Valim
Owner
Guillermo Iguaran
Owner

please don't merge this (yet), I'm finishing the extraction of sprockets-rails from rails (probably I will finish it today)

David Rice

@guilleiguaran awesome!

@josevalim thanks.

I choose to implement this with Rails config vs ENV config based on the fact that the behaviour of rake assets:precompile is already influenced by the Rails config. I think this is the best approach as it means no matter where I run rake assets:precompile the result will be the same.

In my mind the config.assets.digest should determine wether to run precompile:primary or precompile:nondigest not both. The only reason we're doing both is to help people who haven't got all their asset references fixed to return the digested links in all aspects of their application... have I got that right? I've found that's great for bootstrapping a migration of a Rails 3.0 to 3.1 application. However it's really not helping people if they start using a CDN and are referencing nondigest assets unknowingly. It's much better if this fails fast and is fixed on a first deploy to a staging setup.

That's my opinion, and was my first port of call for implementing this but I thought it would be a much more difficult change, that's why I put together this patch which offers all the existing functionality and unobtrusively allows "taking off the training wheels" when you don't need them :)

Whilst it would be less preferable, I'd be up for reworking this to use ENV variables if that was the consensus!

Regarding asset_sync if I've made a mistake in our integration, I'm all ears. We're not actually hooking into both tasks: as of 0.3.0 (Rails 3.2.x) we're enhancing only the assets:precompile:nondigest task as it will always be ran and ran last.

Due to how the usage of Kernal.exec was introduced. We stopped enhancing the assets:precompile task but left it in the default Rake task for backwards compatibility with Rails 3.1.x releases to kick in only if the assets:precompile:nondigest task does not exist. Rather than sniffing Rails versions specifically.

I actually already fired a mail to Heroku for their thoughts on this. I totally agree, I'd love if their deployment strategies were a lot more customisable too.

José Valim
Owner

The only reason we're doing both is to help people who haven't got all their asset references fixed to return the digested links in all aspects of their application... have I got that right?

The reason we are compiling both is because some times it is hard to access the digest version, like in emails, errors pages and other static pages. Indeed, referencing an asset without digest by accident is unfortunate, but removing the non digest version is the wrong answer to this problem.

That said, I don't feel strong about this feature but I am fine with an ENV variable since it feels less intrusive than adding a new option to config.assets.

José Valim
Owner

Ok, I just got feedback from other Rails Core Members and some of them are actually interested in this feature. But we need a few changes to make everything work well. If you are up to the challenge we can work on this together, I can provide any guidance you need. Here are the concerns:

1) We would need to move 500.html, 404.html and friends to app/assets so they would be able to use the digested version of the assets;

2) I think development accepts both /assets/email.png and /assets/DIGEST-email.png. This is bad because if you accidentally use the non digest version in development, you will just catch the error in production;

If we can fix those two points, I believe we can change how this rake task behaves for Rails 4 to not include the non digest versions by default. If someone want the digest version, they can run rake assets:precompile:nondigest manually. Wdyt?

David Rice

Great. Can help out on this for sure.

Both those points, I agree. If you've any ideas on implementing would be good to hear. Any existing tickets out there with further info?

I'll look into 2) though it sounds easy enough.

Biggest concern would be 500.html and 404.html. I've had a think about how this should work...

  • Create a new assets directory app/assets/static
  • .html files in this directory are pre-processed / rendered (to allow .erb, .haml) extensions etc.
  • Generate defaults like app/assets/static/404.html.erb
  • files receive same scope for rendering as other assets to allow for digests
  • files are rendered to public/assets as normal
  • some special handling in the default error handler to render public/assets/404-DIGEST.html instead of public/404.html

Does that sound about right? not sure at the minute if there'd be much of an impact on routes or .htaccess files?

Sam Soffes

@davidjrice that sounds amazing! I basically do that now, only all manually. This would be great.

José Valim
Owner

Yes, the default error handler will need to be improved. Here is the current implementation: https://github.com/rails/rails/blob/master/actionpack/lib/action_dispatch/middleware/public_exceptions.rb

For me, the biggest concern regarding 2) is: can it be an issue for many applications if the 404 page is actually hidden under assets/404-DIGEST.html? Well, if there is an issue they could use assets:precompile:nondigest to get both versions, I am just wondering if this would be the common case or the exception.

@jeremy: you commented that we could also include robots.txt at app/assets/static, but given that assets are compiled to public/assets, that wouldn't actually work. Any ideas in mind? Maybe we could tell sprockets that all files at app/assets/static should actually go to public/ ? Is it a good idea?

Jeremy Kemper
Owner

They should probably still generate digested assets in public/assets then symlink in from public/ so 404.html, robots.txt, etc always reference the latest file.

What do you think about a dir for "root" assets, like app/assets/static or app/assets/public?

David Rice

@jeremy @josevalim I've put together a first pass at getting this to work. I've slightly cheated and used a static middleware to lazily serve content in app/assets/public for development mode. Main bit I'm unsure about for now is how to get the static assets rendering with a .erb extension if present.

Would you suggest a custom middleware?

I still need to ensure referencing assets without a digest in emailers fails correctly.

José Valim
Owner

@davidjrice the pull request is "dirty". Could you please rebase it?

added some commits March 11, 2012
David Rice Make rake assets:precompile:nondigest optional (for applications with…
… fully digested assets)

* Improves speed of precompile significantly.
* Therefore improving speed of deployments OOTB on Heroku etc.
* assets:precompile:nondigest task still runs in Rake and warning issued
* as Rake task still runs, rake task enhancements still possible
* config.assets.nondigest_enabled setting disregarded if not present (backwards compatibility)
d1a8c2e
David Rice Move all static assets from /public to /app/assets/public in Railties
Also remove stylesheets directory and make public an empty dir
636d014
David Rice Generated public directory should have a .gitkeep, add static middlew…
…are to serve app/assets/public in development mode only
142d044
David Rice Only apply app/assets/public middleware if config.serve_static_assets 0d8112d
David Rice Symlink all assets generated from app/assets/public into public 1d0ea03
David Rice

@josevalim sorry, my bad, fixed this now.

José Valim
Owner

@davidjrice thanks! Although I am skeptical about symlinks since they don't work on windows. We would need to check for alternatives on windows. Adding another middleware for app/assets/public also feels weird and as you said it will cause issues for rendering ERB files.

@jeremy any ideas on solving those issues?

David Rice
  • We could just copy the file to public on windows
  • The additional static middleware for app/assets/public is only a temporary POC hack :)
  • A middleware that would generate them from ERB if a .html.erb extension exists or simply serve the static file if it doesn't would be preferable... i'm just not sure of the best way of doing this yet.
José Valim
Owner

@davidjrice Since it is inside app/assets, a developer would expect not only .erb to work, but also .haml and so on.

Ideally I would pipe the request to sprockets and let sprockets do its thing but afaik there is no way for us to tell sprockets to serve only assets for public.

added some commits March 13, 2012
David Rice Only include app/assets/public static middleware when necessary. Impr…
…ove public exceptions middleware to render from app/assets/public as a fallback

* Include app/assets/public static middleware only (if config.serve_static_assets && config.assets.enabled && config.assets.compile) or in development mode
* Exception middleware checks public first and falls back to app/assets/public
975c261
David Rice Improve public exception middleware, add attr for assets_public_path 067a73c
David Rice When symlinking files under assets:precompile handle if there are alr…
…eady existing symlinks by unlinking them first
258ac20
David Rice

@josevalim yeah handling erb, haml etc. would be best.

Thought about integrating with sprockets, not sure where to start with that. (if anyone has any ideas I'd love to hear!)

It's possibly something that needs to go into sprokets-rails though?

José Valim
Owner

Let's summon @josh and @sstephenson and see if they have any ideas/feedback about this.

Joshua Peek
Collaborator
josh commented March 13, 2012

The amount of flags and options in rails for the asset stuff is insane now.

Just put your "nondigest" stuff in public.

David Rice

@josh I know, the flag initially introduced at the start of this pull request was only intended to be an unobtrusive addition so as not to affect any existing functionality.

However after discussing with @josevalim the direction I am now heading towards is moving all the default rails static assets in public to within the asset pipeline so they may be precompiled. Being able to have these files as part of the application, pre-generated on deployment offers some great flexibility.

This could be taken a step further with the functionality I describe in this comment. Which would make rake assets:precompile:all and the new config I proposed unnecessary and rake assets:precompile would simply run either digest or nondigest only, depending on the setting of config.assets.digest. This would in turn simplify the assets:precompile task itself and not require shenanigans with Kernal.exec and would speed up execution.

Guillermo Iguaran
Owner

Closing this here since Sprockets integration was extracted from Rails edge and this couldn't be done in 3-2-stable. Please re-open your PR in sprockets-rails to continue the discussion (and please reference this PR)

Guillermo Iguaran guilleiguaran closed this April 27, 2012
David Rice
Guillermo Iguaran guilleiguaran reopened this April 27, 2012
Egor Homakov

@davidjrice just noticed. keep it up - it will also fix #6421

Guillermo Iguaran

Someone can elaborate a good reason to keep non-digested assets outside of public?

Like @josh I would prefer move non-digested stuff to public and compile only digested version of stuff in app/assets

David Rice

@guilleiguaran think you missed the whole point. If your read the above discussion again you'll see the goal is to generate files such as 404.html at precompilation time.

This allows usage of digested asset paths in 404/500 etc.
This also simplifies the whole digest / non-digest precompilation step and therefore reducing compilation time by 50%

As the last remnants of public/ are static files, however they need treated differently from assets as they are usually used on the server the rails app is served from, as opposed to assets which predominantly get shuffled to an asset host

Nathan Broadbent

Hi, what do you think about #7866 as an alternative solution?

This patch means that only digest assets would be compiled, and non-digest assets are generated from those in only a few milliseconds. In that case, there would probably be no need to make them optional.

@davidjrice - I've also released these changes in a gem for Rails 3.2.8, at https://github.com/ndbroadbent/turbo-sprockets-rails3. I would be grateful if you could try it out and let me know if it solves your problem with long deployments.

David Rice

@ndbroadbent as mentioned, in #7866 I don't think it's an alternative solution. More of an extension to what I have here. I would definitely want a combination of the two in Rails 4!

The goal of this pull request should probably be restated that it is to Move the remaining rails public directory assets into the asset pipeline

Joshua Peek
Collaborator

The goal of this pull request should probably be restated that it is to Move the remaining rails public directory assets into the asset pipeline

No.

Dmitry Vorotilin route referenced this pull request in rails/sprockets-rails October 17, 2012
Closed

Last sprockets changes #9

Steve Klabnik
Collaborator

Because sprockets-rails was moved out of master, there's no way this is going in. Therefore, I'm closing. Please re-submit to sprockets-rails and/or a backport fix to 3-2-stable.

Steve Klabnik steveklabnik closed this November 18, 2012
Johnny Shields johnnyshields referenced this pull request in rails/sprockets-rails November 13, 2013
Closed

Ability to skip non-digest assets? #97

Johnny Shields

Just a note for those who arrive here via Google: sprockets-rails (used in Rails 4.0.0+) no longer compiles non-digest assets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 8 unique commits by 1 author.

Mar 13, 2012
David Rice Make rake assets:precompile:nondigest optional (for applications with…
… fully digested assets)

* Improves speed of precompile significantly.
* Therefore improving speed of deployments OOTB on Heroku etc.
* assets:precompile:nondigest task still runs in Rake and warning issued
* as Rake task still runs, rake task enhancements still possible
* config.assets.nondigest_enabled setting disregarded if not present (backwards compatibility)
d1a8c2e
David Rice Move all static assets from /public to /app/assets/public in Railties
Also remove stylesheets directory and make public an empty dir
636d014
David Rice Generated public directory should have a .gitkeep, add static middlew…
…are to serve app/assets/public in development mode only
142d044
David Rice Only apply app/assets/public middleware if config.serve_static_assets 0d8112d
David Rice Symlink all assets generated from app/assets/public into public 1d0ea03
David Rice Only include app/assets/public static middleware when necessary. Impr…
…ove public exceptions middleware to render from app/assets/public as a fallback

* Include app/assets/public static middleware only (if config.serve_static_assets && config.assets.enabled && config.assets.compile) or in development mode
* Exception middleware checks public first and falls back to app/assets/public
975c261
David Rice Improve public exception middleware, add attr for assets_public_path 067a73c
David Rice When symlinking files under assets:precompile handle if there are alr…
…eady existing symlinks by unlinking them first
258ac20
This page is out of date. Refresh to see the latest.
14  actionpack/lib/action_dispatch/middleware/public_exceptions.rb
@@ -2,20 +2,28 @@ module ActionDispatch
2 2
   # A simple Rack application that renders exceptions in the given public path.
3 3
   class PublicExceptions
4 4
     attr_accessor :public_path
  5
+    attr_accessor :assets_public_path
5 6
 
6 7
     def initialize(public_path)
7 8
       @public_path = public_path
  9
+      @assets_public_path = "#{Rails.root}/app/assets/public"
8 10
     end
9 11
 
10 12
     def call(env)
11  
-      status      = env["PATH_INFO"][1..-1]
12  
-      locale_path = "#{public_path}/#{status}.#{I18n.locale}.html" if I18n.locale
13  
-      path        = "#{public_path}/#{status}.html"
  13
+      status       = env["PATH_INFO"][1..-1]
  14
+      locale_path  = "#{public_path}/#{status}.#{I18n.locale}.html" if I18n.locale
  15
+      path         = "#{public_path}/#{status}.html"
  16
+      locale_asset = "#{assets_public_path}/#{status}.html" if I18n.locale
  17
+      asset        = "#{assets_public_path}/#{status}.html"
14 18
 
15 19
       if locale_path && File.exist?(locale_path)
16 20
         render(status, File.read(locale_path))
17 21
       elsif File.exist?(path)
18 22
         render(status, File.read(path))
  23
+      elsif locale_asset && File.exist?(locale_asset)
  24
+        render(status, File.read(locale_asset))
  25
+      elsif File.exist(asset)
  26
+        render(status, File.read(asset))
19 27
       else
20 28
         [404, { "X-Cascade" => "pass" }, []]
21 29
       end
19  actionpack/lib/sprockets/assets.rake
@@ -30,6 +30,18 @@ namespace :assets do
30 30
   end
31 31
 
32 32
   namespace :precompile do
  33
+    def symlink_public_assets(digest=nil)
  34
+      root = "#{Rails.root}/app/assets/public/"
  35
+      Dir["#{root}*"].each do |p|
  36
+        origin = p
  37
+        path = p.gsub(root, '')
  38
+        source = File.expand_path(File.join(Rails.public_path, ActionController::Base.helpers.asset_path(path)))
  39
+        destination = "#{Rails.public_path}/#{path}"
  40
+        File.unlink(destination) if File.exists?(destination) && File.symlink?(destination)
  41
+        File.symlink(source, destination)
  42
+      end
  43
+    end
  44
+
33 45
     def internal_precompile(digest=nil)
34 46
       unless Rails.application.config.assets.enabled
35 47
         warn "Cannot precompile assets if sprockets is disabled. Please set config.assets.enabled to true"
@@ -54,6 +66,7 @@ namespace :assets do
54 66
                                                :digest => config.assets.digest,
55 67
                                                :manifest => digest.nil?)
56 68
       compiler.compile
  69
+      symlink_public_assets(config.assets.digest)
57 70
     end
58 71
 
59 72
     task :all do
@@ -71,7 +84,11 @@ namespace :assets do
71 84
     end
72 85
 
73 86
     task :nondigest => ["assets:cache:clean"] do
74  
-      internal_precompile(false)
  87
+      if Rails.application.config.assets.nondigest_enabled || Rails.application.config.assets.nondigest_enabled.nil?
  88
+        internal_precompile(false)
  89
+      else
  90
+        warn "Skipping assets:precompile:nondigest set config.assets.nondigest_enabled to true if required"
  91
+      end
75 92
     end
76 93
   end
77 94
 
2  railties/guides/code/getting_started/config/application.rb
@@ -48,6 +48,8 @@ class Application < Rails::Application
48 48
 
49 49
     # Enable the asset pipeline.
50 50
     config.assets.enabled = true
  51
+    # Enable automatic precompilation of nondigest assets
  52
+    config.assets.nondigest_enabled = true
51 53
 
52 54
     # Version of your assets, change this if you want to expire all your assets.
53 55
     config.assets.version = '1.0'
2  railties/guides/source/asset_pipeline.textile
Source Rendered
@@ -637,6 +637,8 @@ In +application.rb+:
637 637
 <erb>
638 638
 # Enable the asset pipeline
639 639
 config.assets.enabled = true
  640
+# Enable automatic precompilation of nondigest assets
  641
+config.assets.nondigest_enabled = true
640 642
 
641 643
 # Version of your assets, change this if you want to expire all your assets
642 644
 config.assets.version = '1.0'
2  railties/guides/source/configuring.textile
Source Rendered
@@ -139,6 +139,8 @@ Rails 3.1, by default, is set up to use the +sprockets+ gem to manage assets wit
139 139
 
140 140
 * +config.assets.enabled+ a flag that controls whether the asset pipeline is enabled. It is explicitly initialized in +config/application.rb+.
141 141
 
  142
+* +config.assets.nondigest_enabled+ a flag that controls whether the automatic precompilation of nondigest asset variants is enabled. It is explicitly initialized and set to true in +config/application.rb+.
  143
+
142 144
 * +config.assets.compress+ a flag that enables the compression of compiled assets. It is explicitly set to true in +config/production.rb+.
143 145
 
144 146
 * +config.assets.css_compressor+ defines the CSS compressor to use. It is set by default by +sass-rails+. The unique alternative value at the moment is +:yui+, which uses the +yui-compressor+ gem.
3  railties/lib/rails/application.rb
@@ -234,6 +234,9 @@ def default_middleware_stack
234 234
         end
235 235
 
236 236
         if config.serve_static_assets
  237
+          if (config.assets.enabled && config.assets.compile) || Rails.env.development?
  238
+            middleware.use ::ActionDispatch::Static, "app/assets/public", false
  239
+          end
237 240
           middleware.use ::ActionDispatch::Static, paths["public"].first, config.static_cache_control
238 241
         end
239 242
 
2  railties/lib/rails/generators/rails/app/app_generator.rb
@@ -96,7 +96,7 @@ def log
96 96
     end
97 97
 
98 98
     def public_directory
99  
-      directory "public", "public", :recursive => false
  99
+      empty_directory_with_gitkeep "public"
100 100
     end
101 101
 
102 102
     def script
0  ...ls/generators/rails/app/templates/public/404.html → ...rs/rails/app/templates/app/assets/public/404.html
File renamed without changes
0  ...ls/generators/rails/app/templates/public/422.html → ...rs/rails/app/templates/app/assets/public/422.html
File renamed without changes
0  ...ls/generators/rails/app/templates/public/500.html → ...rs/rails/app/templates/app/assets/public/500.html
File renamed without changes
0  ...generators/rails/app/templates/public/favicon.ico → ...rails/app/templates/app/assets/public/favicon.ico
File renamed without changes
0  .../generators/rails/app/templates/public/index.html → .../rails/app/templates/app/assets/public/index.html
File renamed without changes
0  .../generators/rails/app/templates/public/robots.txt → .../rails/app/templates/app/assets/public/robots.txt
File renamed without changes
2  railties/lib/rails/generators/rails/app/templates/config/application.rb
@@ -64,6 +64,8 @@ class Application < Rails::Application
64 64
 <% unless options.skip_sprockets? -%>
65 65
     # Enable the asset pipeline.
66 66
     config.assets.enabled = true
  67
+    # Enable automatic precompilation of nondigest assets
  68
+    config.assets.nondigest_enabled = true
67 69
 
68 70
     # Version of your assets, change this if you want to expire all your assets.
69 71
     config.assets.version = '1.0'
0  ...app/templates/public/stylesheets/.empty_directory → ...ators/rails/app/templates/public/.empty_directory
File renamed without changes
28  railties/test/application/assets_test.rb
@@ -461,6 +461,34 @@ class ::PostsController < ActionController::Base ; end
461 461
       assert_equal 0, files.length, "Expected application.js asset to be removed, but still exists"
462 462
     end
463 463
 
  464
+    test "nondigest assets are not precompiled if config.assets.nondigest_enabled set to false" do
  465
+      app_file "app/assets/application.js", "alert();"
  466
+      add_to_config "config.assets.compile = true"
  467
+      add_to_config "config.assets.digest = true"
  468
+      add_to_config "config.assets.nondigest_enabled = false"
  469
+
  470
+      quietly do
  471
+        Dir.chdir(app_path){ `bundle exec rake assets:clean assets:precompile` }
  472
+      end
  473
+
  474
+      files = Dir["#{app_path}/public/assets/application.js"]
  475
+      assert_equal 0, files.length, "Expected application.js asset not to be generated, but was found"
  476
+    end
  477
+
  478
+    test "nondigest assets are compiled if config.assets.nondigest_enabled not set" do
  479
+      app_file "app/assets/application.js", "alert();"
  480
+      add_to_config "config.assets.compile = true"
  481
+      add_to_config "config.assets.digest = true"
  482
+      add_to_config "config.assets.nondigest_enabled = nil"
  483
+
  484
+      quietly do
  485
+        Dir.chdir(app_path){ `bundle exec rake assets:clean assets:precompile` }
  486
+      end
  487
+
  488
+      files = Dir["#{app_path}/public/assets/application.js"]
  489
+      assert_equal 1, files.length, "Expected application.js asset to be generated, but none found"
  490
+    end
  491
+
464 492
     test "asset urls should use the request's protocol by default" do
465 493
       app_with_assets_in_view
466 494
       add_to_config "config.asset_host = 'example.com'"
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.