Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Detect asset changes by saving source_digests in manifest.yml, and speed up non-digest assets #21

Closed
wants to merge 2 commits into from
@ndbroadbent

This feature speeds up the asset pipeline by only recompiling assets whose dependencies have changed. It does this by changing manifest.yml to include :source_digests as well as the digests for compiled assets under the :digest_files key.

This pull request depends on two others: sprockets #367 and rails #7767 (guides update)

Source Digests

By computing source digests, we can quickly determine if the asset needs to be recompiled or not. If nothing has changed in the sources of an asset's dependencies, then compilation is skipped for that asset. Source digests are calculated by removing all processors except ERB, and disabling compression, so the contents of source files are concatenated and hashed. Please see pull request #367 on Sprockets for the required changes.

Generating Non-digest Assets

Instead of compiling twice for both digest and non-digest assets, non-digest assets are now generated by stripping the digests out of the already compiled css and js files. This means we don't have to process and compress the same files twice. I'm making the assumption that asset paths are the only differences between digest/non-digest assets, but please let me know if you can see any problems with this approach.

Rails 3.2.x Support

I've prepared patches for Rails 3.2.8 and Sprockets 2.1.3 on branches named rails3_assets_speedup. I've extracted these patches into a gem called turbo-sprockets-rails3, so that myself and others can start using the feature.
Do you think there's any chance of this feature making it into the Rails 3.2.9 release?

Tests

All the tests are passing for both sprockets and sprockets-rails. I've added one or two tests, but there already seemed to be decent coverage for the areas I was changing. Please let me know if more tests are needed.

Testing this pull request

You'll need to add the following lines to the Gemfile of your Rails 4 app:

gem 'sprockets',       github: 'ndbroadbent/sprockets',       branch: 'allow_assets_without_processing'
gem 'sprockets-rails', github: 'ndbroadbent/sprockets-rails', branch: 'dont_recompile_unchanged_assets'

You should also add the following lines to config/environments/production.rb to view the logs for rake assets:precompile:

config.log_level = :debug
config.logger = Logger.new(STDOUT)

Before / After

Here's some asset compilation times for a smallish Rails app.
(running time RAILS_ENV=production RAILS_GROUPS=assets rake assets:precompile)

Before:

asset:precompile: 26.993s

After:

Task Time
Initial asset:precompile 18.525s
assets:precompile:primary 9.772s
assets:precompile:nondigest 64ms
Unchanged assets 9.386s
assets:precompile:primary 877ms
assets:precompile:nondigest 91ms
One changed application.js dependency 12.386s
assets:precompile:primary 3.910s
assets:precompile:nondigest 48ms
@ndbroadbent ndbroadbent referenced this pull request in capistrano/capistrano
Closed

built-in rails asset compile: skip if no changes to assets? #227

@radar
Collaborator

I :heart: you.

@ndbroadbent

Thanks! :D

@jrochkind jrochkind referenced this pull request in capistrano/capistrano
Closed

Per-server conditional asset precompilation #174

@schneems
Collaborator

:+1: :heart:

@route

Also, instead of compiling twice for both digest and non-digest assets, non-digest assets are now generated from digest assets, with digests stripped from the content of any css and js files. This means we don't have to process and compress the same files twice. I'm making the assumption that asset paths are the only differences between digest/non-digest assets, but please let me know if you can see any problems with this approach.

Yea, it seems good if it compiles non-digest assets just for 64ms, WDYT @guilleiguaran @josevalim ? Relating comment #8 (comment)

@ndbroadbent ndbroadbent Detect asset changes by adding source_digests to manifest.yml
This allows us to quickly determine if the asset needs to be
recompiled. If nothing has changed in the source assets,
then compilation is skipped.
Also, instead of compiling twice for non-digest assets,
digest assets are copied instead, and the known digests are
stripped from the body of css and js files.
The copied js/css files are also gzipped.
746d1ff
@jeremy
Owner

Made some comments on the backport to Rails 3-2-stable @ rails/rails#7866

@josh josh closed this
@asymmetric

Hey @josh, was this closed because it got merged, or for some other reason?

@guilleiguaran

@asymmetric was closed since it doesn't apply anymore with the new version of sprockets-rails

@Simbul

Sorry to reanimate a closed thread.
@guilleiguaran, when you say "doesn't apply anymore", does it mean it's already in the new version of sprockets-rails? Or is it incompatible?

I guess what I really want to know is: what's the status of the effort towards speeding up the pipeline? Is that being tracked anywhere? Is turbo-sprockets-rails3 the answer to that?

@schneems
Collaborator

I'm interested to know as well. We're in the vetting process of getting this behavior fully supported on the Heroku buildpack with the gem for now (heroku/heroku-buildpack-ruby#44), but will want Rails 4 asset caching to work out of the box preferably.

In my very un-professional benchmarks, this behavior dropped precompiling assets on the codetriage.com app from 20 seconds to 5 seconds. One of our biggest consistent complaints is deploy time, and asset compilation has been a huge bottleneck.

@williscool

I'm interested in whether or not this was integrated as well

@brauliobo

Was caching acomplished? I see some references above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 27, 2012
  1. @ndbroadbent
Commits on Oct 6, 2012
  1. @ndbroadbent

    Detect asset changes by adding source_digests to manifest.yml

    ndbroadbent authored
    This allows us to quickly determine if the asset needs to be
    recompiled. If nothing has changed in the source assets,
    then compilation is skipped.
    Also, instead of compiling twice for non-digest assets,
    digest assets are copied instead, and the known digests are
    stripped from the body of css and js files.
    The copied js/css files are also gzipped.
This page is out of date. Refresh to see the latest.
View
7 Gemfile
@@ -3,9 +3,10 @@ source "http://rubygems.org"
# Specify your gem's dependencies in sprockets-rails.gemspec
gemspec
-gem "rails", :git => "git://github.com/rails/rails"
-gem 'activerecord-deprecated_finders', :git => 'git://github.com/rails/activerecord-deprecated_finders'
-gem 'journey', :git => 'git://github.com/rails/journey'
+gem 'rails', github: 'rails/rails', branch: 'master'
+gem 'journey', github: 'rails/journey', branch: 'master'
+gem 'activerecord-deprecated_finders', github: 'rails/activerecord-deprecated_finders', branch: 'master'
+
gem "uglifier", :require => false
gem "mocha", :require => false
gem "jquery-rails"
View
43 lib/sprockets-rails/tasks/assets.rake
@@ -43,7 +43,8 @@ namespace :assets do
config = ::Rails.application.config
config.assets.compile = true
config.assets.digest = digest unless digest.nil?
- config.assets.digests = {}
+ config.assets.digest_files ||= {}
+ config.assets.source_digests ||= {}
original_assets = Sprockets::Rails::Bootstrap.original_assets
env = if !config.assets.digest && original_assets
@@ -55,18 +56,38 @@ namespace :assets do
Rails.application.assets
end
- target = File.join(::Rails.public_path, config.assets.prefix)
- compiler = Sprockets::Rails::StaticCompiler.new(env,
- target,
- config.assets.precompile,
- :digest => config.assets.digest,
- :manifest => digest.nil?)
- compiler.compile
+ target = File.join(::Rails.public_path, config.assets.prefix)
+
+ # If processing non-digest assets, and compiled digest files are
+ # present, then generate non-digest assets from existing assets.
+ # It is assumed that `assets:precompile:nondigest` won't be run manually
+ # if assets have been previously compiled with digests.
+ if !config.assets.digest && config.assets.digest_files.any?
+ generator = Sprockets::Rails::StaticNonDigestGenerator.new(env, target, config.assets.precompile,
+ :digest_files => config.assets.digest_files)
+ generator.generate
+ else
+ compiler = Sprockets::Rails::StaticCompiler.new(env, target, config.assets.precompile,
+ :digest => config.assets.digest,
+ :manifest => digest.nil?,
+ :digest_files => config.assets.digest_files,
+ :source_digests => config.assets.source_digests)
+ compiler.compile
+ end
end
- task :all do
- Rake::Task["assets:precompile:primary"].invoke
- Rake::Task["assets:precompile:nondigest"].invoke if ::Rails.application.config.assets.digest
+ task :all => ["assets:cache:clean"] do
+ internal_precompile
+ if ::Rails.application.config.assets.digest
+ internal_precompile(false)
+
+ # Other gems may want to add hooks to run after the 'assets:precompile:***' tasks.
+ # Since we aren't running separate rake tasks anymore,
+ # we need to manually invoke the extra actions.
+ %w(primary nondigest).each do |asset_type|
+ Rake::Task["assets:precompile:#{asset_type}"].actions[1..-1].each &:call
+ end
+ end
end
task :primary => ["assets:cache:clean"] do
View
10 lib/sprockets/rails/helpers/rails_helper.rb
@@ -11,7 +11,7 @@ def asset_paths
@asset_paths ||= begin
paths = RailsHelper::AssetPaths.new(config, controller)
paths.asset_environment = asset_environment
- paths.asset_digests = asset_digests
+ paths.digest_files = digest_files
paths.compile_assets = compile_assets?
paths.digest_assets = digest_assets?
paths
@@ -96,8 +96,8 @@ def asset_prefix
::Rails.application.config.assets.prefix
end
- def asset_digests
- ::Rails.application.config.assets.digests
+ def digest_files
+ ::Rails.application.config.assets.digest_files
end
def compile_assets?
@@ -116,7 +116,7 @@ def asset_environment
end
class AssetPaths < ::ActionView::AssetPaths #:nodoc:
- attr_accessor :asset_environment, :asset_prefix, :asset_digests, :compile_assets, :digest_assets
+ attr_accessor :asset_environment, :asset_prefix, :digest_files, :compile_assets, :digest_assets
class AssetNotPrecompiledError < StandardError; end
@@ -132,7 +132,7 @@ def asset_for(source, ext)
end
def digest_for(logical_path)
- if digest_assets && asset_digests && (digest = asset_digests[logical_path])
+ if digest_assets && digest_files && (digest = digest_files[logical_path])
return digest
end
View
5 lib/sprockets/rails/railtie.rb
@@ -8,6 +8,7 @@ module Rails
autoload :LazyCompressor, "sprockets/rails/compressors"
autoload :NullCompressor, "sprockets/rails/compressors"
autoload :StaticCompiler, "sprockets/rails/static_compiler"
+ autoload :StaticNonDigestGenerator, "sprockets/rails/static_non_digest_generator"
# TODO: Get rid of config.assets.enabled
class Railtie < ::Rails::Railtie
@@ -35,7 +36,9 @@ class Railtie < ::Rails::Railtie
manifest_path = File.join(::Rails.public_path, config.assets.prefix, "manifest.yml")
if File.exist?(manifest_path)
- config.assets.digests = YAML.load_file(manifest_path)
+ manifest = YAML.load_file(manifest_path)
+ config.assets.digest_files = manifest[:digest_files] || {}
+ config.assets.source_digests = manifest[:source_digests] || {}
end
ActiveSupport.on_load(:action_view) do
View
55 lib/sprockets/rails/static_compiler.rb
@@ -12,16 +12,56 @@ def initialize(env, target, paths, options = {})
@digest = options.fetch(:digest, true)
@manifest = options.fetch(:manifest, true)
@zip_files = options.delete(:zip_files) || /\.(?:css|html|js|svg|txt|xml)$/
+
+ @current_source_digests = options.fetch(:source_digests, {})
+ @current_digest_files = options.fetch(:digest_files, {})
+
+ @digest_files = {}
+ @source_digests = {}
end
def compile
- manifest = {}
+ start_time = Time.now.to_f
+
env.each_logical_path(paths) do |logical_path|
- if asset = env.find_asset(logical_path)
- manifest[logical_path] = write_asset(asset)
+ # Fetch asset without any processing or compression,
+ # to calculate a digest of the concatenated source files
+ unprocessed_asset = env.find_asset(logical_path, :process => false)
+
+ @source_digests[logical_path] = unprocessed_asset.digest
+
+ # Recompile if digest has changed or compiled digest file is missing
+ current_digest_file = @current_digest_files[logical_path]
+ if @source_digests[logical_path] != @current_source_digests[logical_path] ||
+ !(current_digest_file && File.exists?("#{@target}/#{current_digest_file}"))
+
+ if asset = env.find_asset(logical_path)
+ @digest_files[logical_path] = write_asset(asset)
+ end
+
+ else
+ @digest_files[logical_path] = @current_digest_files[logical_path]
+
+ env.logger.debug "Not compiling #{logical_path}, sources digest has not changed " <<
+ "(#{@source_digests[logical_path][0...7]})"
end
end
- write_manifest(manifest) if @manifest
+
+ # Encode all filenames & digests as UTF-8. YAML dumps other string encodings as !binary.
+ @source_digests = encode_hash_as_utf8 @source_digests
+ @digest_files = encode_hash_as_utf8 @digest_files
+
+ if @manifest
+ write_manifest(source_digests: @source_digests, digest_files: @digest_files)
+ end
+
+ # Update digests in Rails config. (Important for when :nondigest is run after :primary)
+ config = ::Rails.application.config
+ config.assets.digest_files = @digest_files
+ config.assets.source_digests = @source_digests
+
+ elapsed_time = ((Time.now.to_f - start_time) * 1000).to_i
+ env.logger.debug "Processed #{'non-' unless @digest}digest assets in #{elapsed_time}ms"
end
def write_manifest(manifest)
@@ -43,6 +83,13 @@ def write_asset(asset)
def path_for(asset)
@digest ? asset.digest_path : asset.logical_path
end
+
+
+ private
+
+ def encode_hash_as_utf8(hash)
+ hash.inject({}) {|h, (k, v)| h[k.encode("UTF-8")] = v.encode("UTF-8"); h }
+ end
end
end
end
View
91 lib/sprockets/rails/static_non_digest_generator.rb
@@ -0,0 +1,91 @@
+require 'fileutils'
+
+module Sprockets
+ module Rails
+ class StaticNonDigestGenerator
+
+ DIGEST_REGEX = /-([0-9a-f]{32})/
+
+ attr_accessor :env, :target, :paths
+
+ def initialize(env, target, paths, options = {})
+ @env = env
+ @target = target
+ @paths = paths
+ @digest_files = options.fetch(:digest_files, {})
+
+ # Parse digests from digest_files hash
+ @asset_digests = Hash[*@digest_files.map {|file, digest_file|
+ [file, digest_file[DIGEST_REGEX, 1]]
+ }.flatten]
+ end
+
+
+ # Generate non-digest assets by making a copy of the digest asset,
+ # with digests stripped from js and css. The new files are also gzipped.
+ # Other assets are copied verbatim.
+ def generate
+ start_time = Time.now.to_f
+
+ env.each_logical_path(paths) do |logical_path|
+ unless digest_path = @digest_files[logical_path]
+ # Fail if any digest files are missing
+ raise "#{logical_path} is missing from :digest_files hash in manifest.yml!" <<
+ " Please run `rake assets:precompile` to recompile your assets with digests."
+ end
+
+ abs_digest_path = "#{@target}/#{digest_path}"
+ abs_logical_path = "#{@target}/#{logical_path}"
+
+ # Remove known digests from css & js
+ if digest_path.match(/\.(?:js|css)$/)
+ asset_body = File.read(abs_digest_path)
+
+ # Find all hashes in the asset body with a leading '-'
+ asset_body.gsub!(DIGEST_REGEX) do |match|
+ # Only remove if known digest
+ $1.in?(@asset_digests.values) ? '' : match
+ end
+
+ # Write non-digest file
+ File.open abs_logical_path, 'w' do |f|
+ f.write asset_body
+ end
+
+ # Also write gzipped asset
+ File.open("#{abs_logical_path}.gz", 'wb') do |f|
+ gz = Zlib::GzipWriter.new(f, Zlib::BEST_COMPRESSION)
+ gz.write asset_body
+ gz.close
+ end
+
+ env.logger.debug "Stripped digests, copied to #{logical_path}, and created gzipped asset"
+
+ else
+ # Otherwise, treat file as binary and copy it.
+ # Ignore paths that have no digests, such as READMEs
+ unless abs_digest_path == abs_logical_path
+ FileUtils.cp_r abs_digest_path, abs_logical_path, :remove_destination => true
+ env.logger.debug "Copied binary asset to #{logical_path}"
+
+ # Copy gzipped asset if exists
+ if File.exist? "#{abs_digest_path}.gz"
+ FileUtils.cp_r "#{abs_digest_path}.gz", "#{abs_logical_path}.gz", :remove_destination => true
+ env.logger.debug "Copied gzipped asset to #{logical_path}.gz"
+ end
+ end
+ end
+
+ mtime = File.mtime(abs_digest_path)
+
+ # Set modification and access times for generated files
+ File.utime(mtime, mtime, abs_logical_path)
+ File.utime(mtime, mtime, "#{abs_logical_path}.gz") if File.exist? "#{abs_logical_path}.gz"
+ end
+
+ elapsed_time = ((Time.now.to_f - start_time) * 1000).to_i
+ env.logger.debug "Generated non-digest assets in #{elapsed_time}ms"
+ end
+ end
+ end
+end
View
33 test/assets_test.rb
@@ -153,9 +153,14 @@ def assert_no_file_exists(filename)
precompile!
manifest = "#{app_path}/public/assets/manifest.yml"
- assets = YAML.load_file(manifest)
- assert_match(/application-([0-z]+)\.js/, assets["application.js"])
- assert_match(/application-([0-z]+)\.css/, assets["application.css"])
+ digests = YAML.load_file(manifest)
+ digest_files, source_digests = digests[:digest_files], digests[:source_digests]
+
+ assert_match(/application-([0-z]+)\.js/, digest_files["application.js"])
+ assert_match(/application-([0-z]+)\.css/, digest_files["application.css"])
+
+ assert_match(/[0-z]+/, source_digests["application.js"])
+ assert_match(/[0-z]+/, source_digests["application.css"])
end
test "the manifest file should be saved by default in the same assets folder" do
@@ -167,8 +172,8 @@ def assert_no_file_exists(filename)
precompile!
manifest = "#{app_path}/public/x/manifest.yml"
- assets = YAML.load_file(manifest)
- assert_match(/application-([0-z]+)\.js/, assets["application.js"])
+ digest_files = YAML.load_file(manifest)[:digest_files]
+ assert_match(/application-([0-z]+)\.js/, digest_files["application.js"])
end
test "precompile does not append asset digests when config.assets.digest is false" do
@@ -183,9 +188,9 @@ def assert_no_file_exists(filename)
manifest = "#{app_path}/public/assets/manifest.yml"
- assets = YAML.load_file(manifest)
- assert_equal "application.js", assets["application.js"]
- assert_equal "application.css", assets["application.css"]
+ digest_files = YAML.load_file(manifest)[:digest_files]
+ assert_equal "application.js", digest_files["application.js"]
+ assert_equal "application.css", digest_files["application.css"]
end
test "assets do not require any assets group gem when manifest file is present" do
@@ -196,8 +201,8 @@ def assert_no_file_exists(filename)
precompile!
manifest = "#{app_path}/public/assets/manifest.yml"
- assets = YAML.load_file(manifest)
- asset_path = assets["application.js"]
+ digest_files = YAML.load_file(manifest)[:digest_files]
+ asset_path = digest_files["application.js"]
require "#{app_path}/config/environment"
@@ -236,7 +241,7 @@ def show_detailed_exceptions?() true end
test "assets raise AssetNotPrecompiledError when manifest file is present and requested file isn't precompiled if digest is disabled" do
app_file "app/views/posts/index.html.erb", "<%= javascript_include_tag 'app' %>"
add_to_config "config.assets.compile = false"
- add_to_config "config.assets.digests = false"
+ add_to_config "config.assets.digest_files = false"
app_file "config/routes.rb", <<-RUBY
AppTemplate::Application.routes.draw do
@@ -286,9 +291,9 @@ def show_detailed_exceptions?() true end
app_file "app/assets/images/rails.png", "image changed"
precompile!
- assets = YAML.load_file(manifest)
+ digest_files = YAML.load_file(manifest)[:digest_files]
- assert_not_equal asset_path, assets["application.css"]
+ assert_not_equal asset_path, digest_files["application.css"]
end
test "precompile appends the md5 hash to files referenced with asset_path and run in production as default even using RAILS_GROUPS=assets" do
@@ -395,7 +400,7 @@ class ::PostsController < ActionController::Base ; end
assert_no_match(/<script src="\/assets\/xmlhr-([0-z]+)\.js"><\/script>/, last_response.body)
end
- test "assets aren't concatened when compile is true is on and debug_assets params is true" do
+ test "assets aren't concatenated when compile is on and debug_assets params is true" do
app_with_assets_in_view
add_to_env_config "production", "config.assets.compile = true"
add_to_env_config "production", "config.assets.allow_debugging = true"
View
2  test/sprockets_helper_test.rb
@@ -351,7 +351,7 @@ def compute_host(source, request, options = {})
end
test "precedence of `config.digest = false` over manifest.yml asset digests" do
- Rails.application.config.assets.digests = {'logo.png' => 'logo-d1g3st.png'}
+ Rails.application.config.assets.digest_files = {'logo.png' => 'logo-d1g3st.png'}
@config.assets.digest = false
assert_equal '/assets/logo.png',
Something went wrong with that request. Please try again.