Reintroduce gzip file generation

This change re-introduces compressed file generation and parallel file writing.

Gzip file generation was taken out in this PR: sstephenson#589

This was then discussed in this issue: #26

It was decided that compressing assets to maximize compression ratio at compile time was a valuable feature and within the scope of sprockets. This is one possible implementation.

Assets are written to disk in parallel using `Concurrent::Future`, since the gzip file cannot be generated until the original file is written to disk, it must process that file first. Speed impacts of writing files in parallel vary based on the number of assets being written to disk, disk speed, and IO contention.

Gzipping can be turned off at the environment level.

cc @fxn

CHANGELOG.md

@@ -1,3 +1,7 @@
+**Master**
+
+* Reintroduce Gzip file generation for non-binary assets.
+
 **3.4.1** (November 25, 2015)
 
 * PathUtils::Entries will no longer error on an empty directory.

guides/building_an_asset_processing_framework.md

@@ -2,6 +2,29 @@
 
 This guide is for using a Sprockets::Environment to process assets. You would use this class directly if you were building a feature similar to Rail's asset pipeline. If you aren't building an asset processing frameworks, you will want to refer to the [End User Asset Generation](end_user_asset_generation.md) guide instead. For a reference use of `Sprockets::Environemnt` see [sprockets-rails](github.com/rails/sprockets-rails).
 
+## Gzip
+
+By default when sprockets generates a compiled asset file it will also produce a gzipped copy of that file. Sprockets only gzips non-binary files such as CSS, javascript, and SVG files.
+
+For example if sprockets is generating
+
+```
+application-12345.css
+```
+
+Then it will also generate a compressed copy in
+
+```
+application-12345.css.gz
+```
+
+You can disable this behavior `Sprockets::Environemnt#gzip=` to something falsey for example:
+
+```ruby
+env = Sprockets::Environment.new(".")
+env.gzip = false
+```
+
 ## WIP
 
 This guide is a work in progress. There are many different groups of people who interact with sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.

guides/end_user_asset_generation.md

@@ -177,6 +177,24 @@ TODO: Explain contents, location, and name of a manifest file.
 
 TODO: Explain default fingerprinting/digest behavior
 
+### Gzip
+
+By default when sprockets generates a compiled asset file it will also produce a gzipped copy of that file. Sprockets only gzips non-binary files such as CSS, javascript, and SVG files.
+
+For example if sprockets is generating
+
+```
+application-12345.css
+```
+
+Then it will also generate a compressed copy in
+
+```
+application-12345.css.gz
+```
+
+This behavior can be disabled, refer to your framework specific documentation.
+
 ## WIP
 
 This guide is a work in progress. There are many different groups of people who interact with sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.

lib/sprockets/compressing.rb

@@ -51,6 +51,21 @@ def js_compressor
       end
     end
 
+    def gzip?
+      return @gzip if defined?(@gzip)
+      true
+    end
+
+    def skip_gzip?
+      !gzip?
+    end
+
+    # Enable or disable the creation of gzip files,
+    # on by default.
+    def gzip=(gzip)
+      @gzip = gzip
+    end
+
     # Assign a compressor to run on `application/javascript` assets.
     #
     # The compressor object must respond to `compress`.

lib/sprockets/manifest.rb

@@ -1,6 +1,10 @@
 require 'json'
 require 'time'
+
+require 'concurrent/future'
+
 require 'sprockets/manifest_utils'
+require 'sprockets/utils/gzip'
 
 module Sprockets
   # The Manifest logs the contents of assets compiled to a single directory. It
@@ -157,7 +161,9 @@ def compile(*args)
         raise Error, "manifest requires environment for compilation"
       end
 
-      filenames = []
+      filenames              = []
+      concurrent_compressors = []
+      concurrent_writers     = []
 
       find(*args) do |asset|
         files[asset.digest_path] = {
@@ -183,12 +189,26 @@ def compile(*args)
           logger.debug "Skipping #{target}, already exists"
         else
           logger.info "Writing #{target}"
-          asset.write_to target
+          write_file = Concurrent::Future.execute { asset.write_to target }
+          concurrent_writers << write_file
         end
-
         filenames << asset.filename
+
+        next if environment.skip_gzip?
+        gzip = Utils::Gzip.new(asset)
+        next if gzip.cannot_compress?(environment.mime_types)
+
+        if File.exist?("#{target}.gz")
+          logger.debug "Skipping #{target}.gz, already exists"
+        else
+          logger.info "Writing #{target}.gz"
+          concurrent_compressors << Concurrent::Future.execute { write_file.wait; gzip.compress(target) }
+        end
+
       end
-      save
+      concurrent_writers.each(&:wait)
+      concurrent_compressors.each(&:wait)
+      Concurrent::Future.execute { self.save }.wait
 
       filenames
     end
@@ -200,6 +220,7 @@ def compile(*args)
     #
     def remove(filename)
       path = File.join(dir, filename)
+      gzip = "#{path}.gz"
       logical_path = files[filename]['logical_path']
 
       if assets[logical_path] == filename
@@ -208,6 +229,7 @@ def remove(filename)
 
       files.delete(filename)
       FileUtils.rm(path) if File.exist?(path)
+      FileUtils.rm(gzip) if File.exist?(gzip)
 
       save
 

lib/sprockets/utils/gzip.rb

@@ -0,0 +1,56 @@
+module Sprockets
+  module Utils
+    class Gzip
+      # Private: Generates a gzipped file based off of reference file.
+      def initialize(asset)
+        @content_type  = asset.content_type
+        @mtime         = asset.mtime
+        @source        = asset.source
+        @charset       = asset.charset
+      end
+
+      # Private: Returns whether or not an asset can be compressed.
+      #
+      # We want to compress any file that is text based.
+      # You do not want to compress binary
+      # files as they may already be compressed and running them
+      # through a compression algorithm would make them larger.
+      #
+      # Return Boolean.
+      def can_compress?(mime_types)
+        # The "charset" of a mime type is present if the value is
+        # encoded text. We can check this value to see if the asset
+        # can be compressed.
+        #
+        # SVG images are text but do not have
+        # a charset defined, this is special cased.
+        @charset || @content_type == "image/svg+xml".freeze
+      end
+
+      # Private: Opposite of `can_compress?`.
+      #
+      # Returns Boolean.
+      def cannot_compress?(mime_types)
+        !can_compress?(mime_types)
+      end
+
+      # Private: Generates a gzipped file based off of reference asset.
+      #
+      # Compresses the target asset's contents and puts it into a file with
+      # the same name plus a `.gz` extension in the same folder as the original.
+      # Does not modify the target asset.
+      #
+      # Returns nothing.
+      def compress(target)
+        PathUtils.atomic_write("#{target}.gz") do |f|
+          gz = Zlib::GzipWriter.new(f, Zlib::BEST_COMPRESSION)
+          gz.mtime = @mtime.to_i
+          gz.write(@source)
+          gz.close
+        end
+
+        nil
+      end
+    end
+  end
+end

sprockets.gemspec

@@ -11,7 +11,8 @@ Gem::Specification.new do |s|
   s.files = Dir["README.md", "CHANGELOG.md", "LICENSE", "lib/**/*.rb"]
   s.executables = ["sprockets"]
 
-  s.add_dependency "rack", "> 1", "< 3"
+  s.add_dependency "rack",            "> 1", "< 3"
+  s.add_dependency "concurrent-ruby", "~> 1.0"
 
   s.add_development_dependency "closure-compiler", "~> 1.1"
   s.add_development_dependency "coffee-script-source", "~> 1.6"

test/test_manifest.rb

@@ -629,4 +629,32 @@ def teardown
     assert paths.include?("mobile/b.js")
     assert !paths.include?("application.js")
   end
+
+  test "compress non-binary assets" do
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ gallery.css application.js logo.svg }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      assert File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to generate gzipped file: '#{original_path}.gz' but it did not"
+    end
+  end
+
+  test "disable file gzip" do
+    @env.gzip = false
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ gallery.css application.js logo.svg }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      refute File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to not generate gzipped file: '#{original_path}.gz' but it did"
+    end
+  end
+
+  test "do not compress binary assets" do
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ blank.gif }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      refute File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to not generate gzipped file: '#{original_path}.gz' but it did"
+    end
+  end
 end