Skip to content
This repository

Rack::File serves gzipped assets when available #479

Open
wants to merge 1 commit into from

11 participants

Jonathan Baudanza James Tucker Hannes Georg James Alexander Rosen Ellis Berner Gioele michrome Zaid Brian Alexander Richard Schneeman John Bachir
Jonathan Baudanza

@raggi, After poking around a bit, I think Rack::File is the cleanest place to put this change.

This is a very small addition to Rack::File. I think adding it to Rack::Static would be slightly more complex. For example, since Rack::File would set all compressed content-types to "application/gzip", we would need to duplicate the Rack::File mime type logic in Rack::Static to correct this.

I looked at the header_rule stuff in Rack::Static, but couldn't see any graceful way to reuse it.

LMK what you think.

James Tucker
Owner

Looks good, one last thing:

I would prefer that this was opt-in (although I'm aware we're running out of initializer params). In the very least, I would prefer it had an option to opt-out.

Thoughts?

Hannes Georg

As I read this the .gz stuff has to be in the same directory as the original files. Can we always assume this? I think it's reasonable to make this configurable alongside the opt-out.

Jonathan Baudanza

@raggi, I changed #initialize to take an options hash instead and made :serve_gzip opt-in.

I was also considering splitting up Rack::File into multiple classes. It's getting a little crowded. It would work something like this:

  # Rack::Files contains the functionality for parsing and validating PATH_INFO
  # Rack::File is responsible for serving one and only one file, and handling Ranges 
  new Rack::Files('assets', :file_server => Rack::File)

  # You can opt into gzip functionality like so.  Rack::Gzfile would be responsible for deciding whether to server
  # a gzipped file and would delegate the serving functionality to a wrapped Rack::File object
  new Rack::Files('assets', :file_server => Rack::GzFile)

LMK what you think. This might be better for another pull request.

Jonathan Baudanza

@hannesg, I wrote this specifically to handle the gzipped files that Sprockets generates next to the uncompressed assets. I think this is a pretty standard practice.

Do you know of any use cases where the compressed files are stored in a different directory? I've not heard of any. But if there is a use case I'm missing, then maybe we should add that option.

Jonathan Baudanza

Also, notice that I removed this false-positive test:

should "not set Content-Type if the mime type is not set" do    
  req = Rack::MockRequest.new(Rack::Lint.new(Rack::File.new(DOCROOT, nil, nil)))
  res = req.get "/cgi/test"
  res.should.be.successful
  res['Content-Type'].should.equal nil
end

This test was passing but the feature wasn't working. It seemed like an odd feature to have, so I just removed it.

James Alexander Rosen

What about a single gzip_path_generator option? If it's present, it receives the original path and is expected to generate the path to the gzipped version. The default could be lambda { |path| "#{path}.gz" } (or nil for opt-in behavior).

Jonathan Baudanza

@jamesarosen That would probably be the best way to implement @hannesg's suggestion. But AFAIK, everyone that is going to use this feature is going to have gzipped files in the same directory with the same name plus ".gz". I'd rather keep it simple unless we see that people actually need this flexibility.

James Tucker
Owner

Apologies, but I'm uncomfortable with the API changed still (particularly the files constructor, which already changed recently).

It's possible, although a little convoluted, to create a temporary workaround by altering path_info to include a .gz, and catching X-Cascade, undoing and retrying when 404'ing.

If we can get this in without changing public APIs significantly, then I'll reconsider adding it to a future 1.5 release.

Jonathan Baudanza

We could create a Rack::GzFile app that wraps Rack::File and adds the additional functionality. We would have to duplicate the part of Rack::File that looks up the mime type, which I don't really like. But we wouldn't have to touch the public API. Sound good?

Ellis Berner

Agree with @raggi, would like this, but not the way it changes the public API.

Currently using https://github.com/romanbsd/heroku-deflater/ as a workaround.

Jonathan Baudanza

I'm happy to do another iteration on this, but I'd like to get some feedback on the proposed API before I do any more work on it.

What does everyone think of creating a new Rack::GzFile app that wraps Rack::File?

Ellis Berner

I like that idea.

Gioele

@jbaudanza:

We would have to duplicate the part of Rack::File that looks up the mime type, which I don't really like.

Can't that part be split out into an helper used by Rack::File, Rack::GzFile and whatever other middleware need to use it?

Jonathan Baudanza

Ok I've created a new Rack::GzFile, as discussed. Rack::File is untouched. LMK what you think.

Ellis Berner

I do fancy this. :+1:

Jonathan Baudanza

For the time being, I've packaged this functionality into a gem:

https://github.com/jbaudanza/action_dispatch-gz_static

This should be useful for anyone running the Rails asset precompiler on Heroku.

Jonathan Baudanza

Hey guys, any thoughts on this? If this isn't going to make it into Rack, I may reopen the issue on the rails repo.

Brian Alexander

I'd like to see this as well. :thumbsup:

Jonathan Baudanza

@raggi any thoughts?

Brian Alexander

@jbaudanza Is there a way to use your Rack::GzFile with Rack:::Static? Possibly this line should be changed to something like:

@file_server = options[:file_server] || Rack::File.new(root)

That way, an instance of Rack::GzFile can be passed in as an option. Or better yet Rack::GzFile could be the default, but I'm not sure if the maintainers of this gem would agree with that.

Jonathan Baudanza

@balexand Yes, definitely. But I'd like for one of the rack maintainers to chime in before I do any more work on this.

I had also intended on submitting a similar change for ActionDispatch::Static in Rails.

Brian Alexander balexand commented on the diff August 01, 2013
lib/rack/gz_file.rb
((13 lines not shown))
  13
+      @file_server = Rack::File.new(root, headers, default_mime)
  14
+      @default_mime = default_mime
  15
+    end
  16
+
  17
+    def call(env)
  18
+      path_info = env['PATH_INFO']
  19
+
  20
+      status, headers, body = @file_server.call(
  21
+        env.merge('PATH_INFO' => path_info + '.gz')
  22
+      )
  23
+
  24
+      gzip_available = status != 404
  25
+
  26
+      if !gzip_available || env['HTTP_ACCEPT_ENCODING'] !~ /\bgzip\b/
  27
+        status, headers, body = @file_server.call(env)
  28
+      else
4
Brian Alexander
balexand added a note August 01, 2013

@jbaudanza While testing this in development, I got this Rack::Lint error. Apparently, the Content-Type shouldn't be set for 304 Not Modified response.

image

I solved this by changing this line to:

elsif status != 304

I don't think this is the best solution. I'd be happy to propose a more robust solution if the maintainers of Rack have any interest in this pull-request.

Jonathan Baudanza
jbaudanza added a note August 01, 2013

Thanks. good catch!

John Bachir
jjb added a note November 24, 2013

@jbaudanza remember to fix this...

@jjb holding off on any work on this until I get a nod from a maintainer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Richard Schneeman

Got lead here via a discussion on the Heroku forums https://discussion.heroku.com/t/rack-zippy-gem-serves-your-gz-assets-on-heroku/206/5 . I'm generally :+1: on the idea.

John Bachir jjb commented on the diff November 24, 2013
test/spec_gz_file.rb
((28 lines not shown))
  28
+        File.size(DOCROOT + '/cgi/assets/compress_me.html').to_s)
  29
+  end
  30
+
  31
+  should "serve a compressed file" do
  32
+    res = request.get('/cgi/assets/compress_me.html', 
  33
+        'HTTP_ACCEPT_ENCODING' => 'gzip')
  34
+
  35
+    gz = Zlib::GzipReader.new(StringIO.new(res.body))
  36
+    gz.read.should.equal "Hello, Rack!"
  37
+    res.headers['Vary'].should.equal 'Accept-Encoding'
  38
+    res.headers['Content-Encoding'].should.equal 'gzip'
  39
+    res.headers['Content-Type'].should.equal 'text/html'
  40
+    res.headers['Content-Length'].should.equal(
  41
+        File.size(DOCROOT + '/cgi/assets/compress_me.html.gz').to_s)
  42
+  end
  43
+end
1
John Bachir
jjb added a note November 24, 2013

should put a newline here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
John Bachir

i wonder if perhaps it would be more appropriate for this middleware to live alongside the regular Rack::File instead of replacing it. it could only serve gz when appropriate, ignore everything else. in a sense this makes the code redundant because you have file-serving code in multiple middleware. but i think overall it will make the code smaller and make its intent more clear and its behavior more bounded.

going even further: instead of serving the file, maybe it should rewrite the request. so if a request comes in for foo.js and accepts gz and foo.js.gz is present, the request is rewritten to be requesting foo.js.gz and then handed down to Rack::File. This will further decrease the code size and complexity.

name it: Rack::AcceptsToFilenameConverter (not really)

Jonathan Baudanza

@jjb I'm not sure I follow your suggestion. The current implementation of GzFile already delegates most of the file serving functionality to a composed instance of Rack::File. Are you suggesting that instead of composition, we have the caller put the Rack::File middleware downstream manually?

John Bachir

@jbaudanza sorry, wasn't clear

idea 1 is to only use the composed instance to serve .gz files. ignore other files. it's up to the user to include another instance of Rack::File downstream as they see fit. users' needs will vary, so this allows more flexibility.

idea 2 (which i think is even better) is to not have your code do any file serving. only rename the incoming request and send it downstream, presumably to a Rack::File which will then serve the .gz file.

make sense?

Jonathan Baudanza

Makes sense. I understand the desire to make the middleware more flexible, but in this instance I think it makes sense to leave GzFile tightly coupled with Rack::File. What GzFile is doing is very specific to serving files off of a filesystem, and I don't think it would make sense to have anything downstream other than Rack::File.

On the other hand, let me know if there's another use case I'm missing.

It's interesting to discuss. But, as I said, I'm not planning on doing any more work on this until I get a nod from a maintainer. For now, I moved this functionality into another gem that I use on my projects.

John Bachir

gotcha

i just experimented with my idea 2 and got it working, but encountered a problem. after asking the downstream Rack::Static to serve the file, the .js.gz file is delivered with content type application/x-gzip and no encoding.

John Bachir

@jbaudanza behold, the evolution and manifestation of all the points I was previously clumsily making, in code form! jbaudanza/action_dispatch-gz_static#3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Feb 26, 2013
Jonathan Baudanza Added GzFile for serving gzipped static files 6b5dcfc
This page is out of date. Refresh to see the latest.
1  lib/rack.rb
@@ -34,6 +34,7 @@ def self.release
34 34
   autoload :ContentType, "rack/content_type"
35 35
   autoload :ETag, "rack/etag"
36 36
   autoload :File, "rack/file"
  37
+  autoload :GzFile, "rack/gz_file"
37 38
   autoload :Deflater, "rack/deflater"
38 39
   autoload :Directory, "rack/directory"
39 40
   autoload :ForwardRequest, "rack/recursive"
41  lib/rack/gz_file.rb
... ...
@@ -0,0 +1,41 @@
  1
+module Rack
  2
+  # Rack::GzFile behaves exactly the same as Rack::File, except that it will
  3
+  # also serve up a gzip encoding of a file, if one is available on the
  4
+  # filesystem.
  5
+  #
  6
+  # For each request, Rack::GzFile first checks the filesystem for a file with a
  7
+  # .gz extension. If one is found, the appropriate encoding headers are added
  8
+  # to the response and the gzip file is served.
  9
+  #
  10
+  # If no .gz file is found, Rack::GzFile will behave exactly like Rack::File.
  11
+  class GzFile
  12
+    def initialize(root, headers={}, default_mime = 'text/plain')
  13
+      @file_server = Rack::File.new(root, headers, default_mime)
  14
+      @default_mime = default_mime
  15
+    end
  16
+
  17
+    def call(env)
  18
+      path_info = env['PATH_INFO']
  19
+
  20
+      status, headers, body = @file_server.call(
  21
+        env.merge('PATH_INFO' => path_info + '.gz')
  22
+      )
  23
+
  24
+      gzip_available = status != 404
  25
+
  26
+      if !gzip_available || env['HTTP_ACCEPT_ENCODING'] !~ /\bgzip\b/
  27
+        status, headers, body = @file_server.call(env)
  28
+      else
  29
+        headers['Content-Type'] = Mime.mime_type(::File.extname(path_info), 
  30
+                                                 @default_mime)
  31
+        headers['Content-Encoding'] = 'gzip'
  32
+      end
  33
+
  34
+      if gzip_available
  35
+        headers['Vary'] = 'Accept-Encoding'
  36
+      end
  37
+
  38
+      [status, headers, body]
  39
+    end
  40
+  end
  41
+end
1  test/cgi/assets/compress_me.html
... ...
@@ -0,0 +1 @@
  1
+Hello, Rack!
BIN  test/cgi/assets/compress_me.html.gz
Binary file not shown
43  test/spec_gz_file.rb
... ...
@@ -0,0 +1,43 @@
  1
+require 'rack/gz_file'
  2
+require 'rack/lint'
  3
+require 'rack/mock'
  4
+require 'zlib'
  5
+
  6
+describe Rack::GzFile do
  7
+  DOCROOT = File.expand_path(File.dirname(__FILE__)) unless defined? DOCROOT
  8
+
  9
+  def request
  10
+    Rack::MockRequest.new(Rack::Lint.new(Rack::GzFile.new(DOCROOT)))
  11
+  end
  12
+
  13
+  should "serve an uncompressed file when gzip is not supported by the server" do
  14
+    res = request.get('/cgi/assets/index.html')
  15
+    res.body.should.equal "### TestFile ###\n"
  16
+    res.headers.should.not.include 'Vary'
  17
+    res.headers.should.not.include('Content-Encoding')
  18
+    res.headers['Content-Length'].should.equal(
  19
+        File.size(DOCROOT + '/cgi/assets/index.html').to_s)
  20
+  end
  21
+
  22
+  should "serve an uncompressed file when gzip is not supported by the client" do
  23
+    res = request.get('/cgi/assets/compress_me.html')
  24
+    res.body.should.equal 'Hello, Rack!'
  25
+    res.headers['Vary'].should.equal 'Accept-Encoding'
  26
+    res.headers.should.not.include('Content-Encoding')
  27
+    res.headers['Content-Length'].should.equal(
  28
+        File.size(DOCROOT + '/cgi/assets/compress_me.html').to_s)
  29
+  end
  30
+
  31
+  should "serve a compressed file" do
  32
+    res = request.get('/cgi/assets/compress_me.html', 
  33
+        'HTTP_ACCEPT_ENCODING' => 'gzip')
  34
+
  35
+    gz = Zlib::GzipReader.new(StringIO.new(res.body))
  36
+    gz.read.should.equal "Hello, Rack!"
  37
+    res.headers['Vary'].should.equal 'Accept-Encoding'
  38
+    res.headers['Content-Encoding'].should.equal 'gzip'
  39
+    res.headers['Content-Type'].should.equal 'text/html'
  40
+    res.headers['Content-Length'].should.equal(
  41
+        File.size(DOCROOT + '/cgi/assets/compress_me.html.gz').to_s)
  42
+  end
  43
+end
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.