Skip to content
This repository

Configuration options for Rack::Deflate #457

Open
wants to merge 2 commits into from

8 participants

Jakub Pawlowicz Ellis Berner Bruno James Tucker Nathanael Jones Claudio Poli John Bachir jayschab
Jakub Pawlowicz

Rack::Deflater currently does not support any configuration options which makes it cumbersome to control.

This pull request adds support for the following options:

  • include - an array of content types enabled for compression
  • if - a lambda for choosing whether to deflate based on current execution scope (request, status, body, or headers)

Examples:
use Rack::Deflater, include: %w(text/json application/json)
use Rack::Deflater, if: lambda { |env, status, headers, body| body.length > 512 }

EDIT: description was update based on a PR discussion

Ellis Berner

Was this created with the intention of skipping image files with Deflator? Because that would be an excellent use case according to https://developers.google.com/speed/docs/best-practices/payload#GzipCompression.

Jakub Pawlowicz

Yes, that's one use case I can think of. In general it's a bad idea to compress requests smaller than TCP packet size as it only adds an overhead on both ends.

This threshold feature can always be configured on the web server level (nginx, Apache) but in case of services which do not proxy requests through a web server (e.g. Heroku Cedar) having such option in Rack::Deflater is the only option available.

Ellis Berner

Definitely important. Would love to see this get merged!

Ellis Berner

TCP packet size is different for everybody? But in general, seems to be about 64K?

Jakub Pawlowicz

64kB is the theoretical upper limit for TCP/IP. In reality it depends on the connection type and does not exceed ~1 kB - still there's no need to compress such small amounts of data.

Bruno

Is this ready to be merged?

Jakub Pawlowicz

Not yet as some new specs are failing under 1.8.7 / jruby / ree. It will be ready to merge within the next 24 hours.

Jakub Pawlowicz

@bpinto it's ready for merge. I've refactored the tests a bit more, turned off deflating to see if all relevant tests fail (they do!) and reverted to a working state. Should be great now!

lib/rack/deflater.rb
((6 lines not shown))
9 9
       @app = app
  10
+
  11
+      @min_content_length = options[:min_content_length] || options['min_content_length']
2
James Tucker Owner
raggi added a note December 29, 2012
  • Lets just support symbol option keys, not strings, consistent with other middleware options.
  • Maybe min_size or min_length? Just a little shorter?
James Tucker Owner
raggi added a note December 29, 2012

Lets default min-content-length to 1kb. There can be some advantages on mobile networks for smaller entities, but that domain has additional solutions and complexities - shared dictionaries are better, lets hope for them in http2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/rack/deflater.rb
... ...
@@ -5,19 +5,20 @@
5 5
 
6 6
 module Rack
7 7
   class Deflater
8  
-    def initialize(app)
  8
+    def initialize(app, options = {})
1
James Tucker Owner
raggi added a note December 29, 2012

Lets document the options, using RDoc style. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/rack/deflater.rb
... ...
@@ -100,5 +101,45 @@ def each
100 101
         deflater.close
101 102
       end
102 103
     end
  104
+
  105
+    private
  106
+
  107
+    def should_deflate?(env, status, headers, body)
  108
+      # Skip compressing empty entity body responses and responses with
  109
+      # no-transform set.
  110
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  111
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  112
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  113
+        return false
  114
+      end
  115
+
  116
+      # Skip if response body is too short
  117
+      if @min_content_length &&
  118
+          @min_content_length > headers['Content-Length'].to_i
1
James Tucker Owner
raggi added a note December 29, 2012

If we default this to some value, we can skip the nil check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/rack/deflater.rb
((10 lines not shown))
  110
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  111
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  112
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  113
+        return false
  114
+      end
  115
+
  116
+      # Skip if response body is too short
  117
+      if @min_content_length &&
  118
+          @min_content_length > headers['Content-Length'].to_i
  119
+        return false
  120
+      end
  121
+
  122
+      # Skip if :include is provided and evaluates to false
  123
+      if @include &&
  124
+          @include.kind_of?(Regexp) &&
  125
+          !@include.match(env['PATH_INFO'])
2
James Tucker Owner
raggi added a note December 29, 2012

If we use env['PATH_INFO'] =~ @include then it's fine as long as @include is nil, or a regex, or a string. If necessary that could be sanitized in the constructor. This will crunch down all the boolean logic here.

James Tucker Owner
raggi added a note December 29, 2012

In fact, if you use === instead of =~ (change operation order) then you naturally get lambda support in 1.9+ as well as string and regex matching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/rack/deflater.rb
((17 lines not shown))
  117
+      if @min_content_length &&
  118
+          @min_content_length > headers['Content-Length'].to_i
  119
+        return false
  120
+      end
  121
+
  122
+      # Skip if :include is provided and evaluates to false
  123
+      if @include &&
  124
+          @include.kind_of?(Regexp) &&
  125
+          !@include.match(env['PATH_INFO'])
  126
+        return false
  127
+      end
  128
+
  129
+      # Skip if :exclude is provided and evaluates to true
  130
+      if @exclude &&
  131
+          @exclude.kind_of?(Regexp) &&
  132
+          @exclude.match(env['PATH_INFO'])
1
James Tucker Owner
raggi added a note December 29, 2012

As for @include

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
James Tucker
Owner

In principle this looks great. I left you some line comments, thanks for your work so far!

Jakub Pawlowicz

Thanks @raggi for all suggestions! I've just applied them so it should be more ready to merge.

James Tucker
Owner

This needs rebasing on top of rack master. I'll be happy to merge this in the next release, but I am out of time for 1.5.0.

Jakub Pawlowicz

That's a pity. I'll merge it as soon as possible to make it into 1.6.

Jakub Pawlowicz

@raggi That should be it!

Nathanael Jones

To me, the safest default scenario is opt-in based on mime-type.

HTML5 Bootstrap includes server-side configuration to compress the following mime-types

text/html    
text/css
text/plain
text/x-component
application/javascript
application/json
application/xml
application/xhtml+xml
application/x-font-ttf
application/x-font-opentype
application/vnd.ms-fontobject
image/svg+xml
image/x-icon;

I would consider the following list instead:

# All html, text, css, and csv content should be compressed
text/plain
text/html
text/csv
text/css

# Only vector graphics and uncompressed bitmaps can benefit from compression.
#GIF, JPG, and PNG already use a lz* algorithm, and certain browsers can get confused.
image/x-icon
image/svg+xml
application/x-font-ttf
application/x-font-opentype
application/vnd.ms-fontobject


# All javascript should be compressed
text/javascript
application/ecmascript
application/json
application/javascript

# All xml should be compressed
text/xml
application/xml
application/xml-dtd
application/soap+xml
application/xhtml+xml
application/rdf+xml
application/rss+xml
application/atom+xml

If it's not too late, I would suggest mime-type evaluation instead of URL regexes. Providing a sane default set would be great; the current usage pattern is causing lots of issues with images and PDFs, since old browsers lie about Accept-Encoding.

Rack::Deflate has become very important, as Heroku's new Cedar stack removed automatic gzip support, and requires that step be moved to the application itself.

Ellis Berner
Jakub Pawlowicz

@raggi - any ideas about mime-type based evaluation?

Nathanael Jones

Should I submit a pull request?

Jakub Pawlowicz

Any ideas how to push it forward?

Claudio Poli

Looks awesome to have; HAProxy 1.5dev19 dropped (temporarly) support for transparent gzip compression of chunked responses so we have to implement it at app levels, the more options, the better.

lib/rack/deflater.rb
... ...
@@ -100,5 +110,42 @@ def each
100 110
         deflater.close
101 111
       end
102 112
     end
  113
+
  114
+    private
  115
+
  116
+    def should_deflate?(env, status, headers, body)
  117
+      # Skip compressing empty entity body responses and responses with
  118
+      # no-transform set.
  119
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  120
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  121
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  122
+        return false
  123
+      end
  124
+
  125
+      # Skip if response body is too short
  126
+      if @min_length > headers['Content-Length'].to_i
1

I'm testing this but using Rainbows!, Rails 3.2.14, http 1.1 and I don't see any Content-Length header in this point, this thus fails and compression is skipped.

Rainbows! was started with the -N option to not insert any default middleware, so my config.ru is:

use Rack::ContentLength
use Rack::Chunked
run Myapp::Application
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
John Bachir

@jakubpawlowicz -- this is an awesome patch. It has a lot of features all together, which is probably part of why it's taking so long to merge. Here are some thoughts on how it can be simplified and improved.

include/exclude

It seems like folks (including me) think that

  • whitelist is superior
  • mime-type/Content-Type is superior

I recommend:

  • remove the exclude option entirely (I can't think of a use case, tell me if I'm missing something)
  • reimplement include to operate on the Content-Type header instead of the URL.
  • thought: would be nice to allow the Content-Type list to accept regexes, to allow for text/* for example. but this could perhaps come in a subsequent pull request, to keep this one more simple.

skip_if

Maybe a better name for this to make it consistent with other DSLs would be unless. Also, IMO it's more simple to make it if. "skip/unless" make it feel like it's supposed to be a filter. "if" is more generic, allowing the user to simply put in a conditional as they see fit. (and indeed, even in the example you gave you have a !=, making your overall example a double negative).

min_content_length

tl;dr: this option should be removed entirely, because it can be trivially implemented with skip_if/unless/if

There are several problems with making a default value.

  • it's a behavior change, so it makes it more controversial to merge
  • you didn't explain how you chose 1024

There are 3 independent reasons I can think of to have a minimum value:

  1. the point at which compressing does not reduce the size, and in fact might increase it. this is an objective value that is relevant to all users. Some quick searching seems to indicate that the number is around 150 bytes.
  2. the point at which the size is smaller than a single packet anyway. to my understanding this varies between environments. Maybe others could speak to this more.
  3. the point at which the time-complexity of compression is not worth the reduction in data-transmission time. This is highly dependent on hardware and applications. Also, it can be balanced with the gzip compression level. setting the compression level to 5 results in a much different time-complexity/compression ratio than 9. so, a good solution to accommodate this tradeoff would also include adjusting the compression level based on content-type. I just checked -- rack::deflate uses zlib's default, which is 6.

To increase the chances of your patch being merged, I think the default should be removed. Instead, there could be a recommended minimum in documentation.

Furthermore, regarding point 3, this makes me think it would be nice to be able to adjust compression level as well. So after your patch is merged, I'd like to experiment with augmenting it so your compression_level option could be passed either an integer or a lambda, which would dictate if something is compressed at all, and with what compression level.

->(size){
case size
when 0..256
  nil
when 256..1024
  6
when 1025..Float::INFINITY
  9
end
}

And as I typed that out I realized: this could be achieved with skip_if -- so, you could rewrite skip_if to expect either a boolean or integer to be returned. When it's an integer, it indicates compression level.

John Bachir

as soon as i posted, that, i realized that include can also be implemented with if. So now I'm a bit conflicted

  • on the one hand, only implementing if and nothing else will be much more simple overall code and allow for greater flexibility for the user
  • on the other hand, the point of offering the library is to make it easy to achieve domain-specific things, so options like include and min_content_length make a lot of sense.

I've seen other libraries offer specific options and then an all-powerful if on the side, so maybe that's fine for here. too. My gut says that going with only include and if is a good approach to start with.

Jakub Pawlowicz

@jjb appreciate your awesome feedback!

  • to be honest I don't remember where 1024 came from (it's been a year since I wrote it) but I like your definition via lambda - let's have it that way.
  • whitelisting via include should work well (so exclude will be gone)
  • min_content_length was a convenience method but your idea is better

If you have a spare moment can you let me know what do you think about @nathanaeljones comment: #457 (comment) ? We could have it as an option for include, e.g.

use Rack::Deflater, { include: Rack::Deflater::DEFAULTS }

Thoughts?

John Bachir

@jakubpawlowicz looks good. ideally the list would be sourced from combining arrays from another project, like...

Rack::Mime::MIME_TYPES.select{|k,v| v.start_with? "text" }

MIME::Types[/^text/] # note that this blows up with mime-types 1.25, works with 2.0, haven't tried 1.25.1

source: http://stackoverflow.com/questions/16003824/get-all-video-mime-types-in-ruby

Jakub Pawlowicz

@jjb that's another good idea! Will wrap up all the changes in the coming days.

Jakub Pawlowicz

@jjb - that should be it!

I decided to use Rack::Mime::MIME_TYPES instead of MIME::Types to save on adding a dependency. So a regular expression can be passed and it is first matched against Content-Type header and then against list of known mime types. We should probably have some sanity checks first against missing Content-Type header too.

Once we are all set I will squeeze all the commits into two - one for refactored spec_deflater.rb and the other one for Deflater options.

Please let me know how it looks to you.

Jakub Pawlowicz

And regarding failing specs I noticed master fails on them too. It shouldn't be an excuse though...

John Bachir

@jakubpawlowicz awesome. the discussion about mime types was regarding where to get a list of defaults. it looks like you are using it to make sure that the provided types are valid. my feeling is that this isn't very useful and you can exclude this entirely.

lib/rack/deflater.rb
((10 lines not shown))
  141
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  142
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  143
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  144
+        return false
  145
+      end
  146
+
  147
+      # Skip if :include is given and does not match request's content type
  148
+      if @include &&
  149
+          !(@include.match(headers['Content-Type']) && Rack::Mime::MIME_TYPES.values.include?(headers['Content-Type']))
  150
+        return false
  151
+      end
  152
+
  153
+      # Skip if :if lambda is given and evaluates to false
  154
+      if @if &&
  155
+          !@if.call(env, status, headers, body)
  156
+        return false
3
John Bachir
jjb added a note December 09, 2013
return false if @if && !@if.call(env, status, headers, body)
John Bachir
jjb added a note December 09, 2013

(just my style opinion...)

Sure, it reads better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jakub Pawlowicz

@jjb sorry my mistake! So let's make sure we are on the same page regarding :include:

  • it should be an array (assembled anyhow user wants)
  • we should default to all content types matching /^text/ - what about application/json?
  • if above then there's no need to make it default explicitly via use Rack::Deflater, { include: Rack::Deflater::DEFAULTS } as plain use Rack::Deflater will be enough
John Bachir

that is one way. but, the current deflater approach is to deflate everything. so, you could keep this default behavior, and only change the behavior if include is specified. I recommend this as it's more likely to be accepted by the maintainers and is just generally a more conservative approach.

So your code already does this :-D. so if you agree, you can just remove the check that it's in Rack::Mime::MIME_TYPES.values and IMO the code will be more simple.

(i think i overexplained this but just trying to be extra clear :-D)

lib/rack/deflater.rb
((6 lines not shown))
  137
+
  138
+    def should_deflate?(env, status, headers, body)
  139
+      # Skip compressing empty entity body responses and responses with
  140
+      # no-transform set.
  141
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  142
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  143
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  144
+        return false
  145
+      end
  146
+
  147
+      # Skip if :include is given and does not match request's content type
  148
+      if @include &&
  149
+          !(@include.match(headers['Content-Type']) && Rack::Mime::MIME_TYPES.values.include?(headers['Content-Type']))
  150
+        return false
  151
+      end
  152
+
4
John Bachir
jjb added a note December 09, 2013
return false if @include && !@include.match(headers['Content-Type'])

Since @include should be an array it should rather read:

return false if @include && !@include.include?(headers['Content-Type'])

or to make it more readable

return false if @include && @include.index(headers['Content-Type']).nil?
John Bachir
jjb added a note December 09, 2013

sounds good. i find the first more readable, i'm not familiar with index and the required .nil? at the end seems complicated

as long as it works as expected I'm fine with the first too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jakub Pawlowicz

Cool, so we are on the same page. Will update the code shortly.

lib/rack/deflater.rb
((13 lines not shown))
21 28
       @app = app
  29
+
  30
+      @if = options[:if]
  31
+      @include = options[:include]
2
John Bachir
jjb added a note December 09, 2013

the symbols are for the DSL but you could make more descriptive variables within the class

@condition = options[:if]
@compressible_types = options[:include]

:+1: nice one!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
John Bachir

I made this SO question to try to find a nice list of types for us to recommended in the documentation http://stackoverflow.com/questions/20477558/where-can-i-find-a-list-of-textual-mime-types

Jakub Pawlowicz

@jjb here you go! Let's see what your SO question brings.

Jakub Pawlowicz

@jjb So it's quashed into two commits, with better docs, and updated PR description

@raggi @nathanaeljones You may like it much more right now!

lib/rack/deflater.rb
... ...
@@ -126,5 +133,25 @@ def close
126 133
         @body.close if @body.respond_to?(:close)
127 134
       end
128 135
     end
  136
+
  137
+    private
  138
+
  139
+    def should_deflate?(env, status, headers, body)
  140
+      # Skip compressing empty entity body responses and responses with
  141
+      # no-transform set.
  142
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  143
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  144
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  145
+        return false
  146
+      end
  147
+
  148
+      # Skip if @compressible_types are given and does not include request's content type
  149
+      return false if @compressible_types && !@compressible_types.include?(headers['Content-Type'])
4
jayschab
jayschab added a note January 07, 2014

First thing, thanks for this code it is a big help. One issue I ran into was this line. When the Content-type has the optional parameter value (in my case the character encoding) the content-type check fails. See RFC-1341. I am using .split(";")[0] and it is working for my use cases.

Thanks @jayschab for a valuable input. I'll add a test case and your solution and we should be all good. :+1:

jayschab
jayschab added a note January 15, 2014

Found a bug in my suggested code in my testing. Content-Type isn't always set.

I ended up changing my version to this:

return false if @compressible_types && !(headers.has_key('Content-Type') && @compressible_types.include?(headers['Content-Type'][/[^;]*/]))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/rack/deflater.rb
... ...
@@ -126,5 +133,26 @@ def close
126 133
         @body.close if @body.respond_to?(:close)
127 134
       end
128 135
     end
  136
+
  137
+    private
  138
+
  139
+    def should_deflate?(env, status, headers, body)
  140
+      # Skip compressing empty entity body responses and responses with
  141
+      # no-transform set.
  142
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  143
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  144
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  145
+        return false
  146
+      end
  147
+
  148
+      # Skip if @compressible_types are given and does not include request's content type
  149
+      content_type = headers['Content-Type'].split(';')[0]
  150
+      return false if @compressible_types && !@compressible_types.include?(content_type)
2

@jayschab I can't see your comment anymore. Does it mean this way is fine?

jayschab
jayschab added a note January 15, 2014

Sorry I didn't forgot I was commenting on an outdated diff (it is still there just not obvious in the GitHub UX #457 (comment)). Copied below:

Found a bug in my suggested code in my testing. Content-Type isn't always set.

I ended up changing my version to this:

return false if @compressible_types && !(headers.has_key('Content-Type') && @compressible_types.include?(headers['Content-Type'][/[^;]*/]))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jakub Pawlowicz Adds improved Rack::Deflater tests. 8e52002
Jakub Pawlowicz Adds deflater options to control compression on per-request level.
* Adds :if option which should be given a lambda accepting env, status, headers, and body options.
* When :if evaluates to false a response body won't be compressed.
* Adds :include option which should be given an array of compressible content types.
* When :include don't include request's content type then response body won't be compressed.
3ae2f30
Jakub Pawlowicz jakubpawlowicz commented on the diff January 15, 2014
lib/rack/deflater.rb
@@ -126,5 +133,25 @@ def close
126 133
         @body.close if @body.respond_to?(:close)
127 134
       end
128 135
     end
  136
+
  137
+    private
  138
+
  139
+    def should_deflate?(env, status, headers, body)
  140
+      # Skip compressing empty entity body responses and responses with
  141
+      # no-transform set.
  142
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  143
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  144
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  145
+        return false
  146
+      end
  147
+
  148
+      # Skip if @compressible_types are given and does not include request's content type
  149
+      return false if @compressible_types && !(headers.has_key?('Content-Type') && @compressible_types.include?(headers['Content-Type'][/[^;]*/]))
1

@jayschab Updated it per your suggestion.

Found it problematic to test scenario without the 'Content-Type' header as it gets set every time in specs.
Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Jan 15, 2014
Jakub Pawlowicz Adds improved Rack::Deflater tests. 8e52002
Jakub Pawlowicz Adds deflater options to control compression on per-request level.
* Adds :if option which should be given a lambda accepting env, status, headers, and body options.
* When :if evaluates to false a response body won't be compressed.
* Adds :include option which should be given an array of compressible content types.
* When :include don't include request's content type then response body won't be compressed.
3ae2f30
This page is out of date. Refresh to see the latest.
39  lib/rack/deflater.rb
@@ -17,19 +17,26 @@ module Rack
17 17
   # directive of 'no-transform' is present, or when the response status
18 18
   # code is one that doesn't allow an entity body.
19 19
   class Deflater
20  
-    def initialize(app)
  20
+    ##
  21
+    # Creates Rack::Deflater middleware.
  22
+    #
  23
+    # [app] rack app instance
  24
+    # [options] hash of deflater options, i.e.
  25
+    #           'if' - a lambda enabling / disabling deflation based on returned boolean value
  26
+    #                  e.g use Rack::Deflater, :if => lambda { |env, status, headers, body| body.length > 512 }
  27
+    #           'include' - a list of content types that should be compressed
  28
+    def initialize(app, options = {})
21 29
       @app = app
  30
+
  31
+      @condition = options[:if]
  32
+      @compressible_types = options[:include]
22 33
     end
23 34
 
24 35
     def call(env)
25 36
       status, headers, body = @app.call(env)
26 37
       headers = Utils::HeaderHash.new(headers)
27 38
 
28  
-      # Skip compressing empty entity body responses and responses with
29  
-      # no-transform set.
30  
-      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
31  
-          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
32  
-         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  39
+      unless should_deflate?(env, status, headers, body)
33 40
         return [status, headers, body]
34 41
       end
35 42
 
@@ -126,5 +133,25 @@ def close
126 133
         @body.close if @body.respond_to?(:close)
127 134
       end
128 135
     end
  136
+
  137
+    private
  138
+
  139
+    def should_deflate?(env, status, headers, body)
  140
+      # Skip compressing empty entity body responses and responses with
  141
+      # no-transform set.
  142
+      if Utils::STATUS_WITH_NO_ENTITY_BODY.include?(status) ||
  143
+          headers['Cache-Control'].to_s =~ /\bno-transform\b/ ||
  144
+         (headers['Content-Encoding'] && headers['Content-Encoding'] !~ /\bidentity\b/)
  145
+        return false
  146
+      end
  147
+
  148
+      # Skip if @compressible_types are given and does not include request's content type
  149
+      return false if @compressible_types && !(headers.has_key?('Content-Type') && @compressible_types.include?(headers['Content-Type'][/[^;]*/]))
  150
+
  151
+      # Skip if @condition lambda is given and evaluates to false
  152
+      return false if @condition && !@condition.call(env, status, headers, body)
  153
+
  154
+      true
  155
+    end
129 156
   end
130 157
 end
431  test/spec_deflater.rb
@@ -6,199 +6,334 @@
6 6
 require 'zlib'
7 7
 
8 8
 describe Rack::Deflater do
9  
-  def deflater(app)
10  
-    Rack::Lint.new Rack::Deflater.new(app)
11  
-  end
12 9
 
13  
-  def build_response(status, body, accept_encoding, headers = {})
14  
-    body = [body]  if body.respond_to? :to_str
  10
+  def build_response(status, body, accept_encoding, options = {})
  11
+    body = [body] if body.respond_to? :to_str
15 12
     app = lambda do |env|
16  
-      res = [status, {}, body]
17  
-      res[1]["Content-Type"] = "text/plain" unless res[0] == 304
  13
+      res = [status, options['response_headers'] || {}, body]
  14
+      res[1]['Content-Type'] = 'text/plain' unless res[0] == 304
18 15
       res
19 16
     end
20  
-    request = Rack::MockRequest.env_for("", headers.merge("HTTP_ACCEPT_ENCODING" => accept_encoding))
21  
-    response = deflater(app).call(request)
22 17
 
23  
-    return response
24  
-  end
  18
+    request = Rack::MockRequest.env_for('', (options['request_headers'] || {}).merge('HTTP_ACCEPT_ENCODING' => accept_encoding))
  19
+    deflater = Rack::Lint.new Rack::Deflater.new(app, options['deflater_options'] || {})
25 20
 
26  
-  def inflate(buf)
27  
-    inflater = Zlib::Inflate.new(-Zlib::MAX_WBITS)
28  
-    inflater.inflate(buf) << inflater.finish
  21
+    deflater.call(request)
29 22
   end
30 23
 
31  
-  should "be able to deflate bodies that respond to each" do
32  
-    body = Object.new
33  
-    class << body; def each; yield("foo"); yield("bar"); end; end
34  
-
35  
-    response = build_response(200, body, "deflate")
36  
-
37  
-    response[0].should.equal(200)
38  
-    response[1].should.equal({
39  
-      "Content-Encoding" => "deflate",
40  
-      "Vary" => "Accept-Encoding",
41  
-      "Content-Type" => "text/plain"
42  
-    })
43  
-    buf = ''
44  
-    response[2].each { |part| buf << part }
45  
-    inflate(buf).should.equal("foobar")
46  
-  end
47  
-
48  
-  should "flush deflated chunks to the client as they become ready" do
49  
-    body = Object.new
50  
-    class << body; def each; yield("foo"); yield("bar"); end; end
  24
+  ##
  25
+  # Constructs response object and verifies if it yields right results
  26
+  #
  27
+  # [expected_status] expected response status, e.g. 200, 304
  28
+  # [expected_body] expected response body
  29
+  # [accept_encoing] what Accept-Encoding header to send and expect, e.g.
  30
+  #                  'deflate' - accepts and expects deflate encoding in response
  31
+  #                  { 'gzip' => nil } - accepts gzip but expects no encoding in response
  32
+  # [options] hash of request options, i.e.
  33
+  #           'app_status' - what status dummy app should return (may be changed by deflater at some point)
  34
+  #           'app_body' - what body dummy app should return (may be changed by deflater at some point)
  35
+  #           'request_headers' - extra reqest headers to be sent
  36
+  #           'response_headers' - extra response headers to be returned
  37
+  #           'deflater_options' - options passed to deflater middleware
  38
+  # [block] useful for doing some extra verification
  39
+  def verify(expected_status, expected_body, accept_encoding, options = {}, &block)
  40
+    accept_encoding, expected_encoding = if accept_encoding.kind_of?(Hash)
  41
+      [accept_encoding.keys.first, accept_encoding.values.first]
  42
+    else
  43
+      [accept_encoding, accept_encoding.dup]
  44
+    end
51 45
 
52  
-    response = build_response(200, body, "deflate")
  46
+    # build response
  47
+    status, headers, body = build_response(
  48
+      options['app_status'] || expected_status,
  49
+      options['app_body'] || expected_body,
  50
+      accept_encoding,
  51
+      options
  52
+    )
  53
+
  54
+    # verify status
  55
+    status.should.equal(expected_status)
  56
+
  57
+    # verify body
  58
+    unless options['skip_body_verify']
  59
+      body_text = ''
  60
+      body.each { |part| body_text << part }
  61
+
  62
+      deflated_body = case expected_encoding
  63
+      when 'deflate'
  64
+        inflater = Zlib::Inflate.new(-Zlib::MAX_WBITS)
  65
+        inflater.inflate(body_text) << inflater.finish
  66
+      when 'gzip'
  67
+        io = StringIO.new(body_text)
  68
+        gz = Zlib::GzipReader.new(io)
  69
+        tmp = gz.read
  70
+        gz.close
  71
+        tmp
  72
+      else
  73
+        body_text
  74
+      end
  75
+
  76
+      deflated_body.should.equal(expected_body)
  77
+    end
53 78
 
54  
-    response[0].should.equal(200)
55  
-    response[1].should.equal({
56  
-      "Content-Encoding" => "deflate",
57  
-      "Vary" => "Accept-Encoding",
58  
-      "Content-Type" => "text/plain"
59  
-    })
60  
-    buf = []
61  
-    inflater = Zlib::Inflate.new(-Zlib::MAX_WBITS)
62  
-    response[2].each { |part| buf << inflater.inflate(part) }
63  
-    buf << inflater.finish
64  
-    buf.delete_if { |part| part.empty? }
65  
-    buf.join.should.equal("foobar")
  79
+    # yield full response verification
  80
+    yield(status, headers, body) if block_given?
66 81
   end
67 82
 
68  
-  # TODO: This is really just a special case of the above...
69  
-  should "be able to deflate String bodies" do
70  
-    response = build_response(200, "Hello world!", "deflate")
  83
+  should 'be able to deflate bodies that respond to each' do
  84
+    app_body = Object.new
  85
+    class << app_body; def each; yield('foo'); yield('bar'); end; end
71 86
 
72  
-    response[0].should.equal(200)
73  
-    response[1].should.equal({
74  
-      "Content-Encoding" => "deflate",
75  
-      "Vary" => "Accept-Encoding",
76  
-      "Content-Type" => "text/plain"
77  
-    })
78  
-    buf = ''
79  
-    response[2].each { |part| buf << part }
80  
-    inflate(buf).should.equal("Hello world!")
  87
+    verify(200, 'foobar', 'deflate', { 'app_body' => app_body }) do |status, headers, body|
  88
+      headers.should.equal({
  89
+        'Content-Encoding' => 'deflate',
  90
+        'Vary' => 'Accept-Encoding',
  91
+        'Content-Type' => 'text/plain'
  92
+      })
  93
+    end
81 94
   end
82 95
 
83  
-  should "be able to gzip bodies that respond to each" do
84  
-    body = Object.new
85  
-    class << body; def each; yield("foo"); yield("bar"); end; end
  96
+  should 'flush deflated chunks to the client as they become ready' do
  97
+    app_body = Object.new
  98
+    class << app_body; def each; yield('foo'); yield('bar'); end; end
86 99
 
87  
-    response = build_response(200, body, "gzip")
  100
+    verify(200, app_body, 'deflate', { 'skip_body_verify' => true }) do |status, headers, body|
  101
+      headers.should.equal({
  102
+        'Content-Encoding' => 'deflate',
  103
+        'Vary' => 'Accept-Encoding',
  104
+        'Content-Type' => 'text/plain'
  105
+      })
88 106
 
89  
-    response[0].should.equal(200)
90  
-    response[1].should.equal({
91  
-      "Content-Encoding" => "gzip",
92  
-      "Vary" => "Accept-Encoding",
93  
-      "Content-Type" => "text/plain"
94  
-    })
  107
+      buf = []
  108
+      inflater = Zlib::Inflate.new(-Zlib::MAX_WBITS)
  109
+      body.each { |part| buf << inflater.inflate(part) }
  110
+      buf << inflater.finish
95 111
 
96  
-    buf = ''
97  
-    response[2].each { |part| buf << part }
98  
-    io = StringIO.new(buf)
99  
-    gz = Zlib::GzipReader.new(io)
100  
-    gz.read.should.equal("foobar")
101  
-    gz.close
  112
+      buf.delete_if { |part| part.empty? }.join.should.equal('foobar')
  113
+    end
102 114
   end
103 115
 
104  
-  should "flush gzipped chunks to the client as they become ready" do
105  
-    body = Object.new
106  
-    class << body; def each; yield("foo"); yield("bar"); end; end
  116
+  # TODO: This is really just a special case of the above...
  117
+  should 'be able to deflate String bodies' do
  118
+    verify(200, 'Hello world!', 'deflate') do |status, headers, body|
  119
+      headers.should.equal({
  120
+        'Content-Encoding' => 'deflate',
  121
+        'Vary' => 'Accept-Encoding',
  122
+        'Content-Type' => 'text/plain'
  123
+      })
  124
+    end
  125
+  end
107 126
 
108  
-    response = build_response(200, body, "gzip")
  127
+  should 'be able to gzip bodies that respond to each' do
  128
+    app_body = Object.new
  129
+    class << app_body; def each; yield('foo'); yield('bar'); end; end
109 130
 
110  
-    response[0].should.equal(200)
111  
-    response[1].should.equal({
112  
-      "Content-Encoding" => "gzip",
113  
-      "Vary" => "Accept-Encoding",
114  
-      "Content-Type" => "text/plain"
115  
-    })
116  
-    buf = []
117  
-    inflater = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
118  
-    response[2].each { |part| buf << inflater.inflate(part) }
119  
-    buf << inflater.finish
120  
-    buf.delete_if { |part| part.empty? }
121  
-    buf.join.should.equal("foobar")
  131
+    verify(200, 'foobar', 'gzip', { 'app_body' => app_body }) do |status, headers, body|
  132
+      headers.should.equal({
  133
+        'Content-Encoding' => 'gzip',
  134
+        'Vary' => 'Accept-Encoding',
  135
+        'Content-Type' => 'text/plain'
  136
+      })
  137
+    end
122 138
   end
123 139
 
124  
-  should "be able to fallback to no deflation" do
125  
-    response = build_response(200, "Hello world!", "superzip")
  140
+  should 'flush gzipped chunks to the client as they become ready' do
  141
+    app_body = Object.new
  142
+    class << app_body; def each; yield('foo'); yield('bar'); end; end
  143
+
  144
+    verify(200, app_body, 'gzip', { 'skip_body_verify' => true }) do |status, headers, body|
  145
+      headers.should.equal({
  146
+        'Content-Encoding' => 'gzip',
  147
+        'Vary' => 'Accept-Encoding',
  148
+        'Content-Type' => 'text/plain'
  149
+      })
  150
+
  151
+      buf = []
  152
+      inflater = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
  153
+      body.each { |part| buf << inflater.inflate(part) }
  154
+      buf << inflater.finish
126 155
 
127  
-    response[0].should.equal(200)
128  
-    response[1].should.equal({ "Vary" => "Accept-Encoding", "Content-Type" => "text/plain" })
129  
-    response[2].to_enum.to_a.should.equal(["Hello world!"])
  156
+      buf.delete_if { |part| part.empty? }.join.should.equal('foobar')
  157
+    end
130 158
   end
131 159
 
132  
-  should "be able to skip when there is no response entity body" do
133  
-    response = build_response(304, [], "gzip")
  160
+  should 'be able to fallback to no deflation' do
  161
+    verify(200, 'Hello world!', 'superzip') do |status, headers, body|
  162
+      headers.should.equal({
  163
+        'Vary' => 'Accept-Encoding',
  164
+        'Content-Type' => 'text/plain'
  165
+      })
  166
+    end
  167
+  end
134 168
 
135  
-    response[0].should.equal(304)
136  
-    response[1].should.equal({})
137  
-    response[2].to_enum.to_a.should.equal([])
  169
+  should 'be able to skip when there is no response entity body' do
  170
+    verify(304, '', { 'gzip' => nil }, { 'app_body' => [] }) do |status, headers, body|
  171
+      headers.should.equal({})
  172
+    end
138 173
   end
139 174
 
140  
-  should "handle the lack of an acceptable encoding" do
141  
-    response1 = build_response(200, "Hello world!", "identity;q=0", "PATH_INFO" => "/")
142  
-    response1[0].should.equal(406)
143  
-    response1[1].should.equal({"Content-Type" => "text/plain", "Content-Length" => "71"})
144  
-    response1[2].to_enum.to_a.should.equal(["An acceptable encoding for the requested resource / could not be found."])
  175
+  should 'handle the lack of an acceptable encoding' do
  176
+    app_body = 'Hello world!'
  177
+    not_found_body1 = 'An acceptable encoding for the requested resource / could not be found.'
  178
+    not_found_body2 = 'An acceptable encoding for the requested resource /foo/bar could not be found.'
  179
+    options1 = {
  180
+      'app_status' => 200,
  181
+      'app_body' => app_body,
  182
+      'request_headers' => {
  183
+        'PATH_INFO' => '/'
  184
+      }
  185
+    }
  186
+    options2 = {
  187
+      'app_status' => 200,
  188
+      'app_body' => app_body,
  189
+      'request_headers' => {
  190
+        'PATH_INFO' => '/foo/bar'
  191
+      }
  192
+    }
  193
+
  194
+    verify(406, not_found_body1, 'identity;q=0', options1) do |status, headers, body|
  195
+      headers.should.equal({
  196
+        'Content-Type' => 'text/plain',
  197
+        'Content-Length' => not_found_body1.length.to_s
  198
+      })
  199
+    end
145 200
 
146  
-    response2 = build_response(200, "Hello world!", "identity;q=0", "SCRIPT_NAME" => "/foo", "PATH_INFO" => "/bar")
147  
-    response2[0].should.equal(406)
148  
-    response2[1].should.equal({"Content-Type" => "text/plain", "Content-Length" => "78"})
149  
-    response2[2].to_enum.to_a.should.equal(["An acceptable encoding for the requested resource /foo/bar could not be found."])
  201
+    verify(406, not_found_body2, 'identity;q=0', options2) do |status, headers, body|
  202
+      headers.should.equal({
  203
+        'Content-Type' => 'text/plain',
  204
+        'Content-Length' => not_found_body2.length.to_s
  205
+      })
  206
+    end
150 207
   end
151 208
 
152  
-  should "handle gzip response with Last-Modified header" do
  209
+  should 'handle gzip response with Last-Modified header' do
153 210
     last_modified = Time.now.httpdate
  211
+    options = {
  212
+      'response_headers' => {
  213
+        'Content-Type' => 'text/plain',
  214
+        'Last-Modified' => last_modified
  215
+      }
  216
+    }
  217
+
  218
+    verify(200, 'Hello World!', 'gzip', options) do |status, headers, body|
  219
+      headers.should.equal({
  220
+        'Content-Encoding' => 'gzip',
  221
+        'Vary' => 'Accept-Encoding',
  222
+        'Last-Modified' => last_modified,
  223
+        'Content-Type' => 'text/plain'
  224
+      })
  225
+    end
  226
+  end
154 227
 
155  
-    app = lambda { |env| [200, { "Content-Type" => "text/plain", "Last-Modified" => last_modified }, ["Hello World!"]] }
156  
-    request = Rack::MockRequest.env_for("", "HTTP_ACCEPT_ENCODING" => "gzip")
157  
-    response = deflater(app).call(request)
  228
+  should 'do nothing when no-transform Cache-Control directive present' do
  229
+    options = {
  230
+      'response_headers' => {
  231
+        'Content-Type' => 'text/plain',
  232
+        'Cache-Control' => 'no-transform'
  233
+      }
  234
+    }
  235
+    verify(200, 'Hello World!', { 'gzip' => nil }, options) do |status, headers, body|
  236
+      headers.should.not.include 'Content-Encoding'
  237
+    end
  238
+  end
  239
+
  240
+  should 'do nothing when Content-Encoding already present' do
  241
+    options = {
  242
+      'response_headers' => {
  243
+        'Content-Type' => 'text/plain',
  244
+        'Content-Encoding' => 'gzip'
  245
+      }
  246
+    }
  247
+    verify(200, 'Hello World!', { 'gzip' => nil }, options)
  248
+  end
158 249
 
159  
-    response[0].should.equal(200)
160  
-    response[1].should.equal({
161  
-      "Content-Encoding" => "gzip",
162  
-      "Vary" => "Accept-Encoding",
163  
-      "Last-Modified" => last_modified,
164  
-      "Content-Type" => "text/plain"
165  
-    })
  250
+  should 'deflate when Content-Encoding is identity' do
  251
+    options = {
  252
+      'response_headers' => {
  253
+        'Content-Type' => 'text/plain',
  254
+        'Content-Encoding' => 'identity'
  255
+      }
  256
+    }
  257
+    verify(200, 'Hello World!', 'deflate', options)
  258
+  end
166 259
 
167  
-    buf = ''
168  
-    response[2].each { |part| buf << part }
169  
-    io = StringIO.new(buf)
170  
-    gz = Zlib::GzipReader.new(io)
171  
-    gz.read.should.equal("Hello World!")
172  
-    gz.close
  260
+  should "deflate if content-type matches :include" do
  261
+    options = {
  262
+      'response_headers' => {
  263
+        'Content-Type' => 'text/plain'
  264
+      },
  265
+      'deflater_options' => {
  266
+        :include => %w(text/plain)
  267
+      }
  268
+    }
  269
+    verify(200, 'Hello World!', 'gzip', options)
173 270
   end
174 271
 
175  
-  should "do nothing when no-transform Cache-Control directive present" do
176  
-    app = lambda { |env| [200, {'Content-Type' => 'text/plain', 'Cache-Control' => 'no-transform'}, ['Hello World!']] }
177  
-    request = Rack::MockRequest.env_for("", "HTTP_ACCEPT_ENCODING" => "gzip")
178  
-    response = deflater(app).call(request)
  272
+  should "deflate if content-type is included it :include" do
  273
+    options = {
  274
+      'response_headers' => {
  275
+        'Content-Type' => 'text/plain; charset=us-ascii'
  276
+      },
  277
+      'deflater_options' => {
  278
+        :include => %w(text/plain)
  279
+      }
  280
+    }
  281
+    verify(200, 'Hello World!', 'gzip', options)
  282
+  end
179 283
 
180  
-    response[0].should.equal(200)
181  
-    response[1].should.not.include "Content-Encoding"
182  
-    response[2].to_enum.to_a.join.should.equal("Hello World!")
  284
+  should "not deflate if content-type is not set but given in :include" do
  285
+    options = {
  286
+      'deflater_options' => {
  287
+        :include => %w(text/plain)
  288
+      }
  289
+    }
  290
+    verify(304, 'Hello World!', { 'gzip' => nil }, options)
183 291
   end
184 292
 
185  
-  should "do nothing when Content-Encoding already present" do
186  
-    app = lambda { |env| [200, {'Content-Type' => 'text/plain', 'Content-Encoding' => 'gzip'}, ['Hello World!']] }
187  
-    request = Rack::MockRequest.env_for("", "HTTP_ACCEPT_ENCODING" => "gzip")
188  
-    response = deflater(app).call(request)
  293
+  should "not deflate if content-type do not match :include" do
  294
+    options = {
  295
+      'response_headers' => {
  296
+        'Content-Type' => 'text/plain'
  297
+      },
  298
+      'deflater_options' => {
  299
+        :include => %w(text/json)
  300
+      }
  301
+    }
  302
+    verify(200, 'Hello World!', { 'gzip' => nil }, options)
  303
+  end
189 304
 
190  
-    response[0].should.equal(200)
191  
-    response[2].to_enum.to_a.join.should.equal("Hello World!")
  305
+  should "deflate response if :if lambda evaluates to true" do
  306
+    options = {
  307
+      'deflater_options' => {
  308
+        :if => lambda { |env, status, headers, body| true }
  309
+      }
  310
+    }
  311
+    verify(200, 'Hello World!', 'deflate', options)
192 312
   end
193 313
 
194  
-  should "deflate when Content-Encoding is identity" do
195  
-    app = lambda { |env| [200, {'Content-Type' => 'text/plain', 'Content-Encoding' => 'identity'}, ['Hello World!']] }
196  
-    request = Rack::MockRequest.env_for("", "HTTP_ACCEPT_ENCODING" => "deflate")
197  
-    response = deflater(app).call(request)
  314
+  should "not deflate if :if lambda evaluates to false" do
  315
+    options = {
  316
+      'deflater_options' => {
  317
+        :if => lambda { |env, status, headers, body| false }
  318
+      }
  319
+    }
  320
+    verify(200, 'Hello World!', { 'gzip' => nil }, options)
  321
+  end
198 322
 
199  
-    response[0].should.equal(200)
200  
-    buf = ''
201  
-    response[2].each { |part| buf << part }
202  
-    inflate(buf).should.equal("Hello World!")
  323
+  should "check for Content-Length via :if" do
  324
+    body = 'Hello World!'
  325
+    body_len = body.length
  326
+    options = {
  327
+      'response_headers' => {
  328
+        'Content-Length' => body_len.to_s
  329
+      },
  330
+      'deflater_options' => {
  331
+        :if => lambda { |env, status, headers, body|
  332
+          headers['Content-Length'].to_i >= body_len
  333
+        }
  334
+      }
  335
+    }
  336
+
  337
+    verify(200, body, 'gzip', options)
203 338
   end
204 339
 end
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.