New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request for gzip Compression for HTML / CSS / JS #92

Closed
ghost opened this Issue Aug 20, 2010 · 9 comments

Comments

Projects
None yet
3 participants
@ghost

ghost commented Aug 20, 2010

I'd like to see optional gzip compression in Bottle, mainly because I have some bigger static JavaScripts in my project. If you reject this request, maybe you could give me a hint where it would be best to add this functionality?

Thanks in advance.

Thomas

@defnull

This comment has been minimized.

Show comment
Hide comment
@defnull

defnull Aug 20, 2010

Member

There is a paste middleware that does this, but it is quite limited:
http://pythonpaste.org/modules/gzipper.html#paste.gzipper.middleware

You are not the only one that requested such a feature. I'll try to add this to the core.

Member

defnull commented Aug 20, 2010

There is a paste middleware that does this, but it is quite limited:
http://pythonpaste.org/modules/gzipper.html#paste.gzipper.middleware

You are not the only one that requested such a feature. I'll try to add this to the core.

@royshan

This comment has been minimized.

Show comment
Hide comment
@royshan

royshan Feb 1, 2011

recently, we released a wsgi gzip middleware. it only compresses selected types of contents. also it is small and easy to configure. check it out at http://code.google.com/p/ibkon-wsgi-gzip-middleware/ .

royshan commented Feb 1, 2011

recently, we released a wsgi gzip middleware. it only compresses selected types of contents. also it is small and easy to configure. check it out at http://code.google.com/p/ibkon-wsgi-gzip-middleware/ .

@sc68cal

This comment has been minimized.

Show comment
Hide comment
@sc68cal

sc68cal Apr 26, 2011

Contributor

I know this is quite an old issue, but I think that GZIP is out of the purview of Bottle, and should be handled by middleware, similar to how authentication is handled by a middleware.

In fact, wouldn't Flup be well suited for this?

EDIT: Looks like those modules were removed.

Contributor

sc68cal commented Apr 26, 2011

I know this is quite an old issue, but I think that GZIP is out of the purview of Bottle, and should be handled by middleware, similar to how authentication is handled by a middleware.

In fact, wouldn't Flup be well suited for this?

EDIT: Looks like those modules were removed.

@defnull

This comment has been minimized.

Show comment
Hide comment
@defnull

defnull Apr 26, 2011

Member

EDIT: Looks like those modules were removed.

For a reason, I suppose. It is not that easy to get compression right. You need to:

  • Compress on the fly and be fast doing so.
  • Do not compress for browsers that don't support it.
  • Do not compress files that are compressed already (images, videos).
  • Do not compress dynamic files.
  • Support two differed compression algorithms (gzip and deflate).
  • Cache compressed files that don't change often.
  • De-validate the cache if one of the files changed anyway.
  • Make sure the cache does not get to big.
  • Do not cache small files because a disc-access would take longer than on-the-fly compression.

And many other edge cases I don't remember. I currently do not now a WSGI middleware that does all this in a satisfying way. It is usually better solved by a web-server module (lighttpd+mod_compress or Apache+mod_deflate).

Since 512048b, static_file adds a Content-Encoding header if the file appears to be compressed. For example, a big_script.js.gz file is sent with Content-Type: application/javascript and Content-Encoding: gzip headers.

If you only want to speed up some big CSS and JavaScript files, you could compress them manually and add some logic to your static-file route:

@route('/static/:filename#.*#')
def static(filename):
if 'gzip' in request.headers.get('Accept-Encoding'):
out = static_file(filename+'.gz', '/path/to/statics/')
if not isinstance(out, HTTPError):
return out
return static_file(filename, '/path/to/statics/')

This searches for a file with a '.gz' extension and returns that if the browser supports it. If not, the uncompressed file is returned.

If done right, something like this could be added to the core I suppose. Pior [1] played with that Idea some time ago. His branch is still interesting.

[1] https://github.com/pior/bottle

Member

defnull commented Apr 26, 2011

EDIT: Looks like those modules were removed.

For a reason, I suppose. It is not that easy to get compression right. You need to:

  • Compress on the fly and be fast doing so.
  • Do not compress for browsers that don't support it.
  • Do not compress files that are compressed already (images, videos).
  • Do not compress dynamic files.
  • Support two differed compression algorithms (gzip and deflate).
  • Cache compressed files that don't change often.
  • De-validate the cache if one of the files changed anyway.
  • Make sure the cache does not get to big.
  • Do not cache small files because a disc-access would take longer than on-the-fly compression.

And many other edge cases I don't remember. I currently do not now a WSGI middleware that does all this in a satisfying way. It is usually better solved by a web-server module (lighttpd+mod_compress or Apache+mod_deflate).

Since 512048b, static_file adds a Content-Encoding header if the file appears to be compressed. For example, a big_script.js.gz file is sent with Content-Type: application/javascript and Content-Encoding: gzip headers.

If you only want to speed up some big CSS and JavaScript files, you could compress them manually and add some logic to your static-file route:

@route('/static/:filename#.*#')
def static(filename):
if 'gzip' in request.headers.get('Accept-Encoding'):
out = static_file(filename+'.gz', '/path/to/statics/')
if not isinstance(out, HTTPError):
return out
return static_file(filename, '/path/to/statics/')

This searches for a file with a '.gz' extension and returns that if the browser supports it. If not, the uncompressed file is returned.

If done right, something like this could be added to the core I suppose. Pior [1] played with that Idea some time ago. His branch is still interesting.

[1] https://github.com/pior/bottle

@sc68cal

This comment has been minimized.

Show comment
Hide comment
@sc68cal

sc68cal Apr 26, 2011

Contributor
    For a reason, I suppose. It is not that easy to get compression right. You need to:
    ...
    And many other edge cases I don't remember. I currently do not now a WSGI middleware that does 
    all this in a satisfying way. 
    It is usually better solved by a web-server module (lighttpd+mod_compress or Apache+mod_deflate).

Agreed. I think re-inventing the wheel to solve all these edge cases is a waste of resources, it would be much better to document the advantages and disadvantages to WSGI middleware, and then also mention web-server modules that implement it better, while linking some middleware projects in the docs. I'd be happy to implement that.

Contributor

sc68cal commented Apr 26, 2011

    For a reason, I suppose. It is not that easy to get compression right. You need to:
    ...
    And many other edge cases I don't remember. I currently do not now a WSGI middleware that does 
    all this in a satisfying way. 
    It is usually better solved by a web-server module (lighttpd+mod_compress or Apache+mod_deflate).

Agreed. I think re-inventing the wheel to solve all these edge cases is a waste of resources, it would be much better to document the advantages and disadvantages to WSGI middleware, and then also mention web-server modules that implement it better, while linking some middleware projects in the docs. I'd be happy to implement that.

@defnull

This comment has been minimized.

Show comment
Hide comment
@defnull

defnull Apr 26, 2011

Member

I am awaiting your pull request :) (and closing this issue)

Member

defnull commented Apr 26, 2011

I am awaiting your pull request :) (and closing this issue)

@defnull defnull closed this Apr 26, 2011

@sc68cal

This comment has been minimized.

Show comment
Hide comment
@sc68cal

sc68cal Apr 26, 2011

Contributor

Actually, could you reopen it? That way when you merge the pull request you can put in a "closes #92" in the message and it'll link up the history.

Contributor

sc68cal commented Apr 26, 2011

Actually, could you reopen it? That way when you merge the pull request you can put in a "closes #92" in the message and it'll link up the history.

@defnull defnull reopened this Apr 26, 2011

@sc68cal

This comment has been minimized.

Show comment
Hide comment
@sc68cal

sc68cal May 12, 2011

Contributor

Sorry for the noise, rebased off your master then pushed again. I'm rushing too much and getting sloppy.

Contributor

sc68cal commented May 12, 2011

Sorry for the noise, rebased off your master then pushed again. I'm rushing too much and getting sloppy.

sc68cal added a commit to sc68cal/bottle that referenced this issue May 12, 2011

sc68cal added a commit to sc68cal/bottle that referenced this issue May 12, 2011

@sc68cal

This comment has been minimized.

Show comment
Hide comment
@sc68cal

sc68cal May 12, 2011

Contributor

sighs - Ok - this time issue_92 is a clean branch you can merge. Had a commit that belongs in a different pull request.

Contributor

sc68cal commented May 12, 2011

sighs - Ok - this time issue_92 is a clean branch you can merge. Had a commit that belongs in a different pull request.

defnull added a commit that referenced this issue May 12, 2011

@defnull defnull closed this May 12, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment