Improve `httpcompression` #162

Closed
wants to merge 1 commit into
from

Projects

None yet

3 participants

@alrra

Improvements:

  • correctly detects SDCH compression according to the specification
  • correctly overwrites the user agent (using the -A, not the -H option)
  • limits the connection and operation time (this is useful for preventing the function from hanging for hours due to slow networks or links going down)
  • shows errors
  • shows redirects

Tested on: Mac OS X v.10.8.2 and Ubuntu 12.04 LTS


Examples:

Input Output [old] Output [new]
adobe.com
adobe.com is not using any encoding
adobe.com 
 ↳ http://www.adobe.com/ [gzip]
"http://www.google.com/s?q=alrra&output=search"
http://www.google.com/s?q=alrra&output=search is encoded using gzip
http://www.google.com/s?q=alrra&output=search [sdch,gzip]
firefox.com
             
firefox.com is encoded using gzip
Content-Encoding: gzip
Content-Encoding: gzip
Content-Encoding: gzip
             
firefox.com 
 ↳ http://www.firefox.com/ 
   ↳ http://www.mozilla.org/firefox/ 
     ↳ http://www.mozilla.org/en-US/firefox/new/ [gzip]
         
google.com/+
             
google.com/+ is encoded using gzip
             
google.com/+ 
 ↳ http://www.google.com/+ 
   ↳ https://plus.google.com/ [gzip]
     ↳ https://accounts.google.com/ServiceLogin?service=oz&continue=https://plus.google.com/?gpsrc%3Dgplp0&hl=ro [gzip]
        
sadsada.dsa
sadsada.dsa is not using any encoding
curl: (6) Could not resolve host: sadsada.dsa; nodename nor servname provided, or not known
@porada

Great piece of code. Haven’t you considered releasing this as a separate tool?

@mathiasbynens

Incredibly nice work, @alrra. I too feel like this belongs to its own file now, e.g. ~/bin/httpcompression. What do you think?

@alrra

I too feel like this belongs to its own file now, e.g. ~/bin/httpcompression. What do you think?

@mathiasbynens Done, changed the pull request. Also, feel free to make any additional / required changes.

Haven’t you considered releasing this as a separate tool?

@porada Are you referring to the same thing as @mathiasbynens ?

@alrra alrra Improve `httpcompression`
- correctly detect `SDCH` compression, see:
  http://www.blogs.zeenor.com/wp-content/uploads/2011/01/Shared_Dictionary_Compression_over_HTTP.pdf
- correctly overwrite the user agent (use the -A, not the -H option)
- limit the connection and operation time (useful for preventing curl
  from hanging for hours due to slow networks or links going down)
- show errors
- show redirects
cf1022f
@mathiasbynens

Finally merged; thanks!

@mrkd mrkd added a commit to mrkd/dotfiles that referenced this pull request Apr 21, 2014
@alrra alrra .functions: Improve `httpcompression` and move it to its own file
Closes #162.
cefafc4
@thorsten thorsten pushed a commit to thorsten/dotfiles that referenced this pull request Dec 12, 2014
@alrra alrra .functions: Improve `httpcompression` and move it to its own file
Closes #162.
11e056e
@dmcass dmcass pushed a commit to dmcass/windows-dotfiles that referenced this pull request Mar 10, 2015
@alrra alrra .functions: Improve `httpcompression` and move it to its own file
Closes #162.
5ee928b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment