gzip is broken under some conditions #162

Closed
glebtv opened this Issue May 5, 2013 · 2 comments

Comments

Projects
None yet
2 participants
@glebtv

glebtv commented May 5, 2013

I'm not exactly shure how and why, but setting user-agent header together with transparent_gzip_decompression gives this error:

require 'httpclient'
clnt = HTTPClient.new
clnt.transparent_gzip_decompression = true
resp = clnt.request(:get, 'http://www.google.ru', {
    header: {
      'User-Agent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0',
    }
})
p resp.headers
/home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient/session.rb:697:in `inflate': incorrect data check (Zlib::DataError)
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient/session.rb:697:in `block in get_body'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient/session.rb:990:in `read_body_chunked'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient/session.rb:701:in `get_body'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient.rb:1088:in `do_get_block'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient.rb:887:in `block in do_request'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient.rb:981:in `protect_keep_alive_disconnected'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient.rb:886:in `do_request'
    from /home/gleb/.rvm/gems/ruby-2.0.0-p0@ml_vk/gems/httpclient-2.3.3/lib/httpclient.rb:774:in `request'
    from dl.rb:6:in `<main>'

And even more strangely, it works fine (no errors, contents are fine) if I enable debug with
clnt.debug_dev = STDERR

@glebtv

This comment has been minimized.

Show comment
Hide comment
@glebtv

glebtv May 5, 2013

  • In case google.ru redirects you somewhere else because of country, you can also try http://www.yandex.ru
  • User agent 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101' works perfectly (without last firefox part)
  • It seems that adding in firefox 20 enables "Transfer-Encoding: chunked" and removes Content-Length from response (google)
  • here are dumps from ff20 http://rscx.ru/ff20.txt and without ff20 http://rscx.ru/good.txt

glebtv commented May 5, 2013

  • In case google.ru redirects you somewhere else because of country, you can also try http://www.yandex.ru
  • User agent 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101' works perfectly (without last firefox part)
  • It seems that adding in firefox 20 enables "Transfer-Encoding: chunked" and removes Content-Length from response (google)
  • here are dumps from ff20 http://rscx.ru/ff20.txt and without ff20 http://rscx.ru/good.txt
@nahi

This comment has been minimized.

Show comment
Hide comment
@nahi

nahi Nov 3, 2014

Owner

Merged to #163, which is fixed perhaps by #167. Sorry for late resopnse.

Owner

nahi commented Nov 3, 2014

Merged to #163, which is fixed perhaps by #167. Sorry for late resopnse.

@nahi nahi closed this Nov 3, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment