Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Missing Zlib gzip bad stream exceptions #369

Open
drbrain opened this Issue Nov 1, 2012 · 9 comments

Comments

Projects
None yet
3 participants

drbrain commented Nov 1, 2012

When running the sparklemotion/mechanize tests I get the following errors when testing handling of bad gzip streams. Zlib::Error subclasses should be raised:

$ jruby -Ilib test/test_mechanize_http_agent.rb -n /test_response_content_encoding_gzip/
Unable to find a $JAVA_HOME at "/usr", continuing with system-provided Java...
Run options: -n /test_response_content_encoding_gzip/ --seed 64057

# Running tests:

..EFE..EE

Finished tests in 0.060000s, 150.0000 tests/s, 116.6667 assertions/s.

  1) Error:
test_response_content_encoding_gzip_checksum_truncated(TestMechanizeHttpAgent):
IOError: footer is not found
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:506:in `eof'
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:512:in `eof?'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:1124:in `auto_io'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:417:in `content_encoding_gunzip'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:788:in `response_content_encoding'
    test/test_mechanize_http_agent.rb:948:in `test_response_content_encoding_gzip_checksum_truncated'
    org/jruby/RubyBasicObject.java:1667:in `__send__'

  2) Failure:
test_response_content_encoding_gzip_corrupt(TestMechanizeHttpAgent) [test/test_mechanize_http_agent.rb:890]:
[Mechanize::Error] exception expected, not
Class: <IOError>
Message: <"Unexpected end of ZLIB input stream">
---Backtrace---
org/jruby/ext/zlib/JZlibRubyGzipReader.java:506:in `eof'
org/jruby/ext/zlib/JZlibRubyGzipReader.java:512:in `eof?'
/Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:1124:in `auto_io'
/Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:417:in `content_encoding_gunzip'
/Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:788:in `response_content_encoding'
test/test_mechanize_http_agent.rb:891:in `test_response_content_encoding_gzip_corrupt'
---------------

  3) Error:
test_response_content_encoding_gzip_no_footer(TestMechanizeHttpAgent):
IOError: footer is not found
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:506:in `eof'
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:512:in `eof?'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:1124:in `auto_io'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:417:in `content_encoding_gunzip'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:788:in `response_content_encoding'
    test/test_mechanize_http_agent.rb:987:in `test_response_content_encoding_gzip_no_footer'
    org/jruby/RubyBasicObject.java:1667:in `__send__'

  4) Error:
test_response_content_encoding_gzip_checksum_corrupt_crc(TestMechanizeHttpAgent):
IOError: incorrect data check
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:506:in `eof'
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:512:in `eof?'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:1124:in `auto_io'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:417:in `content_encoding_gunzip'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:788:in `response_content_encoding'
    test/test_mechanize_http_agent.rb:914:in `test_response_content_encoding_gzip_checksum_corrupt_crc'
    org/jruby/RubyBasicObject.java:1667:in `__send__'

  5) Error:
test_response_content_encoding_gzip_checksum_corrupt_length(TestMechanizeHttpAgent):
IOError: incorrect length check
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:506:in `eof'
    org/jruby/ext/zlib/JZlibRubyGzipReader.java:512:in `eof?'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:1124:in `auto_io'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:417:in `content_encoding_gunzip'
    /Users/drbrain/Work/git/mechanize/lib/mechanize/http/agent.rb:788:in `response_content_encoding'
    test/test_mechanize_http_agent.rb:932:in `test_response_content_encoding_gzip_checksum_corrupt_length'
    org/jruby/RubyBasicObject.java:1667:in `__send__'

9 tests, 7 assertions, 1 failures, 4 errors, 0 skips
Owner

headius commented Nov 2, 2012

So I looked at this with @drbrain a bit. The problem here is that jzlib is basically just raising IOException for all these error conditions that could be independent exception types. The result is that we can't distinguish them other than by their messages, which is about as brittle as it can be.

A short term ugly fix would be to check the error messages and raise the right Ruby exception based on that. The right fix is to patch jzlib to raise more identifiable errors for these cases, so we can re-raise correctly.

Contributor

knu commented Sep 8, 2016

I've made the test suite skip some error tests for JRuby, but there are still problems with JRuby 9000. JRuby 9.1.5.0 seems to raise IOError: footer is not found for a valid stream that JRuby 1.7 can read fine: https://travis-ci.org/sparklemotion/mechanize/jobs/158341977#L394

Any idea?

Owner

headius commented Sep 8, 2016

Well, this rises from the history books! I'm not sure what would cause that error. You can pass -Xbacktrace.style=full to get a mixed Ruby/Java backtrace that will pinpoint where it's coming from.

JRuby 9k uses native (FFI-based) IO for most file and stdio streams now, so it's certainly possible they're interacting badly with jzlib. Let's figure it out!

Contributor

knu commented Sep 9, 2016

OK, I'll kick a test build shortly!

Contributor

knu commented Sep 12, 2016

Somehow build with JRuby 1.7.26 got broken by adding the option: https://travis-ci.org/sparklemotion/mechanize/jobs/158705200

Owner

headius commented Sep 13, 2016

@knu Strange that 1.7 broke, but the result on 9k looks good. We should be able to sort out why we're raising an IOError now and compare it with MRI.

Owner

headius commented Sep 13, 2016

I am able to reproduce locally but I do not know why we're raising these errors just yet.

@headius headius added this to the JRuby 9.1.6.0 milestone Sep 13, 2016

Owner

headius commented Sep 13, 2016

@ymnk You still out there? :-)

I'm not sure why we get these errors. In both cases, JZlibRubyGzipReader.read and eof just try to read from a buffered stream around a jzlib gzip stream. Why would we be seeing these "footer" errors?

@headius headius modified the milestones: JRuby 9.2.0.0, JRuby 9.1.6.0 Nov 8, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment