Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO.copy_stream replaces read chunk even if it's duplicated #5048

Closed
janko opened this Issue Feb 16, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@janko
Copy link

janko commented Feb 16, 2018

Environment

$ jruby -v
jruby 9.1.15.0 (2.3.3) 2017-12-07 929fde8 Java HotSpot(TM) 64-Bit Server VM 25.40-b25 on 1.8.0_40-b27 +jit [darwin-x86_64]
$ uname -a
Darwin Jankos-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Expected Behavior

I'm using a trick in HTTP.rb where I utilize IO.copy_stream to yield chunks of data into a block. However, the tests for this behaviour are failing on JRuby.

I managed to isolate the problem in this script:

require "stringio"

class ProcIO
  def initialize(&block)
    @block = block
  end

  def write(data)
    @block.call(data)
    data.bytesize
  end
end

chunks = []

stringio = StringIO.new("a" * 16*1024 + "b" * 10*1024)
procio   = ProcIO.new { |data| chunks << data.dup }

IO.copy_stream(stringio, procio)

puts chunks

On MRI this prints 16*1024 characters a and 10*1024 characters b.

Actual Behavior

On JRuby this script prints 3 x 8*1024 characters b and 1 x 2*1024 characters b. This totals to the correct number of characters, but they're all b. Since JRuby seems to use chunk size of 8KB (unlike MRI which uses 16KB), I expected the output to be 2 x 8*1024 characters a, 1 x 8*1024 characters b and 1 x 2*1024 characters b.

I tried printing each chunk as it's yielded:

procio   = ProcIO.new { |data| chunks << p(data.dup) }

And the chunks were in fact correct. So it seems that, even though I duplicated the string, JRuby still somehow overrode it when reading subsequent chunks. And in a strange way too, I would have expected to get 4x the last chunk (2*1024 characters b), but instead JRuby kept the string length and still somehow replaced the content.

@headius

This comment has been minimized.

Copy link
Member

headius commented Feb 16, 2018

I'm not at a terminal now, hit can you try this with a snapshot of 9.1.16? I did a bunch of work this cycle to improve copy_stream.

@janko

This comment has been minimized.

Copy link
Author

janko commented Feb 16, 2018

I successfully build JRuby from master by running ./mvnw, but when I try to run bin/jruby bug.rb I get

Error: Could not find or load main class org.jruby.Main

I tried running rbenv shell system just in case, but it still throws the same error.

@headius

This comment has been minimized.

Copy link
Member

headius commented Feb 16, 2018

Try downloading a nightly from the link http://jruby.org/download

@janko

This comment has been minimized.

Copy link
Author

janko commented Feb 16, 2018

@headius Strange, I get the same error when using the executable from the nightly link.

@headius

This comment has been minimized.

Copy link
Member

headius commented Feb 16, 2018

Yeah really seems like an env issue. Try a clean shell, make sure no vars are redirecting commands somewhere they should not be. I'll have a look tomorrow too.

@janko

This comment has been minimized.

Copy link
Author

janko commented Feb 16, 2018

@headius You're right, running from a clean shell worked 👍

And yes, the script now produces correct output on the 9.1.16.0 snapshot, which is great. Thank you very much! ❤️

@janko janko closed this Feb 16, 2018

@headius

This comment has been minimized.

Copy link
Member

headius commented Feb 16, 2018

It seems likely this was the same issue as #4093. I wish you had filed first...that one was complicated to diagnose! 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.