Join GitHub today
copy_stream to SSL seems to read the file into memory #4842
JRuby 9.1.8, Java 1.8.0, Fedora 25
I'm running a script ( included below ) that is streaming a big JSON file ( ~300mb ) to a Solr server that's using SSL authentication. Running on JRuby 9.1.8 ( all default flags ) , it pretty much immediately runs out of heap. If I bump the heap up to 2g, it will run a bit then run out of heap. A dump from VisualVM indicates ~98% of the memory is bytes.
Running on MRI and JRuby 1.7.9 do not have these issues, as it looks like the IO is being streamed out as expected.
Here's the script:
And here's the dump I get when I use jruby -w
Reproduction below. This will OOM or NegativeArraySizeException depending on memory settings:
fake = Class.new do def initialize @io = File.open("/dev/zero") end def read(length = -1) if length == -1 @io.read else @io.read(length) end end end.new IO.copy_stream(fake, "/dev/null")
added a commit
Nov 9, 2017
Thanks, much appriciated.