copy_stream to SSL seems to read the file into memory #4842
Milestone
Comments
cfitz
changed the title
|
Ok, this must be one of the paths that doesn't chunk the transfer properly. It looks like it's falling into logic used for "fake" IO objects that only have a "read" method. We do have chunking logic elsewhere that should also be used for this. |
|
Oh, a workaround would be for copy_stream to receive a third length argument and chunk it manually, but obviously we don't want folks to have to do that. |
|
Reproduction below. This will OOM or NegativeArraySizeException depending on memory settings: fake = Class.new do
def initialize
@io = File.open("/dev/zero")
end
def read(length = -1)
if length == -1
@io.read
else
@io.read(length)
end
end
end.new
IO.copy_stream(fake, "/dev/null") |
headius
added a commit
that referenced
this issue
Nov 9, 2017
* Use arg2 for write argument in duck-typed logic. * Don't use a native ByteBuffer for streams we can't tell are native. * Guard ByteBuffer unwrapping in IOChannel * Update IO excludes.
|
Thanks, much appriciated. |
cfitz
added a commit
to cfitz/archivesspace
that referenced
this issue
Nov 9, 2017
There's a JRuby issue ( jruby/jruby#4842 ) that causes IOs to be read into memory is using SSL. This causes big problems if using Solr or the backend with an SSL socket. This adds a patch to chunk the IO manually, rather then just reading the whole thing in.
cfitz
added a commit
to cfitz/archivesspace
that referenced
this issue
Nov 9, 2017
There's a JRuby issue ( jruby/jruby#4842 ) that causes IOs to be read into memory is using SSL. This causes big problems if using Solr or the backend with an SSL socket. This adds a patch to chunk the IO manually, rather then just reading the whole thing in.
9 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
JRuby 9.1.8, Java 1.8.0, Fedora 25
I'm running a script ( included below ) that is streaming a big JSON file ( ~300mb ) to a Solr server that's using SSL authentication. Running on JRuby 9.1.8 ( all default flags ) , it pretty much immediately runs out of heap. If I bump the heap up to 2g, it will run a bit then run out of heap. A dump from VisualVM indicates ~98% of the memory is bytes[].
So, I think the IO is being put into memory when being copied to a SSL socket, I think..
Running on MRI and JRuby 1.7.9 do not have these issues, as it looks like the IO is being streamed out as expected.
Also, running the script on JRuby 9.1 withouth SSL runs without any memory issues.
Here's the script:
And here's the dump I get when I use jruby -w
The text was updated successfully, but these errors were encountered: