Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Direct Memory Buffer Leak in File IO #863

Closed
stewi2 opened this Issue Jul 9, 2013 · 3 comments

Comments

Projects
None yet
3 participants
@stewi2
Copy link

commented Jul 9, 2013

Apparently the JRuby File class allocates NIO direct memory buffers, but doesn't release them. I wrote the following test program (on jruby-1.7.4 on Java 1.7 on OSX 10.8.4):

i=0
loop do
  i=i+1
  File.open('/dev/null','r') do |infile|
  end
  puts i
  sleep 0.1
end

When I run this with a restricted MaxDirectMemorySize using:

jruby  -J-XX:MaxDirectMemorySize=200K -w leak.rb

it crashes very quickly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Error: Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Exception trace follows:
java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:658)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
    at org.jruby.util.io.ChannelDescriptor.<init>(ChannelDescriptor.java:663)
    at org.jruby.util.io.ChannelDescriptor.<init>(ChannelDescriptor.java:217)
    at org.jruby.util.io.ChannelDescriptor.open(ChannelDescriptor.java:797)
    at org.jruby.RubyFile.sysopen(RubyFile.java:1242)
    at org.jruby.RubyFile.sysopenInternal(RubyFile.java:1199)
    at org.jruby.RubyFile.openFile19(RubyFile.java:1150)
    at org.jruby.RubyFile.initialize19(RubyFile.java:334)
    at org.jruby.RubyFile$INVOKER$i$0$2$initialize19.call(RubyFile$INVOKER$i$0$2$initialize19.gen)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:85)
    at org.jruby.RubyClass.newInstance(RubyClass.java:876)
    at org.jruby.RubyIO.open(RubyIO.java:1144)
    at org.jruby.RubyIO$INVOKER$s$0$0$open.call(RubyIO$INVOKER$s$0$0$open.gen)
    at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:217)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:211)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
    at leak.block_0$RUBY$__file__(leak.rb:6)
    at leak$block_0$RUBY$__file__.call(leak$block_0$RUBY$__file__)
    at org.jruby.runtime.CompiledBlock19.yieldSpecificInternal(CompiledBlock19.java:117)
    at org.jruby.runtime.CompiledBlock19.yieldSpecific(CompiledBlock19.java:92)
    at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
    at org.jruby.RubyKernel.loop(RubyKernel.java:1489)
    at org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
    at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
    at leak.__file__(leak.rb:4)
    at leak.load(leak.rb)
    at org.jruby.Ruby.runScript(Ruby.java:807)
    at org.jruby.Ruby.runScript(Ruby.java:800)
    at org.jruby.Ruby.runNormally(Ruby.java:669)
    at org.jruby.Ruby.runFromMain(Ruby.java:518)
    at org.jruby.Main.doRunFromMain(Main.java:390)
    at org.jruby.Main.internalRun(Main.java:279)
    at org.jruby.Main.run(Main.java:221)
    at org.jruby.Main.main(Main.java:201)

I tried the same thing by simply opening and closing a FileInputStream, and the memory usage stays constant. I used https://blogs.oracle.com/nbprofiler/entry/new_memory_buffer_plugin_for to monitor buffer usage.

@atambo

This comment has been minimized.

Copy link
Member

commented Jul 9, 2013

I ran your script against 1.7.4 and reproduced your OutOfMemoryError. But when I ran it against master it didn't run out of memory so looks like it is already fixed and will be in 1.7.5.

@atambo atambo closed this Jul 9, 2013

@headius

This comment has been minimized.

Copy link
Member

commented Jul 9, 2013

I also ran this on Java 6 and Java 7. Java 6 stabilized very early at around 112MB. I let it run for about 8000 iterations (after dropping the sleep down to 0.01s). Java 7 stabilized higher, around 171MB, after running up to 109k iterations. I'm convinced that this is fixed as well.

Free free to confirm with master and reopen if there are any further issues. Thanks for the report!

@stewi2

This comment has been minimized.

Copy link
Author

commented Jul 9, 2013

Thanks guys for the quick response. I also verified that this does appear to be fixed in master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.