Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File.seek fails on large file #3435

Closed
jegt opened this issue Oct 30, 2015 · 9 comments
Closed

File.seek fails on large file #3435

jegt opened this issue Oct 30, 2015 · 9 comments

Comments

@jegt
Copy link

@jegt jegt commented Oct 30, 2015

File.seek fails on files 2GB or larger on Solaris and Linux.

Test program:
[2147483647, 2147483648].each do |position|
file = File.open('test.tar', 'rb')
file.seek(position)
puts "seek to #{position} succeeded"
end

output:
seek to 2147483647 succeeded
Errno::EBADF: Bad file number - test.tar
seek at org/jruby/RubyIO.java:1648
block in seek.rb at seek.rb:3
each at org/jruby/RubyArray.java:1560
at seek.rb:1

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa Java HotSpot(TM) 64-Bit Server VM 25.65-b01 on 1.8.0_65-b17 +jit [SunOS-amd64]

SunOS org00 5.11 omnios-f090f73 i86pc i386 i86pc

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa OpenJDK 64-Bit Server VM 25.45-b02 on 1.8.0_45-internal-b14 +jit [linux-amd64]

Linux worker04 3.13.0-67-generic #110-Ubuntu SMP Fri Oct 23 13:24:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I discovered this when trying to use multipart uploads to S3 using the official aws-sdk gem.

When using the gem i got mostly Errno::EAGAIN and sometimes Errno::ENOENT. The test program seems to always fail with Errno::EBADF.

The same test program works fine on MRI: ruby 2.1.3p242 (2014-09-19 revision 47630) [i386-solaris2.11]

@jegt jegt changed the title File.seek fails on large file on Solaris File.seek fails on large file Nov 7, 2015
@gucki
Copy link

@gucki gucki commented Jan 22, 2016

The same happens when using tell on a file where the position is > 2 GB.

irb(main):024:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>

irb(main):030:0> f.tell
=> 0

irb(main):025:0> puts f.size
4294967296
=> nil

irb(main):026:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):26:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):027:0> puts f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):27:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):028:0> f.close
=> nil

I tried with latest ubuntu 64-bit openjdk and

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
@gucki
Copy link

@gucki gucki commented Jan 22, 2016

It seems like this is a regression of 9.0.4, as it works with 1.7.21:

jruby 1.7.21 (1.9.3p551) 2015-07-07 a741a82 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
=> 0
irb(main):005:0> f.tell
=> 2147483648
irb(main):006:0> f.seek(3*1024*1024*1024)
=> 0
irb(main):007:0> f.tell
=> 3221225472
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):4:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):005:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):5:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):006:0> f.seek(3*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):6:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):007:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):7:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit
@gucki
Copy link

@gucki gucki commented Jan 22, 2016

@headius Would be great to have this fixed in 9.0.5 👍

@gucki
Copy link

@gucki gucki commented Mar 4, 2016

Is there anything we could do to get this fixed, liking sponsoring the fix?

@enebo enebo added this to the JRuby 9.1.0.0 milestone Mar 7, 2016
@enebo
Copy link
Member

@enebo enebo commented Mar 7, 2016

This almost sounds like we switch IO pos from a long to an int? Marking for 9.1.0.0.

@headius
Copy link
Member

@headius headius commented Apr 20, 2016

JRuby 9.1 is using a long for position in seek, but still failing. Looking into it.

@headius
Copy link
Member

@headius headius commented Apr 20, 2016

Ok, so there's one small bug and one big bug here.

The small bug is that we were expecting all negative return values from tell and lseek to mean there was an error. This is incorrect; our IO#seek actually works fine, but the return value is the new position, and because our native lseek binding only has a 32-bit signed return value, this comes out negative.

The big bug is that our native lseek binding and many paths leading to it all use 32-bit int for the return value.

I'm fixing the first one, which makes seek and tell at least function properly at the native IO level:

[] ~/projects/jruby $ jruby -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
-2147483648

[] ~/projects/jruby $ ruby23 -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
2147483648

Note the second larger problem rears its head here with IO#tell being unable to return file positions larger than a signed 32-bit integer.

I'm going to mark this fixed, since at least the error is gone now and seek is doing what it should under the covers. I'll open a new bug for the larger issue.

@headius
Copy link
Member

@headius headius commented Apr 20, 2016

See #3817 for the larger issue.

@gucki
Copy link

@gucki gucki commented Apr 20, 2016

@headius Thanks! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.