New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File.seek fails on large file #3435

Closed
jegt opened this Issue Oct 30, 2015 · 9 comments

Comments

Projects
None yet
5 participants
@jegt

jegt commented Oct 30, 2015

File.seek fails on files 2GB or larger on Solaris and Linux.

Test program:
[2147483647, 2147483648].each do |position|
file = File.open('test.tar', 'rb')
file.seek(position)
puts "seek to #{position} succeeded"
end

output:
seek to 2147483647 succeeded
Errno::EBADF: Bad file number - test.tar
seek at org/jruby/RubyIO.java:1648
block in seek.rb at seek.rb:3
each at org/jruby/RubyArray.java:1560
at seek.rb:1

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa Java HotSpot(TM) 64-Bit Server VM 25.65-b01 on 1.8.0_65-b17 +jit [SunOS-amd64]

SunOS org00 5.11 omnios-f090f73 i86pc i386 i86pc

jruby 9.0.3.0 (2.2.2) 2015-10-21 633c9aa OpenJDK 64-Bit Server VM 25.45-b02 on 1.8.0_45-internal-b14 +jit [linux-amd64]

Linux worker04 3.13.0-67-generic #110-Ubuntu SMP Fri Oct 23 13:24:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I discovered this when trying to use multipart uploads to S3 using the official aws-sdk gem.

When using the gem i got mostly Errno::EAGAIN and sometimes Errno::ENOENT. The test program seems to always fail with Errno::EBADF.

The same test program works fine on MRI: ruby 2.1.3p242 (2014-09-19 revision 47630) [i386-solaris2.11]

@jegt jegt changed the title from File.seek fails on large file on Solaris to File.seek fails on large file Nov 7, 2015

@gucki

This comment has been minimized.

Show comment
Hide comment
@gucki

gucki Jan 22, 2016

The same happens when using tell on a file where the position is > 2 GB.

irb(main):024:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>

irb(main):030:0> f.tell
=> 0

irb(main):025:0> puts f.size
4294967296
=> nil

irb(main):026:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):26:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):027:0> puts f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):27:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):028:0> f.close
=> nil

I tried with latest ubuntu 64-bit openjdk and

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

gucki commented Jan 22, 2016

The same happens when using tell on a file where the position is > 2 GB.

irb(main):024:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>

irb(main):030:0> f.tell
=> 0

irb(main):025:0> puts f.size
4294967296
=> nil

irb(main):026:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):26:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):027:0> puts f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):27:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:90:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/console.rb:9:in `start'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:69:in `console'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
  from /home/gucki/.rbenv/versions/jruby-9.0.4.0/lib/ruby/gems/shared/gems/railties-4.1.14/lib/rails/commands.rb:17:in `<top>'
  from org/jruby/RubyKernel.java:939:in `require'
  from bin/rails:4:in `<top>'

irb(main):028:0> f.close
=> nil

I tried with latest ubuntu 64-bit openjdk and

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
@gucki

This comment has been minimized.

Show comment
Hide comment
@gucki

gucki Jan 22, 2016

It seems like this is a regression of 9.0.4, as it works with 1.7.21:

jruby 1.7.21 (1.9.3p551) 2015-07-07 a741a82 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
=> 0
irb(main):005:0> f.tell
=> 2147483648
irb(main):006:0> f.seek(3*1024*1024*1024)
=> 0
irb(main):007:0> f.tell
=> 3221225472
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):4:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):005:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):5:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):006:0> f.seek(3*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):6:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):007:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):7:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

gucki commented Jan 22, 2016

It seems like this is a regression of 9.0.4, as it works with 1.7.21:

jruby 1.7.21 (1.9.3p551) 2015-07-07 a741a82 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
=> 0
irb(main):005:0> f.tell
=> 2147483648
irb(main):006:0> f.seek(3*1024*1024*1024)
=> 0
irb(main):007:0> f.tell
=> 3221225472
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit

jruby 9.0.4.0 (2.2.2) 2015-11-12 b9fb7aa Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [linux-amd64]

irb(main):001:0> f = File.open("/tmp/test.raw", "rb")
=> #<File:/tmp/test.raw>
irb(main):002:0> f.size
=> 4294967296
irb(main):003:0> f.tell
=> 0
irb(main):004:0> f.seek(2*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):4:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):005:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):5:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):006:0> f.seek(3*1024*1024*1024)
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1648:in `seek'
  from (irb):6:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):007:0> f.tell
Errno::ENOENT: No such file or directory - /tmp/test.raw
  from org/jruby/RubyIO.java:1534:in `pos'
  from (irb):7:in `<eval>'
  from org/jruby/RubyKernel.java:978:in `eval'
  from org/jruby/RubyKernel.java:1291:in `loop'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from org/jruby/RubyKernel.java:1098:in `catch'
  from /home/gucki/.rbenv/versions/jruby-9.0.4/bin/irb:13:in `<top>'
irb(main):008:0> f.close
=> nil
irb(main):009:0> exit
@gucki

This comment has been minimized.

Show comment
Hide comment
@gucki

gucki Jan 22, 2016

@headius Would be great to have this fixed in 9.0.5 👍

gucki commented Jan 22, 2016

@headius Would be great to have this fixed in 9.0.5 👍

@gucki

This comment has been minimized.

Show comment
Hide comment
@gucki

gucki Mar 4, 2016

Is there anything we could do to get this fixed, liking sponsoring the fix?

gucki commented Mar 4, 2016

Is there anything we could do to get this fixed, liking sponsoring the fix?

@enebo enebo added this to the JRuby 9.1.0.0 milestone Mar 7, 2016

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Mar 7, 2016

Member

This almost sounds like we switch IO pos from a long to an int? Marking for 9.1.0.0.

Member

enebo commented Mar 7, 2016

This almost sounds like we switch IO pos from a long to an int? Marking for 9.1.0.0.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Apr 20, 2016

Member

JRuby 9.1 is using a long for position in seek, but still failing. Looking into it.

Member

headius commented Apr 20, 2016

JRuby 9.1 is using a long for position in seek, but still failing. Looking into it.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Apr 20, 2016

Member

Ok, so there's one small bug and one big bug here.

The small bug is that we were expecting all negative return values from tell and lseek to mean there was an error. This is incorrect; our IO#seek actually works fine, but the return value is the new position, and because our native lseek binding only has a 32-bit signed return value, this comes out negative.

The big bug is that our native lseek binding and many paths leading to it all use 32-bit int for the return value.

I'm fixing the first one, which makes seek and tell at least function properly at the native IO level:

[] ~/projects/jruby $ jruby -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
-2147483648

[] ~/projects/jruby $ ruby23 -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
2147483648

Note the second larger problem rears its head here with IO#tell being unable to return file positions larger than a signed 32-bit integer.

I'm going to mark this fixed, since at least the error is gone now and seek is doing what it should under the covers. I'll open a new bug for the larger issue.

Member

headius commented Apr 20, 2016

Ok, so there's one small bug and one big bug here.

The small bug is that we were expecting all negative return values from tell and lseek to mean there was an error. This is incorrect; our IO#seek actually works fine, but the return value is the new position, and because our native lseek binding only has a 32-bit signed return value, this comes out negative.

The big bug is that our native lseek binding and many paths leading to it all use 32-bit int for the return value.

I'm fixing the first one, which makes seek and tell at least function properly at the native IO level:

[] ~/projects/jruby $ jruby -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
-2147483648

[] ~/projects/jruby $ ruby23 -e "f = File.open('/tmp/test.raw', 'rb'); p f.seek(2*1024*1024*1024); p f.tell"
0
2147483648

Note the second larger problem rears its head here with IO#tell being unable to return file positions larger than a signed 32-bit integer.

I'm going to mark this fixed, since at least the error is gone now and seek is doing what it should under the covers. I'll open a new bug for the larger issue.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Apr 20, 2016

Member

See #3817 for the larger issue.

Member

headius commented Apr 20, 2016

See #3817 for the larger issue.

@gucki

This comment has been minimized.

Show comment
Hide comment
@gucki

gucki Apr 20, 2016

@headius Thanks! 👍

gucki commented Apr 20, 2016

@headius Thanks! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment