New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fails to load iseq cache in Linux #20

Closed
SamSaffron opened this Issue Apr 29, 2017 · 17 comments

Comments

Projects
None yet
5 participants
@SamSaffron
Copy link
Contributor

SamSaffron commented Apr 29, 2017

I have xfs + xattrs configured on my linux box, when booting I get:

/home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bootsnap-0.2.12/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Argument list too long (Errno::E2BIG)
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bootsnap-0.2.12/lib/bootsnap/compile_cache/iseq.rb:30:in `load_iseq'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bootsnap-0.2.12/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/bootsnap-0.2.12/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
	from /home/sam/.rbenv/versions/2.4.1/lib/ruby/2.4.0/psych/visitors.rb:3:in `<main>'
@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented Apr 30, 2017

xattr appear to be working OK here:

require 'ffi-xattr'
`touch a`
xattr = Xattr.new("a")
str = 'x'*16000 + 'y'*16000
xattr['user.foo'] = str
xattr = Xattr.new("a")
puts xattr['user.foo'].length
puts xattr['user.foo'] == str
32000
true

on ext4 you are limited on size but stuff seems unlimited on xfs

@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented Apr 30, 2017

I added some puts statements

The file it is failing on is:

/home/sam/.rbenv/versions/2.4.1/lib/ruby/2.4.0/psych/visitors/to_ruby.rb
@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented Apr 30, 2017

this can be cause #define XATTR_SIZE_MAX 65536 on linux... will check

... confirmed :( ...

I guess the simplest workaround would be to store multiple values for cases where size exceeds size max.

So for a 100K file

user.autocc.value == 64k of the data
user.autocc.value1 == rest of data

Then when you get a value you check if it is == to max size and if it is you try for value1 and so on.

@burke

This comment has been minimized.

Copy link
Member

burke commented May 1, 2017

Interesting. The maximum is rather a lot larger on darwin. Probably we should just transition away from xattr somehow.

@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented May 3, 2017

I was thinking, one way of saving a bit of storage is to run the iseq stream via snappy

@burke

This comment has been minimized.

Copy link
Member

burke commented May 3, 2017

Yeah, that's definitely true. It would help. The major issue though is that the limit is 64MB on darwin and 64kB on linux. Inflating that limit to an effective ~90kB is certainly going to help in some cases, but not enough.

The other complicating factor is that to decompress a string would require a roundtrip through ruby or to link libsnappy into the bootsnap bundle somehow.

@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented May 3, 2017

yeah a round trip to ruby would be out of the question here, but linking libsnappy should be straight forward I think, it compiles quite fast.

there is a trick though that can make this quite workable

If you read user.autocc.value and get XATTR_SIZE_MAX back you can proceed to read user.autocc.value1 and so on. 99% of the time it would fit into one xattr (with libsnappy) but the 1% case would be easily catered for by splitting.

You only write XATTR_SIZE_MAX in cases where you have to split across multiple attributes.

@burke

This comment has been minimized.

Copy link
Member

burke commented May 3, 2017

Yeah, actually, that seems like a very reasonable way to solve this.

@wjordan

This comment has been minimized.

Copy link
Contributor

wjordan commented May 3, 2017

The major issue though is that the limit is 64MB on darwin and 64kB on linux.

actually the xattr limit is 64kB on linux only for JFS, XFS, or Reiserfs filesystems. The limit is 16kB for Btrfs, and for ext2/3/4 the limit on all of a file's xattrs is only 4kB, so a user.autocc.value1 approach won't work.

See man xattr(7) for details:

   The VFS imposes limitations that an attribute names is limited to 255
   bytes and an attribute value is limited to 64 kB. [...]

   In the current ext2, ext3, and ext4 filesystem implementations, the
   total bytes used by the names and values of all of a file's extended
   attributes must fit in a single filesystem block (1024, 2048 or 4096
   bytes, depending on the block size specified when the filesystem was
   created).

   In the Btrfs, XFS, and Reiserfs filesystem implementations, there is
   no practical limit on the number of extended attributes associated
   with a file, and the algorithms used to store extended attribute
   information on disk are scalable.

   In the Btrfs filesystem implementation, the total bytes used for the
   name, value, and implementation overhead bytes is limited to the
   filesystem nodesize value (16 kB by default).
@SamSaffron

This comment has been minimized.

Copy link
Contributor

SamSaffron commented May 3, 2017

@burke

This comment has been minimized.

Copy link
Member

burke commented May 4, 2017

Yeah, the thing here is that I went down a bit of a rabbit hole with the xattr thing and it worked well enough in practice on our macOS dev machines that I haven't walked back out of it yet. It's probably not a great long-term feature simply because it doesn't behave nicely on linux, though the attribute it has of tying cache lifecycle to the source file is quite nice.

@jules2689

This comment has been minimized.

Copy link
Member

jules2689 commented May 4, 2017

Personally, I'd love to see some brainstorming on a way to transition from xattr to something that is supported by at least mac OS and Linux. I also know the Rails team has put effort into making everything work on Windows, so if it works for that too, that'd be grand.

Anyone have any ideas?

@burke

This comment has been minimized.

Copy link
Member

burke commented May 4, 2017

likely the right solution is the boring one: cache things at tmp/cache/bootsnap-compile-cache/A/B where B is a hash over the file path and A is the first 2 chars of B.

@jdrago999

This comment has been minimized.

Copy link

jdrago999 commented May 20, 2017

I just got the same error while running ubuntu 16.04 inside docker on macOS.

@burke

This comment has been minimized.

Copy link
Member

burke commented May 25, 2017

Should be fixed by #43

@burke

This comment has been minimized.

Copy link
Member

burke commented May 25, 2017

curious to know if 0.3.0.pre works for you.

@burke

This comment has been minimized.

Copy link
Member

burke commented May 26, 2017

0.3.0 released

@burke burke closed this May 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment