vagrant-spk enter-grain: Value too large for defined data type

[JamborJan]

I have pulled the latest release of vagrant-spk and also checked that my vagrant config file is up to date. When using vagrant-spk enter-grain I get the following error. I'm working on the Worpress on Sandstorm app.

jj$ vagrant-spk enter-grain
Looking for apps in dev mode...
Looking for grains...

This will run a shell in the context of a grain. Here is a list of running
grains you can attach to. Press enter to choose the first one, or type the
number next to the grain ID to choose it.

1. wB7zurWomYfd5XursS3XtS

Your choice: [1] 
OK. Chosen grain ID wB7zurWomYfd5XursS3XtS, whose child PID is 26100. Attaching...
Adding enter_grain program to the grain in /tmp/vagrant-spk/enter-grain...
mkdir: cannot create directory ‘/proc/26100/root/tmp/vagrant-spk’: Value too large for defined data type
Traceback (most recent call last):
  File "/usr/local/bin/vagrant-spk", line 984, in <module>
    main()
  File "/usr/local/bin/vagrant-spk", line 981, in main
    operation(args)
  File "/usr/local/bin/vagrant-spk", line 814, in shell
    enter_grain_binary_path,
  File "/usr/local/bin/vagrant-spk", line 715, in inject_enter_grain_into_grain
    stdin=enter_grain_binary)
  File "/usr/local/bin/vagrant-spk", line 293, in call_vagrant_ssh_command_providing_stdin
    vagrant_ssh_command_string)
ValueError: ('Yikes, the command crashed. Command was: %s', 'sudo mkdir -p /proc/26100/root/tmp/vagrant-spk && if which sha1sum >/dev/null && sudo sha1sum /proc/26100/root/tmp/vagrant-spk/enter-grain 2>/dev/null | grep -q ^11198904136a7e8079d41ef50af613d21786b5b8 ; then sudo chmod 755 /proc/26100/root/tmp/vagrant-spk/enter-grain && exit 0 ; fi && sudo dd of=/proc/26100/root/tmp/vagrant-spk/enter-grain 2>/dev/null && sudo chmod 755 /proc/26100/root/tmp/vagrant-spk/enter-grain')

[ocdtrekkie]

This involves way too much magic for me to even guess what's happening here. Asheesh is likely the only person on the planet who understands how enter-grain works. :/ And that is one heck of a command that crashed it.

[JamborJan]

Hey @ocdtrekkie, did you hear anything from the core / old Sandstorm guys about that issue? It would be great to have the enter grain tool back for debugging apps.

[ocdtrekkie]

@JamborJan While the vagrant-spk devs are aware of current progress, I haven't actually discussed any of the work with them. (The only actually discussed topic was the copyright PR.)

I haven't actually looked at enter-grain at all, but I could probably at least take a gander. I kinda wonder if it's possible to break up that command a bit or test individual pieces of it to try and isolate what's going on.

[paulproteus]

Hey all - I do remember a bunch of how enter-grain works.

Having said that: @kentonv , do you recognize this error message at all?

mkdir: cannot create directory ‘/proc/26100/root/tmp/vagrant-spk’: Value too large for defined data type

I wonder if that comes, indirectly, from the FUSE filesystem...?

[paulproteus]

For what it's worth:

I wrote enter-grain because a suitably-new version of nsenter wasn't widely-available within vagrant-spk VMs at the time. I now wonder if vagrant-spk ssh plus liberal use of nsenter would be more reliable. If so, that'd be excellent.

You can see a process ID in the above printout. You could try:

nsenter --target $PID --mount --uts --ipc --net --pid

where you replace $PID with the process ID printed above. @JamborJan , let me know if you get a chance to try that, and how well it works for you.

[JamborJan]

Hey @paulproteus ,

Thanks for your help. It somehow works even if it is not straight forward:

• run app in spk dev
• create or open a grain
• try spk enter-grain
• select grain id
• wait for the error message + remember pid
• ssh into the running vm
• run nsenter

I can then navigate around but as soon as I use tab to autocomplete paths and filenames I get:

cannot create temp file for here-document: Value too large for defined data type

Does that help somehow? For now it's kind of okay to use this workaround. I know that there is currently nobody available but it would be cool if this feature could be properly wrapped in a spk command again.

[ocdtrekkie]

@JamborJan If we have a working command, it isn't shockingly hard to wrap things in vagrant-spk commands. See my recent #236. The same code which grabs the PID for enter-grain can be used to assemble nsenter. But my guess is if you're hitting that same error message when using nsenter, there's some issue that affects both nsenter and enter-grain that needs to be fixed.

[rainbowjaw]

https://groups.google.com/d/msg/sandstorm-dev/GslQeMhNGSg/91I1dkcTBQAJ

A app developer expressed how big of a stumbling block this error and errors like it are to sandstorm.

[paulproteus]

Hi @rainbowjaw ,

I think the problem relates to GoogleContainerTools/distroless#225 . Perhaps it's true that we're using a 32-bit userland in the Debian VM, but that the filesystem uses 64-bit inode numbers. (In my opinion, this is a really weird possibility.)

I'd be happy to remote-pair-program on this for an hour or two with someone if they want to try to take a look themselves and have me looking over their shoulder.

Email me if you want that: asheesh@asheesh.org

Best wishes!

Asheesh.

[ocdtrekkie]

I know @JamborJan is out of the Sandstorm market at the moment. But perhaps @kpreid would be interested in this offer!

[kpreid]

Sorry, I have no experience with Linux container programming and I don't think I could efficiently get up to speed on it (and I have too many projects already). Hence my suggesting that someone else ought to do it! I'd be interested in contributing towards a bounty though.

[abliss]

I just hit this today. I think I can fix it.

[ocdtrekkie]

@abliss That would be really cool. Especially since I am going to try to wrap up a new vagrant-spk release soon.

[abliss]

I suspect that GoogleContainerTools/distroless#225 may be a red herring: it seems to be a busybox bug https://bugs.busybox.net/show_bug.cgi?id=11651 which in turn seems to be a uclibc bug regarding 64-bit inodes. However, the sandstorm issue reproduces even with low inode numbers and even using a `mkdir` that is not using busybox or uclibc at all. I think we may be closer to https://bugzilla.kernel.org/show_bug.cgi?id=183461 , which is a kernel bug(?) about the mkdir() syscall specifically in the case of a tmpfs inside a user namespace when uid and gid are unmapped. That does appear to match our situation. A simple workaround is to use /var/tmp, instead of /tmp, to house vagrant-spk inside the vm. This seems to fix `enter-grain` for me.

[abliss]

NB enter-grain still has other issues, likely also related to the unmapped uid. e.g. /proc is not mounted, so ps doesn't work, and I have no access to /var. So it may not be super useful as a debugging tool. Rewriting it to use a modern nsenter (and to move off python2.7) still sounds like a good idea to me.

[ocdtrekkie]

@abliss, I had marked this issue as part of my 1.0 milestone since you seemed optimistic you could fix it. I want to do this next release soon, so I don't want to feature creep it, but it may be worthwhile to chat about what the next version after 1.0 should look like.