Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vagrant-spk enter-grain: Value too large for defined data type #213

Closed
JamborJan opened this issue Mar 12, 2018 · 16 comments · Fixed by #246
Closed

vagrant-spk enter-grain: Value too large for defined data type #213

JamborJan opened this issue Mar 12, 2018 · 16 comments · Fixed by #246

Comments

@JamborJan
Copy link
Contributor

@JamborJan JamborJan commented Mar 12, 2018

I have pulled the latest release of vagrant-spk and also checked that my vagrant config file is up to date. When using vagrant-spk enter-grain I get the following error. I'm working on the Worpress on Sandstorm app.

jj$ vagrant-spk enter-grain
Looking for apps in dev mode...
Looking for grains...

This will run a shell in the context of a grain. Here is a list of running
grains you can attach to. Press enter to choose the first one, or type the
number next to the grain ID to choose it.

1. wB7zurWomYfd5XursS3XtS

Your choice: [1] 
OK. Chosen grain ID wB7zurWomYfd5XursS3XtS, whose child PID is 26100. Attaching...
Adding enter_grain program to the grain in /tmp/vagrant-spk/enter-grain...
mkdir: cannot create directory ‘/proc/26100/root/tmp/vagrant-spk’: Value too large for defined data type
Traceback (most recent call last):
  File "/usr/local/bin/vagrant-spk", line 984, in <module>
    main()
  File "/usr/local/bin/vagrant-spk", line 981, in main
    operation(args)
  File "/usr/local/bin/vagrant-spk", line 814, in shell
    enter_grain_binary_path,
  File "/usr/local/bin/vagrant-spk", line 715, in inject_enter_grain_into_grain
    stdin=enter_grain_binary)
  File "/usr/local/bin/vagrant-spk", line 293, in call_vagrant_ssh_command_providing_stdin
    vagrant_ssh_command_string)
ValueError: ('Yikes, the command crashed. Command was: %s', 'sudo mkdir -p /proc/26100/root/tmp/vagrant-spk && if which sha1sum >/dev/null && sudo sha1sum /proc/26100/root/tmp/vagrant-spk/enter-grain 2>/dev/null | grep -q ^11198904136a7e8079d41ef50af613d21786b5b8 ; then sudo chmod 755 /proc/26100/root/tmp/vagrant-spk/enter-grain && exit 0 ; fi && sudo dd of=/proc/26100/root/tmp/vagrant-spk/enter-grain 2>/dev/null && sudo chmod 755 /proc/26100/root/tmp/vagrant-spk/enter-grain')
@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented Mar 12, 2018

This involves way too much magic for me to even guess what's happening here. Asheesh is likely the only person on the planet who understands how enter-grain works. :/ And that is one heck of a command that crashed it.

@JamborJan

This comment has been minimized.

Copy link
Contributor Author

@JamborJan JamborJan commented Jun 25, 2018

Hey @ocdtrekkie, did you hear anything from the core / old Sandstorm guys about that issue? It would be great to have the enter grain tool back for debugging apps.

@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented Jun 25, 2018

@JamborJan While the vagrant-spk devs are aware of current progress, I haven't actually discussed any of the work with them. (The only actually discussed topic was the copyright PR.)

I haven't actually looked at enter-grain at all, but I could probably at least take a gander. I kinda wonder if it's possible to break up that command a bit or test individual pieces of it to try and isolate what's going on.

@paulproteus

This comment has been minimized.

Copy link
Collaborator

@paulproteus paulproteus commented Jul 1, 2018

Hey all - I do remember a bunch of how enter-grain works.

Having said that: @kentonv , do you recognize this error message at all?

mkdir: cannot create directory ‘/proc/26100/root/tmp/vagrant-spk’: Value too large for defined data type

I wonder if that comes, indirectly, from the FUSE filesystem...?

@paulproteus

This comment has been minimized.

Copy link
Collaborator

@paulproteus paulproteus commented Jul 1, 2018

For what it's worth:

I wrote enter-grain because a suitably-new version of nsenter wasn't widely-available within vagrant-spk VMs at the time. I now wonder if vagrant-spk ssh plus liberal use of nsenter would be more reliable. If so, that'd be excellent.

You can see a process ID in the above printout. You could try:

nsenter --target $PID --mount --uts --ipc --net --pid

where you replace $PID with the process ID printed above. @JamborJan , let me know if you get a chance to try that, and how well it works for you.

@JamborJan

This comment has been minimized.

Copy link
Contributor Author

@JamborJan JamborJan commented Jul 2, 2018

Hey @paulproteus ,

Thanks for your help. It somehow works even if it is not straight forward:

  • run app in spk dev
  • create or open a grain
  • try spk enter-grain
  • select grain id
  • wait for the error message + remember pid
  • ssh into the running vm
  • run nsenter

I can then navigate around but as soon as I use tab to autocomplete paths and filenames I get:

cannot create temp file for here-document: Value too large for defined data type

Does that help somehow? For now it's kind of okay to use this workaround. I know that there is currently nobody available but it would be cool if this feature could be properly wrapped in a spk command again.

@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented Jul 2, 2018

@JamborJan If we have a working command, it isn't shockingly hard to wrap things in vagrant-spk commands. See my recent #236. The same code which grabs the PID for enter-grain can be used to assemble nsenter. But my guess is if you're hitting that same error message when using nsenter, there's some issue that affects both nsenter and enter-grain that needs to be fixed.

@rainbowjaw

This comment has been minimized.

Copy link

@rainbowjaw rainbowjaw commented May 26, 2019

https://groups.google.com/d/msg/sandstorm-dev/GslQeMhNGSg/91I1dkcTBQAJ

A app developer expressed how big of a stumbling block this error and errors like it are to sandstorm.

@paulproteus

This comment has been minimized.

Copy link
Collaborator

@paulproteus paulproteus commented May 28, 2019

Hi @rainbowjaw ,

I think the problem relates to GoogleContainerTools/distroless#225 . Perhaps it's true that we're using a 32-bit userland in the Debian VM, but that the filesystem uses 64-bit inode numbers. (In my opinion, this is a really weird possibility.)

I'd be happy to remote-pair-program on this for an hour or two with someone if they want to try to take a look themselves and have me looking over their shoulder.

Email me if you want that: asheesh@asheesh.org

Best wishes!

Asheesh.

@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented May 28, 2019

I know @JamborJan is out of the Sandstorm market at the moment. But perhaps @kpreid would be interested in this offer!

@kpreid

This comment has been minimized.

Copy link

@kpreid kpreid commented May 28, 2019

Sorry, I have no experience with Linux container programming and I don't think I could efficiently get up to speed on it (and I have too many projects already). Hence my suggesting that someone else ought to do it! I'd be interested in contributing towards a bounty though.

@abliss

This comment has been minimized.

Copy link
Contributor

@abliss abliss commented Jan 22, 2020

I just hit this today. I think I can fix it.

@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented Jan 22, 2020

@abliss That would be really cool. Especially since I am going to try to wrap up a new vagrant-spk release soon.

@ocdtrekkie ocdtrekkie added this to the vagrant-spk 1.0 milestone Jan 22, 2020
@abliss

This comment has been minimized.

Copy link
Contributor

@abliss abliss commented Jan 23, 2020

abliss added a commit to abliss/vagrant-spk that referenced this issue Jan 23, 2020
Housing it in /tmp, which is a tmpfs, runs afoul of this kernel bug:
https://bugzilla.kernel.org/show_bug.cgi?id=183461

Fixes sandstorm-io#213 .
@abliss

This comment has been minimized.

Copy link
Contributor

@abliss abliss commented Jan 23, 2020

NB enter-grain still has other issues, likely also related to the unmapped uid. e.g. /proc is not mounted, so ps doesn't work, and I have no access to /var. So it may not be super useful as a debugging tool. Rewriting it to use a modern nsenter (and to move off python2.7) still sounds like a good idea to me.

@ocdtrekkie

This comment has been minimized.

Copy link
Collaborator

@ocdtrekkie ocdtrekkie commented Jan 23, 2020

@abliss, I had marked this issue as part of my 1.0 milestone since you seemed optimistic you could fix it. I want to do this next release soon, so I don't want to feature creep it, but it may be worthwhile to chat about what the next version after 1.0 should look like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants
You can’t perform that action at this time.