"failed to obtain exit status for remote process" timeout when mounting

[Saviq]

I just got this "failed to obtain exit status for remote process" error out of the blue. I had been building snaps for a few days. Can no longer build snaps. I am on the latest Ubuntu 18.04. Tried rebooting Ubuntu - no help.

$ multipass version
multipass  v0.5
multipassd v0.5

not sure why the versions are different to the multipass 2018.10.1 etc. numbering?

Anyway, the snap disable multipass / snap enable multipass trick above failed for me, because multipassd seemed to get hosed and I wasn't sure how to restart it:

$ snapcraft
Launching a VM.
launch failed: cannot connect to the multipass socket                           
Please ensure multipassd is running and '/run/multipass_socket' is accessible
An error occurred when trying to launch the instance with 'multipass': returned exit code 2.
Ensure that 'multipass' is setup correctly and try again.

$ snap enable multipassd
snap "multipassd" is not installed

So I

sudo snap remove multipass 
sudo snap install multipass --beta --classic

and things seemed to start working again. But then running snapcraft I got the same error I started with:

Launching a VM.
Launched: snapcraft-rubber-band-async                                           
mount failed: The following errors occurred:
error mounting "/var/cache/snapcraft/snaps": failed to obtain exit status for remote process 'id -u': timeout
An error occurred when trying to mount using 'multipass': returned exit code 2. 

also getting

mount failed: The following errors occurred:
error mounting "/var/cache/snapcraft/snaps": failed to obtain exit status for remote process 'which sshfs': timeout

P.S. Installing multipass from --edge led to snapcraft just locking up. So I'm sticking with beta.

Any ideas?

Originally posted by @abulka in #552 (comment)

[Saviq]

Hi @abulka, I created a new issue for you.

I'm afraid it's not immediately clear what's going on. It's as if the instance starts fine, but locks up some time after and no commands reach it any more.

Can you try launching an instance manually and see if it behaves fine?

multipass launch --name test
multipass shell test

If that works fine, try the snapcraft image:

multipass launch --name test-snapcraft snapcraft:core18
multipass shell test-snapcraft

Can you ping the instance IP, as shown by:

multipass list

FWIW you can restart snaps with snap restart multipass (the snap name, not the daemon).

Thanks, let us know how you fare. Come to #multipass on Freenode IRC if you can, for a more in-person experience :)

[abulka]

@Saviq I tried those commands you suggested - all work ok. Am able to shell in and ping. Yet my snapcraft build fails with failed to obtain exit status for remote process 'which sshfs': timeout or failed to obtain exit status for remote process 'id -u': timeout Here is my session:

$ snapcraft --shell-after
Launching a VM.
Skipping pull copy-stuff (already ran)                                          
Skipping pull desktop-gtk3 (already ran)
Skipping pull rubber-band-async (already ran)
Skipping build copy-stuff (already ran)
Skipping build desktop-gtk3 (already ran)
Copying needed target link from the system /lib/x86_64-linux-gnu/liblzma.so.5.2.2
Copying needed target link from the system /lib/x86_64-linux-gnu/libpcre.so.3.13.3
Cleaning later steps and re-building rubber-band-async ('override-build' property changed)
/root/parts/rubber-band-async/install/usr/bin/python3: No module named pip
Failed to run '/root/parts/rubber-band-async/install/usr/bin/python3 -m pip wheel --no-index --find-links /root/parts/rubber-band-async/python-packages --wheel-dir /tmp/tmpytyegx1n --requirement /root/project/requirements-linux.txt https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.4-cp36-cp36m-linux_x86_64.whl': Exited with code 1.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
An error occurred when trying to execute 'sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft snap' with 'multipass': returned exit code 2.

$ snapcraft --shell-after
Issues while validating snapcraft.yaml: The 'apps' property does not match the required schema: 'rubber-band-async' is not a valid app name. App names consist of upper- and lower-case alphanumeric characters and hyphens. They cannot start or end with a hyphen.
andy@ubuntu:~/Devel/pynsource/Research/async play$ snapcraft --shell-after
Launching a VM.
Skipping pull copy-stuff (already ran)                                          
Skipping pull desktop-gtk3 (already ran)
Skipping pull rubber-band-async (already ran)
Skipping build copy-stuff (already ran)
Skipping build desktop-gtk3 (already ran)
Building rubber-band-async 
/root/parts/rubber-band-async/install/usr/bin/python3: No module named pip
Failed to run '/root/parts/rubber-band-async/install/usr/bin/python3 -m pip wheel --no-index --find-links /root/parts/rubber-band-async/python-packages --wheel-dir /tmp/tmp25bqvk48 --requirement /root/project/requirements-linux.txt https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.4-cp36-cp36m-linux_x86_64.whl': Exited with code 1.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
An error occurred when trying to execute 'sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft snap' with 'multipass': returned exit code 2.

$ snapcraft --shell-after
Launching a VM.
Skipping pull copy-stuff (already ran)                                          
Skipping pull desktop-gtk3 (already ran)
Skipping pull rubber-band-async (already ran)
Skipping build copy-stuff (already ran)
Skipping build desktop-gtk3 (already ran)
Building rubber-band-async 
/root/parts/rubber-band-async/install/usr/bin/python3: No module named pip
Failed to run '/root/parts/rubber-band-async/install/usr/bin/python3 -m pip wheel --no-index --find-links /root/parts/rubber-band-async/python-packages --wheel-dir /tmp/tmpem483pn4 --requirement /root/project/requirements-linux.txt https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.4-cp36-cp36m-linux_x86_64.whl': Exited with code 1.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
An error occurred when trying to execute 'sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft snap' with 'multipass': returned exit code 2.

$ snapcraft --shell-after
Launching a VM.
Skipping pull copy-stuff (already ran)                                          
Skipping pull desktop-gtk3 (already ran)
Skipping pull rubber-band-async (already ran)
Skipping build copy-stuff (already ran)
Skipping build desktop-gtk3 (already ran)
Building rubber-band-async 
/root/parts/rubber-band-async/install/usr/bin/python3: No module named pip
Failed to run '/root/parts/rubber-band-async/install/usr/bin/python3 -m pip wheel --no-index --find-links /root/parts/rubber-band-async/python-packages --wheel-dir /tmp/tmph_pfk6oq --requirement /root/project/requirements-linux.txt https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.4-cp36-cp36m-linux_x86_64.whl': Exited with code 1.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.
An error occurred when trying to execute 'sudo -i env SNAPCRAFT_HAS_TTY=True snapcraft snap' with 'multipass': returned exit code 2.

$ snapcraft clean

$ snapcraft --shell-after
Launching a VM.
Launched: snapcraft-rubber-band-async                                           
mount failed: The following errors occurred:
error mounting "/var/cache/snapcraft/snaps": failed to obtain exit status for remote process 'which sshfs': timeout
An error occurred when trying to mount using 'multipass': returned exit code 2. 
andy@ubuntu:~/Devel/pynsource/Research/async play$ snapcraft --shell-after
Launching a VM.
core 16-2.37.4 from Canonical✓ installed
snapcraft 3.2 from Canonical✓ installed
core18 18 from Canonical✓ installed
mount failed: The following errors occurred:
error mounting "/root/project": failed to obtain exit status for remote process 'which sshfs': timeout
An error occurred when trying to mount using 'multipass': returned exit code 2.

$ multipass list
Name                    State             IPv4             Release
test                    RUNNING           10.90.89.227     Ubuntu 18.04 LTS
test-snapcraft          RUNNING           10.90.89.32      Ubuntu Snapcraft builder for Core 18
snapcraft-rubber-band-asyncRUNNING           10.90.89.113     Ubuntu Snapcraft builder for Core 18

$ multipass list
Name                    State             IPv4             Release
test                    RUNNING           10.90.89.227     Ubuntu 18.04 LTS
test-snapcraft          RUNNING           10.90.89.32      Ubuntu Snapcraft builder for Core 18
snapcraft-rubber-band-asyncRUNNING           10.90.89.113     Ubuntu Snapcraft builder for Core 18

$ snapcraft --shell-after
Launching a VM.
mount failed: The following errors occurred:                                    
error mounting "/root/project": failed to obtain exit status for remote process 'id -u': timeout
An error occurred when trying to mount using 'multipass': returned exit code 2. 
$ 

Side Note: This is straight after doing a snapcraft clean which I am forced to do because of the pip not found bug or must give at least one requirement bug.

Further, the fact that snapcraft -s is broken see this issue which Evan has confirmed is a bug, for core18 based snaps, which ironically are the recommended snaps!, means a full snapcraft clean is needed, which means waiting for 2 hours whilst my VM re-downloads everything and builds everything. Then I have one shot to make a build otherwise its back to a clean and 2 hour wait again. Very frustrating.

But then I hit this multipass bug, which means I'm stuck for half a day till it magically clears up by itself (last time it did eventually). Update: I stopped all multipass VMs, did the snap restart multipass and snapcraft is now running ok.

[Saviq]

@abulka sorry, you seem to have won the lottery for bugs between Multipass and snapcraft… Those multipass timeouts suggest the VMs were too slow to respond... Are there many running at the same time? What was your host's load at the time?

IIUC you managed to get going in the end?

[abulka]

@Saviq yes, I managed to get going in the end - thanks. The snap restart multipass trick was the key. Though you would think a reboot of my VM would have fixed it too, but didn't. Perhaps the snap restart multipass does something extra. Not sure if I have to stop all the multipass VMs before issuing that command, but I did anyway. Anyway, this problem occurs now and then - would be good if it could be tracked down and fixed. The Ubuntu 18.04 VM is the only VM running on a powerful iMac, with nothing going on - definitely not a CPU issue.

As for the lottery for bugs - yes its been a minefield! Add to that the fact that once I build a snap on Ubuntu 18.04 it won't even install via GUI on 16.04 due to this issue and has to be installed on 16.04 via command line instead. And even then, that doesn't work unless you also manually install sudo snap install core because core doesn't install automatically (IMO it should). So yeah, I've been pretty lucky! ;-)

[Saviq]

@abulka re: restarting, that would suspend and resume the instances, so it might indeed not be enough.

We'll be releasing a new version soon with a bunch of changes, hopefully it will improve your situation.

Are you running any firewall (-like) software? A VPN? Little Snitch? They may play tricks on the VMs' networking, too. EDIT the iMac part tricked me, you're using the snap, so that doesn't apply.

[abulka]

No firewall - just plain Ubuntu out of the box install. Mac host has nothing special, no little snitch. No VPN anywhere. Today I finally released my snap - after a week of struggles! https://snapcraft.io/pynsource :-)

…

On Wed, 20 Mar 2019 at 18:41, Michał Sawicz ***@***.***> wrote: @abulka <https://github.com/abulka> re: restarting, that would suspend and resume the instances, so it might indeed not be enough. We'll be releasing a new version soon with a bunch of changes, hopefully it will improve your situation. Are you running any firewall (-like) software? A VPN? Little Snitch? They may play tricks on the VMs' networking, too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#683 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK77CuI7dQd222Pr2PkjBgeJbv-rZjEUks5vYeY-gaJpZM4b5OnT> .

[kronos-cm]

Hi there, I had a similar problem, what solved it was to follow the steps in the Multipass subsection from the microk8s documentation:
To work within the VM environment more easily, you can run a shell:

multipass shell microk8s-vm
Then install the MicroK8s snap and configure the network:

sudo snap install microk8s --classic --channel=1.18/stable
sudo iptables -P FORWARD ACCEPT

[thomergil]

This problem was caused for me by running a VPN on the host. Once I stopped the VPN on the host, this problem went away.