New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unix domain sockets return spurious data from getsockopt(SO_PEERCRED) if the peer has already departed, which breaks ssh-agent forwarding for Ubuntu 18.04 #3183

Closed
0xabu opened this Issue May 11, 2018 · 29 comments

Comments

Projects
None yet
@0xabu
Copy link

0xabu commented May 11, 2018

Since upgrading from Ubuntu 16.04 to 18.04, I've noticed that my ssh-agent setup no longer works reliably. For the purposes of debugging, I ran ssh-agent as:

/usr/bin/ssh-agent -s -d -a /tmp/.sshagent.sock

After doing this, I can add keys to the agent, and make exactly one SSH connection. Later, trying to connect to the agent fails with Error connecting to agent: Connection refused. Before the SSH connection (when everything still works), getsockopt appears to behave as expected, returning the pid/uid/gid of the process connecting to the socket:

accept(3</unknown>, {sa_family=AF_UNIX}, [110->2]) = 4</unknown>
getsockopt(4</unknown>, SOL_SOCKET, SO_PEERCRED, {pid=369, uid=1000, gid=1000}, [12]) = 0
getuid()                                = 1000

However later, getsockopt returns -1 as uid/gid which obviously makes ssh-agent unhappy:

accept(3</unknown>, {sa_family=AF_UNIX}, [110->2]) = 4</unknown>
getsockopt(4</unknown>, SOL_SOCKET, SO_PEERCRED, {pid=0, uid=-1, gid=-1}, [12]) = 0
getuid()                                = 1000
getuid()                                = 1000
write(2</dev/pts/1>, "uid mismatch: peer euid 42949672"..., 48) = 48
close(4</unknown>)                      = 0
close(3</unknown>)                      = 0

I'm on RS4 build 17134. /tmp is on an lxfs mountpoint.

@DarthSpock

This comment has been minimized.

Copy link

DarthSpock commented May 11, 2018

How did you upgrade? do-release-upgrade or installing the new 18.04 app?

@0xabu

This comment has been minimized.

Copy link

0xabu commented May 11, 2018

@DarthSpock, do-release-upgrade -d

@0xabu

This comment has been minimized.

Copy link

0xabu commented May 11, 2018

Some more diagnosis: this bug is caused by initiating an SSH connection that uses agent connection forwarding (ssh -O ForwardAgent=yes). When that connects, it does a test connection to the auth sock:

connect(9</unknown>, {sa_family=AF_UNIX, sun_path="/tmp/.sshagent.sock"}, 110) = 0
close(9</unknown>)                      = 0

On the SSH agent side, this one connect/close pair causes the bogus data returned from getsockopt SO_PEERCRED (probably because the peer is already gone by the time of the getsockopt call):

accept(3</unknown>, {sa_family=AF_UNIX}, [110->2]) = 5</unknown>
getsockopt(5</unknown>, SOL_SOCKET, SO_PEERCRED, {pid=0, uid=-1, gid=-1}, [12]) = 0
getuid()                                = 1000
getuid()                                = 1000
write(2</dev/pts/1>, "uid mismatch: peer euid 42949672"..., 48) = 48
close(5</unknown>)                      = 0
close(3</unknown>)                      = 0

… and after that time the ssh-agent remains alive but is unusable because it closed its end of the socket. This is the only thing that differs between Ubuntu 16.04 and 18.04 -- in earlier versions, the agent would log the error message and drop that specific connection but keep its listening socket open; now (probably for security paranoia) it becomes unusable after one failed connection.

So there's a simple workaround for now: don't use agent forwarding.

@0xabu 0xabu changed the title Unix domain sockets sometimes return spurious data from getsockopt(SO_PEERCRED), breaks ssh-agent for Ubuntu 18.04 Unix domain sockets return spurious data from getsockopt(SO_PEERCRED) if the peer has already departed, which breaks ssh-agent forwarding for Ubuntu 18.04 May 11, 2018

@mdkeehan

This comment has been minimized.

Copy link

mdkeehan commented May 12, 2018

I have noticed my ssh-agent exhibiting similar behaviour after upgrading to 18.04 using
do-release-upgrade -d

@rarenerd

This comment has been minimized.

Copy link

rarenerd commented May 15, 2018

I can confirm having the same issue, using Ubuntu 18.04 from the Microsoft Store on a new Windows 10 installation.

@rarenerd

This comment has been minimized.

Copy link

rarenerd commented May 15, 2018

My config is identical to the example in your link. This does not resolve the issue for me.

@0xabu

This comment has been minimized.

Copy link

0xabu commented May 15, 2018

@DarthSpock, I already have the metadata flag on my drvfs mounts, but I don't think it's relevant here anyway because the ssh-agent socket is on a different lxfs mountpoint.

@0xabu

This comment has been minimized.

Copy link

0xabu commented May 15, 2018

@DarthSpock the issue is the new socket close call in the error-handling path of the ssh-agent binary. Why would that be any different from a clean install? I'm pretty sure this is just another minor compatibility bug in the implementation of getsockopt.

@rarenerd

This comment has been minimized.

Copy link

rarenerd commented May 15, 2018

@DarthSpock I'm doing this on a clean Windows 10 install with a clean Ubuntu 18.04 install. I've since removed the 18.04 install and installed 16.04 where I don't have this issue.

@0xabu

This comment has been minimized.

Copy link

0xabu commented May 15, 2018

It's a bug in WSL.

@rarenerd

This comment has been minimized.

Copy link

rarenerd commented Jun 5, 2018

Is there anything I can do to help progress this ticket? Anything needed to reproduce the issue?

@DarthSpock

This comment has been minimized.

Copy link

DarthSpock commented Jun 5, 2018

@rarenerd By following CONTRIBUTING.md

@d-rez

This comment has been minimized.

Copy link

d-rez commented Jun 26, 2018

I can confirm a clean installation of Ubuntu 18.04 on 1803 build of Windows 10 results in this problem (WSL was installed after updating to 1803)

@Brian-Perkins

This comment has been minimized.

Copy link
Collaborator

Brian-Perkins commented Jun 28, 2018

There were changes in Insider Build 17704 that may address this.

@alexwh

This comment has been minimized.

Copy link

alexwh commented Jul 7, 2018

I can confirm that SSH agent forwarding works on 17711.

@thegass

This comment has been minimized.

Copy link

thegass commented Jul 21, 2018

will this get fixed for non insider-builds? or do we have to wait for the fall-update of Windows 10?
wsl is kind of useless without working ssh-agent.

@d-rez

This comment has been minimized.

Copy link

d-rez commented Jul 21, 2018

@thegass it's not useless, just install Ubuntu 16 instead of 18 from the windows store, it works fine

@thegass

This comment has been minimized.

Copy link

thegass commented Jul 21, 2018

@d-rez will try the old version. (tried already opensuse but that has the same problem)

@CoolCold

This comment has been minimized.

Copy link

CoolCold commented Jul 29, 2018

* ssh(1): Add a ProxyJump option and corresponding -J command-line
   flag to allow simplified indirection through a one or more SSH
   bastions or "jump hosts".

was introduced in version 7.3 ( http://www.openssh.com/txt/release-7.3 ) which is not the case for ubuntu 16.04:

Version: 1:7.2p2-4ubuntu2.2
coolcold@x230-coolcold:~$ dpkg -s openssh-client|egrep ^Version:
Version: 1:7.2p2-4ubuntu2.2

and is limiting factor for me, for example.

@rcarmo

This comment has been minimized.

Copy link

rcarmo commented Aug 7, 2018

Any idea if this is getting rolled up into a prod update soon? I can't revert to 16.04 solely for the sake of SSH (because I need some of the updated userland), and yet I really need SSH forwarding to work, otherwise I can't checkout repositories on remote machines with my own key.

@benhillis

This comment has been minimized.

Copy link
Member

benhillis commented Aug 7, 2018

@rcarmo - We typically only service security and reliability issues, so this won't be available in a non-Insiders build until the next update due this fall.

@shoffmeister

This comment has been minimized.

Copy link

shoffmeister commented Aug 7, 2018

Could you run multiple distributions in parallel (16.04 for git clone et al; 18.04 for a current userland) and use the Windows host as the shared storage provider (/mnt/c/git-repose-here)?

@0xabu

This comment has been minimized.

Copy link

0xabu commented Aug 7, 2018

@rcarmo have you tried grabbing the ssh-agent binary from 16.04 and using that as a workaround?

@rcarmo

This comment has been minimized.

Copy link

rcarmo commented Aug 7, 2018

It's not practical to maintain two copies of Ansible, Terraform, az, my custom pyenvs and $DIVINITY knows what else and switch over to another distro "just" for SSH. I just reinstalled everything today after realizing 18.04 had new GCC and "good enough" Go, and having a critical function like agent forwarding conk out on me is a major issue (I pretty much live inside the terminal other than a browser and corp e-mail).

@rcarmo

This comment has been minimized.

Copy link

rcarmo commented Aug 7, 2018

@0xabu from the thread above I was under the impression this was also broken in other distros, but yes, that worked, thanks!

Not that I'm happy about it, but it is a semi-acceptable workaround for WSL. Which should probably have off-cycle updates (hint hint).

Here's a quick HOWTO for a fix (and for the non-squeamish):

If you're reading this after August 2018, go here to check you're getting the latest version from xenial-updates :

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.4_amd64.deb
dpkg -x openssh-client_7.2p2-4ubuntu2.4_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent

(Edits for readability.)

@apsdsm

This comment has been minimized.

Copy link

apsdsm commented Aug 19, 2018

@rcarmo this worked for me, thanks!

@masoo

This comment has been minimized.

Copy link

masoo commented Oct 4, 2018

#3183 (comment)
Even if you use the method
In my environment Windows 10 18.09, "ssh-agent" has stopped working properly.
How is everyone?

#3183 (comment)
Thank you.
It worked fine.

@jstangroome

This comment has been minimized.

Copy link

jstangroome commented Oct 4, 2018

I can confirm that Windows 10 October 2018 Update Version 1809 (OS Build 17763.1) works fine with Ubuntu Bionic's ssh-agent from the openssh-client apt package version 1:7.6p1-4.

https://blogs.windows.com/windowsexperience/2018/10/02/how-to-get-the-windows-10-october-2018-update/

@trueromio

This comment has been minimized.

Copy link

trueromio commented Dec 12, 2018

Just posting @rcarmo workaround with fixed package url

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.6_amd64.deb
dpkg -x openssh-client_7.2p2-4ubuntu2.6_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment