feat: add Unix forwarding server implementations

[deansheather]

Adds optional (disabled by default) implementations of local->remote and remote->local Unix forwarding through OpenSSH's protocol extensions:

• streamlocal-forward@openssh.com - cancel-streamlocal-forward@openssh.com - forwarded-streamlocal@openssh.com
• direct-streamlocal@openssh.com

Adds tests for Unix forwarding, reverse Unix forwarding and reverse TCP forwarding.

I recently had to write this code for my job so we could support GPG forwarding and couldn't find an implementation here so I thought I'd contribute it upstream so others can use it easily.

[gustavosbarreto]

@henrybarreto, could you take a look at this to see if it makes sense to support it in @shellhub-io?

[samchouse]

I'd love to see this PR merged in so that Tailscale can support it on their end. Since they want to start moving back to the original copy of this repo, this PR is now blocking support for stuff like GPG agent forwarding through Tailscale SSH. See tailscale/tailscale#12081 (comment).

While I was working on my PR over at Tailscale, I made a couple changes to streamlocal.go for everything to work properly (mainly the file permissions and using the unlink function from https://github.com/coder/coder/blob/main/agent/agentssh/forward.go which I believe their license permits). I'm not sure if these changes would also benefit this repo or if it's specific to Tailscale.

@@ -128,12 +127,27 @@ func (h *ForwardedUnixHandler) HandleSSHRequest(ctx Context, srv *Server, req *g
                        return false, nil
                }
 
+               // https://github.com/coder/coder/blob/main/agent/agentssh/forward.go
+               // Remove existing socket if it exists. We do not use os.Remove() here
+               // so that directories are kept. Note that it's possible that we will
+               // overwrite a regular file here. Both of these behaviors match OpenSSH,
+               // however, which is why we unlink.
+               err = unlink(addr)
+               if err != nil && !errors.Is(err, fs.ErrNotExist) {
+                       // TODO: log
+                       return false, nil
+               }
+
                ln, err := net.Listen("unix", addr)
                if err != nil {
                        // TODO: log unix listen failure
                        return false, nil
                }
 
+               if err := os.Chmod(addr, os.FileMode(0777)); err != nil {
+                       // TODO: log permission change failure
+                       return false, nil
+               }
+
                // The listener needs to successfully start before it can be added to
                // the map, so we don't have to worry about checking for an existing
                // listener as you can't listen on the same socket twice.
@@ -202,3 +216,15 @@ func (h *ForwardedUnixHandler) HandleSSHRequest(ctx Context, srv *Server, req *g
                return false, nil
        }
 }
+
+// https://github.com/coder/coder/blob/main/agent/agentssh/forward.go
+// unlink removes files and unlike os.Remove, directories are kept.
+func unlink(path string) error {
+       // Ignore EINTR like os.Remove, see ignoringEINTR in os/file_posix.go
+       // for more details.
+       for {
+               err := syscall.Unlink(path)
+               if !errors.Is(err, syscall.EINTR) {
+                       return err
+               }
+       }
+}

[gustavosbarreto]

Before we can merge, we'll need to resolve the conflict. Could you please take a look at it?

[deansheather]

@Xenfo I applied your patch minus the chmod thing. I don't think it's a good default to chmod to 777.

[samchouse]

Yeah I had also brought that concern up in the Tailscale PR. I'm not really sure what could be done better here since I'm not too familiar with the library or proper permissions. Ideally they would be set so that user that SSHed is able to use it.

If there isn't a better default permission, it's essential that there's some way to set the permissions (maybe through a separate function that can be called by the library user?) for Tailscale since it runs as root. Any user not SSHing as root would not be able to access the sockets that are forwarded.

[gustavosbarreto]

In my opinion, this responsibility should lie with the user of the ReverseUnixForwardingCallback. The callback should receive the socket information and return a net.Listener because that is what matters. All the filesystem business logic should be handled by the user. This way, the library remains completely agnostic about how applications handle this type of forwarding.

[deansheather]

I'll make some changes later today.

[samchouse]

Hey @deansheather if you need any help I'm willing to give this a go.

@gustavosbarreto is this what you're suggesting?

ReverseUnixForwardingCallback: ssh.ReverseUnixForwardingCallback(func(ctx Context, socketPath string) net.Listener {
	ln, err := net.Listen("unix", addr)
	if err != nil {
		return nil
	}

	if err := os.Chmod(addr, os.FileMode(0777)); err != nil {
		return nil
	}

	return ln
}),

If so then how would the check for allowing reverse forwarding be done? If we check if the callback returns nil, we can only check after the unlink call since net.Listener can't be created before and we'd already have started modifying the system to do that.

[deansheather]

I forgot to do it. If you want to do it, go ahead, but please keep my details in co-authored-by

[samchouse]

I opened deansheather#1 which implements it for remote forwarding, I'm unsure if this is also necessary for local forwarding. Lmk if the double callback looks fine.

[deansheather]

Thanks for your changes @samchouse

@gustavosbarreto the reverse forwarding callback now looks like:

// ReverseUnixForwardingCallback is a hook for allowing reverse unix forwarding
// (streamlocal-forward@openssh.com). Returning ErrRejected will reject the
// request.
type ReverseUnixForwardingCallback func(ctx Context, socketPath string) (net.Listener, error)

[iongion]

Looking for this functionality very much also

[belak]

I've given the PR a quick read - it all looks good to me, even though I don't know the specifics of how forwarding works with openssh, this all seems fairly logical.

I see there were changes pushed last month. Would you say this is ready to merge as-is, or are there any leftover changes you'd like to make first? I'd be happy to merge this when I get the go-ahead from you, @deansheather.

[deansheather]

I think it's all ready to go

[zerosnacks]

Thanks @deansheather, am looking for this functionality as well to enable local->remote Yubikey GPG signing w/ Tailscale SSH

[deansheather]

I wonder if maybe the Forward variant of the unix socket code should also require that the callback returns a net.Conn instead of just dialing directly. I can make the change if you'd like this @belak

[belak]

I've been looking around https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD, section 2.4 ("connection: Unix domain socket forwarding") and I'm a bit confused.

Direct is a channel request, streamlocal is a global request, but it's a little unclear how they're supposed to work:

I assume ReverseUnixForwardingCallback allows returning a net.Listener because there's a separate request you can make to open a conn against the target socket?

And LocalUnixForwardingCallback allows turning a channel into a unix socket proxy? If that's the case, net.Conn would be more in line with net.Listener, but I'm unclear how you'd handle this if you wanted to emulate a unix domain socket. Maybe that's not worth putting in scope?

// LocalUnixForwardingCallback is a hook for allowing unix forwarding
// (direct-streamlocal@openssh.com)
type LocalUnixForwardingCallback func(ctx Context, socketPath string) bool

// ReverseUnixForwardingCallback is a hook for allowing reverse unix forwarding
// (streamlocal-forward@openssh.com). Returning ErrRejected will reject the
// request.
type ReverseUnixForwardingCallback func(ctx Context, socketPath string) (net.Listener, error)

[deansheather]

The SSH implementation is very similar to local and reverse TCP forwarding and this implementation was based off the existing TCP implementation.

I assume ReverseUnixForwardingCallback allows returning a net.Listener because there's a separate request you can make to open a conn against the target socket?

Reverse forwarding is for forwarding client connections that originate from a new unix socket on the server to some listener on the client. Typically used to allow programs like curl running on the server to dial a socket on the client machine.

In the original implementation of this PR the callback just returned bool and the ssh package would handle creating a Unix socket, but it was changed to allow creating a fake listener or to control some aspects of the opened socket like permissions.

And LocalUnixForwardingCallback allows turning a channel into a unix socket proxy? If that's the case, net.Conn would be more in line with net.Listener, but I'm unclear how you'd handle this if you wanted to emulate a unix domain socket. Maybe that's not worth putting in scope?

If we change the callback to return a net.Conn it can be used to e.g. dial a different socket path or emulate a fake connection (e.g. you could handle the request in process without creating a real socket).

I would propose changing the API to:

// LocalUnixForwardingCallback is a hook for allowing unix forwarding
// (direct-streamlocal@openssh.com). Returning ErrRejected will reject the request.
type LocalUnixForwardingCallback func(ctx Context, socketPath string) (net.Conn, error)

[belak]

Thanks for taking the time to respond and clear that up - I think that change makes sense. Let me know when it's ready and I can merge it. :)

[deansheather]

I've made the changes. I'd also like to get someone from my company to review it first since we've made some changes to our version of this code in the last 22 odd months 🤣

[mafredri]

Looks great, I especially like the new abstraction with callbacks for allowing custom handling of the connection.

My only nit is unix request handling which doesn't currently allow for our use-case in coder/coder (the SimpleUnixReverseForwardingCallback function does, but the handler doesn't).

I also think documentation on how to enable this would be good to add either here or a follow-up PR, essentially you need to both set RequestHandlers and callback functions, which may be a bit confusing to users. Since the default is reject, would it make sense to add these request handlers to DefaultRequestHandlers?

[mafredri]

Granted, we haven't introduced a similar option to SSH StreamLocalBindUnlink here, and the default is no for OpenSSH, but in coder/coder we want its yes-behavior. (And my personal opinion is that it's the more useful default.)

If yes, the same socket can be forwarded by multiple sessions, in this case each session should maintain the connections that were opened while it was active. Otherwise the following scenario can't be supported:

1. Open SSH connection (a) with forwarded socket (/tmp/my.sock)
2. (a) Open /tmp/my.sock
3. Open SSH connection (b)
4. (b) Open /tmp/my.sock
5. Open SSH connection (c) with forwarded socket (/tmp/my.sock, overwrite)
6. (c) Open /tmp/my.sock
7. Close connection (a)
8. (a) closed (b) Socket closed (c) Socket remains open

In the current implementation, (c) socket can't remain open as the socket wasn't overwritten.

Also: Consider returning true here instead as a default (see motivation in below link).

See: https://github.com/coder/coder/blob/b828412edd913bef6665cf8a0b2ca7ac93334012/agent/agentssh/forward.go#L76-L91

[mafredri]

We use context aware dialer in the other callback, should we use (&net.ListenConfig{}).Listen(ctx, "unix", addr) here as well?