Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsolicited response using NetConn #415

Open
maggie44 opened this issue Oct 31, 2023 · 1 comment
Open

Unsolicited response using NetConn #415

maggie44 opened this issue Oct 31, 2023 · 1 comment

Comments

@maggie44
Copy link
Contributor

maggie44 commented Oct 31, 2023

I have the below for communicating via a websocket to a remote docker instance using Docker's Go SDK ('client' is the Docker SDK package):

	"github.com/docker/docker/api/types"
	"github.com/docker/docker/client"
	"nhooyr.io/websocket"

...


	dockerHost := "http://docker"
	wrappedConn := websocket.NetConn(ctx, c, websocket.MessageBinary)

	// Custom dial function that returns the wrapped WebSocket connection
	customDial := func(ctx context.Context, network, addr string) (net.Conn, error) {
		return wrappedConn, nil
	}
	cli, _ := client.NewClientWithOpts(client.WithDialContext(customDial), client.WithHost(dockerHost))

On the receiving end I accept the request and relay to the Docker socket:

...
	// Connect to Docker Unix Socket with context
	dialer := &net.Dialer{}
	dockerConn, err := dialer.DialContext(ctx, "unix", "/var/run/docker.sock")
	if err != nil {
		slog.Error("error connecting to Docker daemon", "error", err)
		return
	}
	defer dockerConn.Close()

	// Relay from Docker to WebSocket
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		defer wg.Done()
		_, err := io.Copy(websocket.NetConn(ctx, wsConn, websocket.MessageBinary), dockerConn)
		if err != nil && ctx.Err() == nil {
			slog.Error("error relaying data from Docker socket to WebSocket", "error", err)
		}
	}()

	// Relay from WebSocket to Docker
	_, err = io.Copy(dockerConn, websocket.NetConn(ctx, wsConn, websocket.MessageBinary))
	if err != nil {
		slog.Error("error relaying data from WebSocket to Docker socket", "error", err)
	}

It works great out the box, brilliant feature, with one exception. When trying to connect to attach to a container, the Docker API hijacks the connection and for some reason this seems to stump the NetConn:

	execID, err := cli.ContainerExecCreate(ctx, containerID, execConfig)
	if err != nil {
		panic(err)
	}

	// Attach to the exec instance
	resp, err := cli.ContainerExecAttach(ctx, execID.ID, types.ExecStartCheck{})
	if err != nil {
		panic(err)
	}
	defer resp.Close()

Error message:

2023/10/30 17:45:19 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\nConnection"; err=<nil>

The error varies on each request

2023/10/30 17:52:22 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\nConnection: Upgrade\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nDocker-Experimental: false\r\nOstype: linux\r\nServer: Docker/24.0.6 (linux)\r\nUpgrade: tcp\r\n\r\n"; err=<nil>
2023/10/30 17:53:37 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 101 UPGRADED\r\nApi-Version: 1.43\r\n"; err=<nil>

If I call cli.ContainerExecAttach it goes through, but if I call cli.ContainerExecCreate and then cli.ContainerExecAttach on the same connection consecutively it errors. Something about cli.ContainerExecAttach specifically which does a hijack and isn't happy unless it is the first request made on the websocket. Other non-hijack consecutive commands go through ok.

Managed to narrow it down to on the docker end to: https://github.com/moby/moby/blob/311b9ff0aa93aa55880e1e5f8871c4fb69583426/client/hijack.go#L86C1-L86C1

Hard to understand why it would conflict with the websocket connection only on second requests.

@nhooyr
Copy link
Contributor

nhooyr commented Nov 12, 2023

I strongly suspect this is related to net/http's Transport not giving us back the raw net.Conn when performing an upgrade. Thus as we read the data from the handle we do get, it's processing it to ensure there's no unexpected HTTP. But I'll have to investigate to confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants