Buildctl sends malformed :authority header when connecting to Buildkitd TCP service

[everpeace]

TL;DR

• I setup buildkitd tcp service behind envoy proxy (with docker-compose)
• In this environment buildctl can connect buildkitd directly, But can't through envoy.
• envoy log says: invalid frame: Invalid HTTP header field was received on stream 1
• after I digged up, buildctl sends an:authority pseudo header :authority tcp://127.0.0.1:1234.
• but, this is not rfc compliant. valid authority header does not contain a scheme (https://tools.ietf.org/html/rfc3986#section-3.2)

What I've faced

I tried to run Buildkitd TCP service behind http proxy (envoy proxy, actually) because I needed to filter out Prune/DiskUsage method. So, I setup an experimental environment with docker-compose:

https://gist.github.com/everpeace/ea9582347e2c9d0bb519cdae1b438ead

this experimental environment spawns:

• envoy proxy on localhost:1234
  • this forwards to buildkitd
• buildkitd on localhost:11234
  • just for debug

In this environment,

buildctl can connect to buildkitd directly:

$ buildctl --debug --addr tcp://127.0.0.1:11234 debug workers
ID                              PLATFORMS
xflodox4is8qyxb9ewq9bl1l3       linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6

However buildctl can't connect to buildkitd through envoy:

$ buildctl --debug --addr tcp://127.0.0.1:1234 debug workers
error: rpc error: code = Unavailable desc = transport is closing
failed to list workers
github.com/moby/buildkit/client.(*Client).ListWorkers
        /src/client/workers.go:32
github.com/moby/buildkit/cmd/buildctl/debug.listWorkers
        /src/cmd/buildctl/debug/workers.go:41
github.com/urfave/cli.HandleAction
        /src/vendor/github.com/urfave/cli/app.go:523
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:174
github.com/urfave/cli.(*App).RunAsSubcommand
        /src/vendor/github.com/urfave/cli/app.go:404
github.com/urfave/cli.Command.startApp
        /src/vendor/github.com/urfave/cli/command.go:373
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:102
github.com/urfave/cli.(*App).Run
        /src/vendor/github.com/urfave/cli/app.go:276
main.main
        /src/cmd/buildctl/main.go:100
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

And, envoy's debug log says: invalid frame: Invalid HTTP header field was received on stream 1.

envoy_1      | [2020-07-14 13:51:17.913][14][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C8] new connection
envoy_1      | [2020-07-14 13:51:17.914][14][debug][http2] [source/common/http/http2/codec_impl.cc:970] [C8] updating connection-level initial window size to 268435456
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http] [source/common/http/conn_manager_impl.cc:268] [C8] new stream
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http2] [source/common/http/http2/codec_impl.cc:670] [C8] invalid frame: Invalid HTTP header field was received on stream 1
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:283] [C8] dispatch error: The user callback function failed
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:1936] [C8][S4570179258856159313] stream reset
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl.cc:109] [C8] closing data_to_write=0 type=2
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl_base.cc:30] [C8] setting delayed close timer with timeout 1000 ms
envoy_1      | [2020-07-14T13:51:17.916Z] "- - HTTP/2" 0 DPE 0 0 1 - "-" "-" "-" "-" "-"
envoy_1      | [2020-07-14 13:51:18.919][14][debug][connection] [source/common/network/connection_impl_base.cc:44] [C8] triggered delayed close
envoy_1      | [2020-07-14 13:51:18.920][14][debug][connection] [source/common/network/connection_impl.cc:200] [C8] closing socket: 1
envoy_1      | [2020-07-14 13:51:18.920][14][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C8] adding to cleanup list
envoy_1      | [2020-07-14 13:51:18.924][12][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C9] new connection
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:558] [C9] remote close
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:200] [C9] closing socket: 0
envoy_1      | [2020-07-14 13:51:18.927][12][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C9] adding to cleanup list
envoy_1      | [2020-07-14 13:51:19.212][1][debug][main] [source/server/server.cc:177] flushing stats

The issue

After I digged up, I found a similar issue in linkerd linkerd/linkerd2#2111 (comment). And I also confirmed buildctl on the latest master branch still sends a malformed :authority pseudo header :authority tcp://127.0.0.1:1234. According to RFC3986, valid authority header does not contain a scheme.

(I confirmed with wireshark)

Possible Solution

As the linkerd issue described above linkerd/linkerd2#2111 (comment) mentioned, because buildkit uses own dialers, we could probably use WithAuthority grpc's dial option.