tilt 0.31.3 errors in a docker-in-docker environment

[kmannislands]

Expected Behavior

We've been running tilt ci in a docker-in-docker environment prepared with kind for some time now. This has been working pretty much flawlessly so far.

Current Behavior

Our CI stage for tilt recently started failing after we rebuilt CI base images (installing tilt via scripts/install.sh) and moved from tilt 0.31.1 to 0.31.3.

Without other changes, tilt at 0.31.3 started having connectivity issues to our docker host, reporting Error: Failed to connect to Docker: Error response from daemon: Client sent an HTTP request to an HTTPS server.. Other docker-in-docker stages running in the same pipeline/against the same base images using docker directly continue to work.

I've tested 0.31.2 and the error is absent there so it seems the change was introduced in 0.31.3.

I'm a bit hesitant to call this a bug since it may be a quirk of our docker-in-docker setup and/or docker versions we're running but wanted to report anyways in case others happen to experience the same issue and I can save someone a couple hours of debugging!

Context

tilt doctor Output with 0.31.3 (broken)

$ tilt doctor
00:00
Tilt: v0.33.3, built 2023-07-25
System: linux-amd64
---
Docker
- Host: [default]
- Server Version: 
- API Version: 
- Builder: 
- Compose Version: v2.20.2
---
Kubernetes
- Env: kind
- Context: kind-kind
- Cluster Name: kind-kind
- Namespace: gitlab
- Container Runtime: containerd
- Version: v1.25.3
- Cluster Local Registry: &RegistryHosting{Host:localhost:43355,HostFromClusterNetwork:ctlptl-registry:5000,HostFromContainerRuntime:,Help:https://github.com/tilt-dev/ctlptl,SingleName:,}
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗
The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: default
- Machine: 8b3eb96cc7abe5d1e52c3fc977be44f3
- Repo: sCvwMqBThzkPcNL6vzjf7g==
$ tilt ci --timeout 60m0s
00:49
Tilt started on http://localhost:10350/
v0.33.3, built 2023-07-25
Tilt analytics disabled: Environment variable CI=true
I0807 23:42:49.799282     788 handler.go:232] Adding GroupVersion tilt.dev v1alpha1 to ResourceManager
Initial Build
Loading Tiltfile at: /builds/engineering/surgery-app/Tiltfile
# redacted
Successfully loaded Tiltfile (22.325466ms)
Auto-detected local registry from environment: &RegistryHosting{Host:localhost:43355,HostFromClusterNetwork:ctlptl-registry:5000,HostFromContainerRuntime:,Help:https://github.com/tilt-dev/ctlptl,SingleName:,}
ERROR: Failed to connect to Docker: Error response from daemon: Client sent an HTTP request to an HTTPS server.
Error: Failed to connect to Docker: Error response from daemon: Client sent an HTTP request to an HTTPS server.

Here tilt doctor can't seem to report docker versions but it's still docker server 20.10.17 and the CLI is was tested with 24.0.2 and 24.0.5 with the same result.

tilt doctor Output with 0.31.1 (working)

Tilt: v0.33.1, built 2023-06-28
System: linux-amd64
---
Docker
- Host: tcp://docker:2376
- Server Version: 20.10.17
- API Version: 1.41
- Builder: 2
- Compose Version: v2.20.2
---
Kubernetes
- Env: kind
- Context: kind-kind
- Cluster Name: kind-kind
- Namespace: gitlab
- Container Runtime: containerd
- Version: v1.25.3
- Cluster Local Registry: &RegistryHosting{Host:localhost:43773,HostFromClusterNetwork:ctlptl-registry:5000,HostFromContainerRuntime:,Help:https://github.com/tilt-dev/ctlptl,SingleName:,}
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: default
- Machine: 839585879f48f03603cf2e4f9886b66f
- Repo: sCvwMqBThzkPcNL6vzjf7g==

About Your Use Case

Tilt continues to run fine for me locally, only the tilt ci usage with DnD was blocked.

[nicks]

thanks for the report! 0.31.3 contains an upgrade to the docker client, so maybe the tls configs aren't being picked up properly now? can you post the output of:

env | grep DOCKER

?

[kmannislands]

Sure, in both cases the relevant env variables are:

DOCKER_TLS_VERIFY=1
DOCKER_TLS_CERTDIR=/certs
DOCKER_CERT_PATH=/certs/client
DOCKER_BUILDKIT=1
DOCKER_HOST=tcp://docker:2376
DOCKER_REGISTRY=<redacted ecr url>
DOCKER_HOST=tcp://docker:2376
DOCKER_DRIVER=overlay2

[quzhi1]

We have having the same issue with tilt up and tilt ci. Is there anyway we can fix it other than using a older version?

[quzhi1]

This is the output of env | grep DOCKER on my machine:

DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://127.0.0.1:59860
DOCKER_CERT_PATH=/Users/zhi.q@nylas.com/.minikube/certs
MINIKUBE_ACTIVE_DOCKERD=minikube
DOCKER_BUILDKIT=0
COMPOSE_DOCKER_CLI_BUILD=0