GO 1.20.6 breaks cri-dockerd

[adpavlov]

Environmental Info:
K3s Version: 1.27.4

Describe the bug:

Issue is same as https://discuss.kubernetes.io/t/http-invalid-host-header-error-while-executing-a-command-in-pod/24868
Downgrade to v1.27.3 solves issue

[brandond]

Are you also using cri-dockerd? If so, the one bundled with k3s, or did you install it separately?

[mgabeler-lee-6rs]

Not the original reporter, but suffering from this too. In my case, yes, we are using the copy of cri-dockerd embedded in k3s so that we can use docker in place of containerd

[brandond]

We are strict about building K3s releases with the same version of golang as upstream Kubernetes, so reverting to an older golang release (and reintroducing a CVE) is not a choice for us. I'll see if there's a good way to pull in the fix from moby/moby#45942

[adpavlov]

Are you also using cri-dockerd? If so, the one bundled with k3s, or did you install it separately?

yes, backend is docker from k3s itself. And seems like it's also affecting pod to pod communication as well since my deployment failed to start.

[jstewart612]

Confirming this bit me as well. This renders Longhorn Engine Image pods incapable of becoming ready. They shell out in a way that hits this exact same error message.

[mdrahman-suse]

Validated on master branch with commit 9702f92

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

PRETTY_NAME="Ubuntu 22.04.2 LTS"

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
cluster-init: true
docker: true

Additional files

• nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s
3. Ensure k3s is up and running
4. Deploy nginx-deployment.yaml
5. Check cri-dockerd version
6. Check deployments

$ kubectl get pods -A | grep nginx
default       nginx-deployment-cbdccf466-b4cqr         1/1     Running     0          21s
default       nginx-deployment-cbdccf466-xh4xl         1/1     Running     0          21s

Replication Results:

• k3s version used for replication:

k3s version v1.27.4+k3s1 (36645e73)
go version go1.20.6

• cri-dockerd version from logs

cri-dockerd version v0.3.2-k3s2 (HEAD)

• Execute command in Pod's container via kubectl exec, observed the error http: invalid Host header

$ kubectl exec -it -n default nginx-deployment-cbdccf466-b4cqr -- bash
error: Internal error occurred: error executing command in container: http: invalid Host header

Validation Results:

• k3s version used for validation:

k3s version v1.27.4+k3s-9702f923 (9702f923)
go version go1.20.6

• cri-dockerd version from logs

cri-dockerd version v0.3.4-k3s1 (HEAD)

• Execute command in Pod's container via kubectl exec, observed no error

$ kubectl exec -it -n default nginx-deployment-cbdccf466-b4cqr -- bash
root@nginx-deployment-cbdccf466-b4cqr:/#

Additional context / logs:

• Tested on new install and upgrades