Node hostname is set to pod name

[benceszikora]

Description
We have noticed 2 nodes in our cluster that had their hostname set to the name of a pod running on them. At first glance it seems similar to issue #2122, but in our case the pod is not privileged and supposedly that issue was fixed already.

I can't really find anything related in the logs, the only interesting lines are like this:

Nov 18 13:42:48 fpig-kubeletl041 kubelet[56281]: E1118 13:42:48.686656   56281 remote_runtime.go:200] CreateContainer in sandbox "4e96475de1aac6f731a0498938701584cc340cea77e36bf4f8c02cd6d5a3c5c2" from runtime service failed: rpc error: code = Unknown desc = container create failed: container_linux.go:329: creating new parent process caused "container_linux.go:1762: running lstat on namespace path \"/proc/106264/ns/ipc\" caused \"lstat /proc/106264/ns/ipc: no such file or directory\""

Steps to reproduce the issue:
I can't reliably reproduce the issue. So far it happened on two nodes and both pods are Rook radosgw deployments.

Describe the results you received:
The hostnames of the nodes are set to the name of a pod running on them.

Describe the results you expected:
The hostname does not change

Additional information you deem important (e.g. issue happens only occasionally):
We have updated to crio 1.16.0 last Friday, and we have not seen this issue before.

Output of crio --version:

crio version 1.16.0
commit: "fa99ff4ae2aa45115bf3c6bb33db07191db2518e-dirty"

Example affected pod manifest:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 7.150.187.7/32
  generateName: rook-ceph-rgw-rook-objectstore-d-66975c8799-
  labels:
    app: rook-ceph-rgw
    ceph_daemon_id: rook-objectstore
    rgw: rook-objectstore
    rook_cluster: rook-ceph
    rook_object_store: rook-objectstore
  name: rook-ceph-rgw-rook-objectstore-d-66975c8799-lk98t
  namespace: rook-ceph
spec:
  containers:
  - args:
    - --fsid=7aeb9c40-f19a-49a4-9a2e-af7eaf85f214
    - --keyring=/etc/ceph/keyring-store/keyring
    - --log-to-stderr=true
    - --err-to-stderr=true
    - --mon-cluster-log-to-stderr=true
    - '--log-stderr-prefix=debug '
    - --default-log-to-file=false
    - --default-mon-cluster-log-to-file=false
    - --mon-host=$(ROOK_CEPH_MON_HOST)
    - --mon-initial-members=$(ROOK_CEPH_MON_INITIAL_MEMBERS)
    - --id=rook-objectstore
    - --setuser=ceph
    - --setgroup=ceph
    - --foreground
    - --name=client.rook.objectstore.d
    - --host=$(POD_NAME)
    - --rgw-mime-types-file=/etc/ceph/rgw/mime.types
    - --rgw-log-nonexistent-bucket=true
    - --rgw-intent-log-object-name-utc=true
    - --rgw-enable-usage-log=true
    - --rgw-frontends=beast port=80 ssl_port=443 ssl_certificate=/etc/ceph/private/rgw-cert.pem
    - --rgw-zone=rook-objectstore
    - --rgw-zonegroup=rook-objectstore
    command:
    - radosgw
    env:
    - name: CONTAINER_IMAGE
      value: ceph/ceph:v14.2.4-20190917
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_MEMORY_LIMIT
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: limits.memory
    - name: POD_MEMORY_REQUEST
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: requests.memory
    - name: POD_CPU_LIMIT
      valueFrom:
        resourceFieldRef:
          divisor: "1"
          resource: limits.cpu
    - name: POD_CPU_REQUEST
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: requests.cpu
    - name: ROOK_CEPH_MON_HOST
      valueFrom:
        secretKeyRef:
          key: mon_host
          name: rook-ceph-config
    - name: ROOK_CEPH_MON_INITIAL_MEMBERS
      valueFrom:
        secretKeyRef:
          key: mon_initial_members
          name: rook-ceph-config
    image: ceph/ceph:v14.2.4-20190917
    imagePullPolicy: IfNotPresent
    lifecycle:
      postStart:
        exec:
          command:
          - chown
          - --recursive
          - ceph:ceph
          - /var/log/ceph
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 80
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: rgw
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
    securityContext:
      privileged: false
    volumeMounts:
    - mountPath: /etc/ceph
      name: rook-config-override
      readOnly: true
    - mountPath: /etc/ceph/keyring-store/
      name: rook-ceph-rgw-rook-objectstore-d-keyring
      readOnly: true
    - mountPath: /var/log/ceph
      name: rook-ceph-log
    - mountPath: /var/lib/ceph/rgw/ceph-rook-objectstore
      name: ceph-daemon-data
    - mountPath: /etc/ceph/rgw
      name: rook-ceph-rgw-rook-objectstore-mime-types
      readOnly: true
    - mountPath: /etc/ceph/private
      name: rook-ceph-rgw-cert
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-f7zlb
      readOnly: true
  restartPolicy: Always
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      items:
      - key: config
        mode: 292
        path: ceph.conf
      name: rook-config-override
    name: rook-config-override
  - name: rook-ceph-rgw-rook-objectstore-d-keyring
    secret:
      defaultMode: 420
      secretName: rook-ceph-rgw-rook-objectstore-d-keyring
  - hostPath:
      path: /data/rook/rook-ceph/log
      type: ""
    name: rook-ceph-log
  - emptyDir: {}
    name: ceph-daemon-data
  - configMap:
      defaultMode: 420
      name: rook-ceph-rgw-rook-objectstore-mime-types
    name: rook-ceph-rgw-rook-objectstore-mime-types
  - name: rook-ceph-rgw-cert
    secret:
      defaultMode: 420
      items:
      - key: cert
        mode: 256
        path: rgw-cert.pem
      secretName: rook-objectstore-certificate
  - name: default-token-f7zlb
    secret:
      defaultMode: 420
      secretName: default-token-f7zlb

[haircommander]

hey @benceszikora thanks for reporting the issue

this bug is known to us, and we're working on a proper fix in CRI-O. Unforunately, most of our developers are at kubecon this week, and are pretty busy. I hope to finish tackling it in the next two weeks. It's definitely my top priority.

In the meantime, you can mitigate this by increasing the kernel.pid_max value to the maximum for your system.

[benceszikora]

@haircommander Thank you, we'll increase pid_max in the meantime. Do you know if thelstat /proc/106264/ns/ipc: no such file or directory message is related or should I keep looking into that?

[haircommander]

That's part of the same issue 😄

[sergeyfd]

Hello,

Any updates on the fix? We are seeing the same issue.

[haircommander]

This was fixed in #3042 which is in 1.16.2. backports to 1.15 and 1.14 are also planned

[sergeyfd]

@haircommander could you provide some details on how that fix is related to the issue with host name changes? I don't think that conmon was OOM killed in our case.

[haircommander]

The only situation in which we have seen this happen recently is described here https://capsule8.com/blog/oomypod-nothin-to-cri-o-bout/, which can only happen with a conmon OOM or conmon being killed before the container is.

[sergeyfd]

Thanks a lot! And I just confirmed that a container that managed to change host name was indeed OOM killed, so it must be it.

[haircommander]

ah hah, then increasing pids_max and using 1.16.2 should significantly decrease the chance of that happening again. Stay tuned for 1.17 where this situation should be fixed entirely :)

[haircommander]

we now set manage_ns_lifecycle by default in CRI-O, which allows CRI-O to have better control over the namespace lifecycle. Further, we've made conmon have an oom adjust score of -1000, so it is never OOM'd. As such, this issue should be entirely prevented.

As such, I'm closing this issue. please reopen if you disagree