oc cluster down is leaving secrets mounted

[jmontleon]

oc cluster down is leaving secrets mounted

Version

$ oc version
oc v3.10.0-alpha.0+1691e4d-400
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://127.0.0.1:8443
openshift v3.10.0-alpha.0+942dfa5-404
kubernetes v1.9.1+a0ce1bc657

Steps To Reproduce

$ mount | grep openshift

[jmontleo@jmontleo ~]$ oc cluster up --routing-suffix=172.18.0.1.nip.io --service-catalog=true --tag=latest
Starting OpenShift using openshift/origin:latest ...
I0328 18:31:45.062340   27712 config.go:38] Running "create-master-config"
I0328 18:31:47.316098   27712 config.go:45] Running "create-node-config"
I0328 18:31:48.944002   27712 flags.go:31] Running "create-kubelet-flags"
I0328 18:31:49.899157   27712 run_kubelet.go:48] Running "start-kubelet"
I0328 18:31:50.377985   27712 run_self_hosted.go:156] Waiting for the kube-apiserver to be ready.
I0328 18:32:11.381991   27712 apply_template.go:77] Installing "openshift-apiserver"
I0328 18:32:11.381991   27712 apply_template.go:77] Installing "kube-proxy"
I0328 18:32:11.382005   27712 apply_template.go:77] Installing "kube-dns"
I0328 18:32:13.387162   27712 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-apiserver"
I0328 18:32:54.430107   27712 run_self_hosted.go:196] openshift-apiserver available
I0328 18:32:54.430197   27712 apply_template.go:77] Installing "openshift-controller-manager"
I0328 18:32:56.240982   27712 interface.go:41] Finished installing "openshift-controller-manager"
I0328 18:32:56.341753   27712 apply_list.go:48] Installing "openshift/centos7"
I0328 18:32:56.341783   27712 apply_list.go:48] Installing "openshift/django quickstart"
I0328 18:32:56.341827   27712 apply_list.go:48] Installing "kube-system/heapster standalone"
I0328 18:32:56.341894   27712 apply_list.go:48] Installing "openshift/postgresql"
I0328 18:32:56.341917   27712 apply_list.go:48] Installing "openshift/nodejs quickstart"
I0328 18:32:56.341933   27712 apply_list.go:48] Installing "openshift/rails quickstart"
I0328 18:32:56.341958   27712 apply_list.go:48] Installing "openshift-infra/template service broker registration"
I0328 18:32:56.342032   27712 apply_list.go:48] Installing "kube-system/prometheus"
I0328 18:32:56.342045   27712 apply_list.go:48] Installing "openshift/sample pipeline"
I0328 18:32:56.342024   27712 apply_list.go:48] Installing "openshift-infra/template service broker apiserver"
I0328 18:32:56.342362   27712 apply_list.go:48] Installing "openshift-infra/service catalog"
I0328 18:32:56.342722   27712 apply_list.go:48] Installing "openshift/mongodb"
I0328 18:32:56.341951   27712 apply_list.go:48] Installing "openshift/dancer quickstart"
I0328 18:32:56.341972   27712 apply_list.go:48] Installing "openshift/cakephp quickstart"
I0328 18:32:56.341974   27712 apply_list.go:48] Installing "openshift-infra/template service broker rbac"
I0328 18:32:56.341993   27712 apply_list.go:48] Installing "openshift/mysql"
I0328 18:32:56.342008   27712 apply_list.go:48] Installing "openshift-infra/web console server template"
I0328 18:32:56.342012   27712 apply_list.go:48] Installing "openshift/mariadb"
I0328 18:32:56.341909   27712 apply_list.go:48] Installing "openshift/jenkins pipeline persistent"
I0328 18:32:56.351850   27712 registry_install.go:56] Running "openshift-image-registry"
scc "privileged" added to: ["system:serviceaccount:default:registry"]
I0328 18:33:02.605154   27712 interface.go:41] Finished installing "openshift/centos7" "openshift/mongodb" "openshift/mariadb" "openshift/postgresql" "openshift/cakephp quickstart" "openshift/dancer quickstart" "openshift/mysql" "openshift/django quickstart" "openshift/nodejs quickstart" "openshift/rails quickstart" "openshift/jenkins pipeline persistent" "openshift/sample pipeline" "kube-system/prometheus" "kube-system/heapster standalone" "openshift-infra/service catalog" "openshift-infra/template service broker rbac" "openshift-infra/template service broker registration" "openshift-infra/template service broker apiserver" "openshift-infra/web console server template" "openshift-image-registry"
I0328 18:33:02.618161   27712 admin.go:48] Running "install-router"
I0328 18:33:14.520026   27712 apply_template.go:77] Installing "tsb-apiserver"
I0328 18:33:27.432023   27712 apply_template.go:77] Installing "tsb-registration"
OpenShift server started.

The server is accessible via web console at:
    https://127.0.0.1:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

$ oc cluster down

$ rm -rf openshift.local.clusterup/
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods': Permission denied
rm: cannot remove 'openshift.local.clusterup/etcd/member': Permission denied

[jmontleo@jmontleo ~]$ sudo rm -rf openshift.local.clusterup/
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/de76288a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/kube-dns-token-w9xqv': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/de762be5-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/kube-proxy-token-r5cxl': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/de79b19a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/openshift-apiserver-token-ncjtv': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/f686db8a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/openshift-controller-manager-token-cgrdl': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/server-certificate': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/metrics-server-certificate': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/router-token-dvrq5': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/fc03f158-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/registry-token-vcsph': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/ff33e565-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-controller-token-66zqg': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/ff33e565-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-ssl': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/ff33e623-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/apiserver-ssl': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/ff33e623-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-apiserver-token-tzsd5': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/031ca1de-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/apiserver-token-ng95l': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/031ca1de-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/serving-cert': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/10c5db22-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/webconsole-token-kmkdk': Device or resource busy
rm: cannot remove 'openshift.local.clusterup/openshift.local.volumes/pods/10c5db22-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/serving-cert': Device or resource busy

$ mount | grep openshift
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/de76288a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/kube-dns-token-w9xqv type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/de762be5-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/kube-proxy-token-r5cxl type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/de79b19a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/openshift-apiserver-token-ncjtv type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/f686db8a-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/openshift-controller-manager-token-cgrdl type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/router-token-dvrq5 type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/metrics-server-certificate type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/fb86671c-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/server-certificate type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/fc03f158-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/registry-token-vcsph type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/ff33e565-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-ssl type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/ff33e623-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/apiserver-ssl type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/ff33e623-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-apiserver-token-tzsd5 type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/ff33e565-32d7-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/service-catalog-controller-token-66zqg type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/031ca1de-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/serving-cert type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/031ca1de-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/apiserver-token-ng95l type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/10c5db22-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/serving-cert type tmpfs (rw,relatime,seclabel)
tmpfs on /home/jmontleo/openshift.local.clusterup/openshift.local.volumes/pods/10c5db22-32d8-11e8-ab43-64006a559656/volumes/kubernetes.io~secret/webconsole-token-kmkdk type tmpfs (rw,relatime,seclabel)

Current Result

Secrets are left mounted

Expected Result

oc cluster down cleans up after itself

Additional Information

Having to do the following to completely clean up:

$ for i in $(mount | grep openshift | awk '{ print $3}'); do sudo umount "$i"; done && sudo rm -rf ./openshift.local.clusterup

[gnufied]

Is this for all pods or only some? Can you post pod's yaml that is using the secret?

/sig storage

[jmontleon]

I don't think it is all of them. The service catalog apiserver and controller-manager look like they're at least a couple of them. These were created with the --service-catalog=true option to oc cluster up.

$ oc export po -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      openshift.io/scc: restricted
    creationTimestamp: null
    generateName: apiserver-65bbbcf4cc-
    labels:
      app: apiserver
      pod-template-hash: "2166679077"
    ownerReferences:
    - apiVersion: extensions/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: apiserver-65bbbcf4cc
      uid: b4d2d2e5-334d-11e8-b778-64006a559656
  spec:
    containers:
    - args:
      - apiserver
      - --admission-control
      - KubernetesNamespaceLifecycle,DefaultServicePlan,ServiceBindingsLifecycle,ServicePlanChangeValidator,BrokerAuthSarCheck
      - --storage-type
      - etcd
      - --secure-port
      - "6443"
      - --etcd-servers
      - http://localhost:2379
      - -v
      - "10"
      - --cors-allowed-origins
      - 127.0.0.1
      - --feature-gates
      - OriginatingIdentity=true
      command:
      - service-catalog
      image: docker.io/openshift/origin-service-catalog:latest
      imagePullPolicy: IfNotPresent
      name: apiserver
      ports:
      - containerPort: 6443
        protocol: TCP
      resources: {}
      securityContext:
        capabilities:
          drop:
          - KILL
          - MKNOD
          - SETGID
          - SETUID
        runAsUser: 1000100000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/kubernetes-service-catalog
        name: apiserver-ssl
        readOnly: true
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: service-catalog-apiserver-token-lz8w4
        readOnly: true
    - env:
      - name: ETCD_DATA_DIR
        value: /data-dir
      image: quay.io/coreos/etcd
      imagePullPolicy: IfNotPresent
      name: etcd
      resources: {}
      securityContext:
        capabilities:
          drop:
          - KILL
          - MKNOD
          - SETGID
          - SETUID
        runAsUser: 1000100000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /data-dir
        name: data-dir
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: service-catalog-apiserver-token-lz8w4
        readOnly: true
    dnsPolicy: ClusterFirst
    nodeName: localhost
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 1000100000
      seLinuxOptions:
        level: s0:c10,c5
    serviceAccount: service-catalog-apiserver
    serviceAccountName: service-catalog-apiserver
    terminationGracePeriodSeconds: 30
    volumes:
    - name: apiserver-ssl
      secret:
        defaultMode: 420
        items:
        - key: tls.crt
          path: apiserver.crt
        - key: tls.key
          path: apiserver.key
        secretName: apiserver-ssl
    - emptyDir: {}
      name: data-dir
    - name: service-catalog-apiserver-token-lz8w4
      secret:
        defaultMode: 420
        secretName: service-catalog-apiserver-token-lz8w4
  status:
    phase: Pending
    qosClass: BestEffort

- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      openshift.io/scc: restricted
    creationTimestamp: null
    generateName: controller-manager-9dcb9888c-
    labels:
      app: controller-manager
      pod-template-hash: "587654447"
    ownerReferences:
    - apiVersion: extensions/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: controller-manager-9dcb9888c
      uid: b4d3e078-334d-11e8-b778-64006a559656
  spec:
    containers:
    - args:
      - controller-manager
      - --secure-port
      - "6444"
      - -v
      - "5"
      - --leader-election-namespace
      - kube-service-catalog
      - --broker-relist-interval
      - 5m
      - --feature-gates
      - OriginatingIdentity=true
      - --feature-gates
      - AsyncBindingOperations=true
      command:
      - service-catalog
      image: docker.io/openshift/origin-service-catalog:latest
      imagePullPolicy: IfNotPresent
      name: controller-manager
      ports:
      - containerPort: 6444
        protocol: TCP
      resources: {}
      securityContext:
        capabilities:
          drop:
          - KILL
          - MKNOD
          - SETGID
          - SETUID
        runAsUser: 1000100000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/kubernetes-service-catalog
        name: service-catalog-ssl
        readOnly: true
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: service-catalog-controller-token-6h4xk
        readOnly: true
    dnsPolicy: ClusterFirst
    nodeName: localhost
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
      fsGroup: 1000100000
      seLinuxOptions:
        level: s0:c10,c5
    serviceAccount: service-catalog-controller
    serviceAccountName: service-catalog-controller
    terminationGracePeriodSeconds: 30
    volumes:
    - name: service-catalog-ssl
      secret:
        defaultMode: 420
        items:
        - key: tls.crt
          path: apiserver.crt
        - key: tls.key
          path: apiserver.key
        secretName: apiserver-ssl
    - name: service-catalog-controller-token-6h4xk
      secret:
        defaultMode: 420
        secretName: service-catalog-controller-token-6h4xk
  status:
    phase: Pending
    qosClass: BestEffort
kind: List
metadata: {}

[jmontleon]

I guess this might be related to #19139 as well.

[jmontleon]

Even without the service catalog I see secrets being left mounted for registry, router, metrics, webconsole, and new kube-* pods.

[jwforres]

@deads2k is this issue still relevant with all of your recent changes?

[deads2k]

@deads2k is this issue still relevant with all of your recent changes?

Looks like a kubelet thing.

@sjenning I think this is normal behavior, but oc cluster down should probably umount them. I'm not familiar with how to find them or how to unmount them. Here's where cluster down starts: https://github.com/openshift/origin/blob/master/pkg/oc/bootstrap/docker/down.go

/assign @sjenning

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[coreydaley]

/remove-lifecycle stale