Skip to content
This repository has been archived by the owner on Sep 9, 2022. It is now read-only.

cat: can't open '/tmp/secret': No such file or directory #26

Closed
amitkarpe opened this issue Aug 12, 2020 · 7 comments
Closed

cat: can't open '/tmp/secret': No such file or directory #26

amitkarpe opened this issue Aug 12, 2020 · 7 comments

Comments

@amitkarpe
Copy link

Hi Team,

Current code for webserver.yaml is not working.

Confirmed that created secret is accessible using aws command.

 ➜ aws secretsmanager get-secret-value --secret-id test_secretB --query SecretString --output text
{"username":"user3", "password": "pass3"}

Following is the webserver.yaml code where serviceAccountName set to use default sa.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: webserver
  name: webserver
spec:
  replicas: 1
  selector:
    matchLabels:
      run: webserver
  template:
    metadata:
      annotations:
        secrets.k8s.aws/sidecarInjectorWebhook: enabled
        secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:test_secretB-wFblqy
      labels:
        run: webserver
    spec:
      serviceAccountName:  default
      containers:
      - image: busybox:1.28
        name: webserver
        command: ['sh', '-c', 'echo $(cat /tmp/secret) && sleep 3600']

 ➜ kubectl version --short
Client Version: v1.18.4
Server Version: v1.15.11-eks-af3caf

 ➜ helm ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
secret-inject   secret          1               2020-08-12 11:42:00.697245 +0800 +08    deployed        secret-inject-0.1.2     1

 ➜ kubectl get mutatingwebhookconfiguration
NAME                CREATED AT
aws-secret-inject   2020-08-12T03:42:01Z

 ➜ k get sa default -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/test_secret
  creationTimestamp: "2020-08-11T08:59:55Z"
  name: default
  namespace: secret
  resourceVersion: "22054368"
  selfLink: /api/v1/namespaces/secret/serviceaccounts/default
  uid: 7e10b31f-47a7-4f0c-8bf1-1c3f5afc79de
secrets:
- name: default-token-d5cwh

 ➜ kl secret-inject-7b8b67fc48-hk87h -f
2020/08/12 03:43:35 http: TLS handshake error from 10.23.0.60:59350: remote error: tls: bad certificate
2020/08/12 03:44:46 http: TLS handshake error from 10.23.1.233:49986: remote error: tls: bad certificate
2020/08/12 06:14:50 http: TLS handshake error from 10.23.0.60:49230: remote error: tls: bad certificate
^C

 ➜ kgp webserver-888fc6786-4z7hp -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
    secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:test_secretB-wFblqy
    secrets.k8s.aws/sidecarInjectorWebhook: enabled
  creationTimestamp: "2020-08-12T06:14:50Z"
  generateName: webserver-888fc6786-
  labels:
    pod-template-hash: 888fc6786
    run: webserver
  name: webserver-888fc6786-4z7hp
  namespace: secret
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: webserver-888fc6786
    uid: 81fcc122-e428-49d0-a0ba-71889a875b45
  resourceVersion: "22205783"
  selfLink: /api/v1/namespaces/secret/pods/webserver-888fc6786-4z7hp
  uid: 6087099f-7f9b-4908-a56d-ee3398e657a9
spec:
  containers:
  - command:
    - sh
    - -c
    - echo $(cat /tmp/secret) && sleep 3600
    image: busybox:1.28
    imagePullPolicy: IfNotPresent
    name: webserver
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-d5cwh
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-23-7-28.ap-southeast-1.compute.internal
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-d5cwh
    secret:
      defaultMode: 420
      secretName: default-token-d5cwh
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-08-12T06:14:50Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-08-12T06:14:52Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-08-12T06:14:52Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-08-12T06:14:50Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://b47228f01099a63006aba623a2c99966432baca592b51c73f5979124337117b5
    image: busybox:1.28
    imageID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    lastState: {}
    name: webserver
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-08-12T06:14:51Z"
  hostIP: 10.23.7.28
  phase: Running
  podIP: 10.23.6.103
  qosClass: BestEffort
  startTime: "2020-08-12T06:14:50Z"

 ➜ kl -l run=webserver -f
cat: can't open '/tmp/secret': No such file or directory

^C
@amitkarpe
Copy link
Author

Hi,

Looks like init container did not run and so secrets-init-container is missing in the pod description.

 ➜ kubectl describe pod -l run=webserver
Name:           webserver-888fc6786-4z7hp
Namespace:      secret
Priority:       0
Node:           ip-10-23-7-28.ap-southeast-1.compute.internal/10.23.7.28
Start Time:     Wed, 12 Aug 2020 14:14:50 +0800
Labels:         pod-template-hash=888fc6786
                run=webserver
Annotations:    kubernetes.io/psp: eks.privileged
                secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:test_secretB-wFblqy
                secrets.k8s.aws/sidecarInjectorWebhook: enabled
Status:         Running
IP:             10.23.6.103
IPs:            <none>
Controlled By:  ReplicaSet/webserver-888fc6786
Containers:
  webserver:
    Container ID:  docker://b47228f01099a63006aba623a2c99966432baca592b51c73f5979124337117b5
    Image:         busybox:1.28
    Image ID:      docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      echo $(cat /tmp/secret) && sleep 3600
    State:          Running
      Started:      Wed, 12 Aug 2020 14:14:51 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-d5cwh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-d5cwh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-d5cwh
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                    Message
  ----    ------     ----  ----                                                    -------
  Normal  Scheduled  15m   default-scheduler                                       Successfully assigned secret/webserver-888fc6786-4z7hp to ip-10-23-7-28.ap-southeast-1.compute.internal
  Normal  Pulled     15m   kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal  Container image "busybox:1.28" already present on machine
  Normal  Created    15m   kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal  Created container webserver
  Normal  Started    15m   kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal  Started container webserver

@antonosmond
Copy link

@amitkarpe You have the same problem I did. See #24.
Until #21 is merged, secret-inject must be deployed in the default namespace.

@amitkarpe
Copy link
Author

Hi @antonosmond ,

I tried with default namespace still, I am facing the same issue.

@amitkarpe
Copy link
Author

Hi @jicowan,

I have created new EKS cluster. I used default namespace.
I was able to see secrets-init-container as "Init Containers". But can got access /tmp/secret.

 ➜ kl -l app=webserver -f

cat: can't open '/tmp/secret': No such file or directory
^C

Deployment Description

➜ kdp -l app=webserver                                                                                                                                                                                                             [52/577]
Name:         webserver-7c597ffbfc-qx7kf
Namespace:    default
Priority:     0
Node:         ip-192-168-252-96.ec2.internal/192.168.252.96
Start Time:   Fri, 14 Aug 2020 12:48:10 +0800
Labels:       app=webserver
              pod-template-hash=7c597ffbfc
              run=webserver
Annotations:  kubernetes.io/psp: eks.privileged
              secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:us-east-1:333438771545:secret:mysec-FfLLqF
              secrets.k8s.aws/sidecarInjectorWebhook: enabled
Status:       Running
IP:           192.168.193.231
IPs:
  IP:           192.168.193.231
Controlled By:  ReplicaSet/webserver-7c597ffbfc
Init Containers:
  secrets-init-container:
    Container ID:   docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
    Image:          docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1
    Image ID:       docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:079028dde7738a622ecbf35ee6b345bba7a0a5f40584c4e97e21535ced0cc421
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 14 Aug 2020 12:48:18 +0800
      Finished:     Fri, 14 Aug 2020 12:48:18 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      SECRET_ARN:                    (v1:metadata.annotations['secrets.k8s.aws/secret-arn'])
      AWS_ROLE_ARN:                 arn:aws:iam::333438771545:role/test_sec
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /tmp from secret-vol (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-k8jgl (ro)
Containers:
  webserver:
    Container ID:  docker://60ed9642c55aba1a593c286e6a73ddc6281f6d0e9d6e7551a2c1d18c2521df8b
    Image:         amitkarpe/aws-cli
    Image ID:      docker-pullable://amitkarpe/aws-cli@sha256:2cd871e804eb7d3ae645129079174c51de102d4d91fc6781c22df74c439f320d
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      echo $(cat /tmp/secret) && sleep 3600
    State:          Running
      Started:      Fri, 14 Aug 2020 12:48:24 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      AWS_ROLE_ARN:                 arn:aws:iam::333438771545:role/test_sec
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /tmp/ from secret-vol (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-k8jgl (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  default-token-k8jgl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-k8jgl
    Optional:    false
  secret-vol:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                     Message
  ----    ------     ----  ----                                     -------
  Normal  Scheduled  27s   default-scheduler                        Successfully assigned default/webserver-7c597ffbfc-qx7kf to ip-192-168-252-96.ec2.internal
  Normal  Pulling    26s   kubelet, ip-192-168-252-96.ec2.internal  Pulling image "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1"
  Normal  Pulled     20s   kubelet, ip-192-168-252-96.ec2.internal  Successfully pulled image "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1"
  Normal  Created    19s   kubelet, ip-192-168-252-96.ec2.internal  Created container secrets-init-container
  Normal  Started    19s   kubelet, ip-192-168-252-96.ec2.internal  Started container secrets-init-container
  Normal  Pulling    18s   kubelet, ip-192-168-252-96.ec2.internal  Pulling image "amitkarpe/aws-cli"
  Normal  Pulled     14s   kubelet, ip-192-168-252-96.ec2.internal  Successfully pulled image "amitkarpe/aws-cli"
  Normal  Created    14s   kubelet, ip-192-168-252-96.ec2.internal  Created container webserver
  Normal  Started    13s   kubelet, ip-192-168-252-96.ec2.internal  Started container webserver

Pod status

  status:                                                                                                                                                                                                                            [4/970]
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2020-08-14T04:48:19Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2020-08-14T04:48:24Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2020-08-14T04:48:24Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2020-08-14T04:48:10Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://60ed9642c55aba1a593c286e6a73ddc6281f6d0e9d6e7551a2c1d18c2521df8b
      image: amitkarpe/aws-cli:latest
      imageID: docker-pullable://amitkarpe/aws-cli@sha256:2cd871e804eb7d3ae645129079174c51de102d4d91fc6781c22df74c439f320d
      lastState: {}
      name: webserver
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2020-08-14T04:48:24Z"
    hostIP: 192.168.252.96
    initContainerStatuses:
    - containerID: docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
      image: amazon/aws-secrets-manager-secret-sidecar:v0.1.1
      imageID: docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:079028dde7738a622ecbf35ee6b345bba7a0a5f40584c4e97e21535ced0cc421
      lastState: {}
      name: secrets-init-container
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
          exitCode: 0
          finishedAt: "2020-08-14T04:48:18Z"
          reason: Completed
          startedAt: "2020-08-14T04:48:18Z"
    phase: Running
    podIP: 192.168.193.231
    podIPs:
    - ip: 192.168.193.231
    qosClass: BestEffort
    startTime: "2020-08-14T04:48:10Z"

@amitkarpe
Copy link
Author

Hi,
When I tried to access s3 or SM, I got following error:

 ➜ export POD=$(kubectl get pods -l "app=${app}" -o jsonpath="{.items[0].metadata.name}"); kubectl exec -it $POD  -- sh
/ # aws secretsmanager get-secret-value --secret-id mysec --region us-east-1

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
/ # aws s3 ls

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
/ #

@jicowan
Copy link
Contributor

jicowan commented Aug 14, 2020

Looks like the service account you are referencing in your pod.spec doesn't have permission to read the secret from Secrets Manager. If the SA is not mapped to a role that has access to your secret, the init container will not be able to write the secret to the volume.

@amitkarpe
Copy link
Author

@jicowan
Thank you for your feedback.
I was not using the correct Trust Relationship. After using more generic like "system:serviceaccount:*" and "StringLike" instead of "StringEquals", I was able to solve the problem.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@amitkarpe @antonosmond @jicowan