Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EBS Volume (AWS) ]AttachVolume.Attach failed for volume "redis-data": error finding instance ip-10-10-178-213.cn-north-1.compute.internal: "instance not found" #77920

Open
onlylin opened this issue May 15, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@onlylin
Copy link

commented May 15, 2019

What happened:
Warning FailedAttachVolume 3s (x6 over 19s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-41771a5f-76f6-11e9-91d3-021bc1c59f08" : error finding instance ip-10-10-178-213.cn-north-1.compute.internal: "instance not found"

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
ec2 tag:
kubernetes.io/cluster/ip-10-10-178-213.cn-north-1.compute.internal | owned

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g: cat /etc/os-release):
    NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a):
    Linux ip-10-10-178-14.cn-north-1.compute.internal 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    kubeadm
  • Network plugin and version (if this is a network-related bug):
  • Others:
@onlylin

This comment has been minimized.

Copy link
Author

commented May 15, 2019

/sig aws

@k8s-ci-robot k8s-ci-robot added sig/aws and removed needs-sig labels May 15, 2019

@onlylin

This comment has been minimized.

Copy link
Author

commented May 15, 2019

storageclass yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp2
  fsType: ext4

pvc yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: redis-data
spec:
  accessModes:
    - ReadWriteOnce  
  storageClassName: gp2
  resources:
    requests:
      storage: 5Gi

pod yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  template:
    metadata:
      labels:
        application: redis
        version: 3.2.5
    spec:
      containers:
      - name: redis
        image: redis:3.2.5
        volumeMounts:
        - mountPath: /data
          name: redis-data
      volumes:
        - name: redis-data
          persistentVolumeClaim:
            claimName: redis-data

kubectl describe pv

Name:              pvc-41771a5f-76f6-11e9-91d3-021bc1c59f08
Labels:            failure-domain.beta.kubernetes.io/region=cn-north-1
                   failure-domain.beta.kubernetes.io/zone=cn-north-1a
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      gp2
Status:            Bound
Claim:             default/redis-data
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          5Gi
Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/zone in [cn-north-1a]
                   failure-domain.beta.kubernetes.io/region in [cn-north-1]
Message:
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://cn-north-1a/vol-083ec7d3a5aeec204
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

kubectl describe pvc

Name:          redis-data
Namespace:     default
StorageClass:  gp2
Status:        Bound
Volume:        pvc-41771a5f-76f6-11e9-91d3-021bc1c59f08
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
               volume.kubernetes.io/selected-node: ip-10-10-178-213.cn-north-1.compute.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age                From                         Message
  ----       ------                 ----               ----                         -------
  Normal     WaitForFirstConsumer   14m (x8 over 15m)  persistentvolume-controller  waiting for first consumer to be created before binding
  Normal     ProvisioningSucceeded  14m                persistentvolume-controller  Successfully provisioned volume pvc-41771a5f-76f6-11e9-91d3-021bc1c59f08 using kubernetes.io/aws-ebs
Mounted By:  redis-75b5db55c6-zg7v6

kubectl describe pod

Name:               redis-75b5db55c6-zg7v6
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-10-178-213.cn-north-1.compute.internal/10.10.178.213
Start Time:         Wed, 15 May 2019 09:47:33 +0000
Labels:             application=redis
                    pod-template-hash=75b5db55c6
                    version=3.2.5
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      ReplicaSet/redis-75b5db55c6
Containers:
  redis:
    Container ID:
    Image:          redis:3.2.5
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from redis-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8m7j5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  redis-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-data
    ReadOnly:   false
  default-token-8m7j5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8m7j5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason              Age                 From                                                   Message
  ----     ------              ----                ----                                                   -------
  Normal   Scheduled           14m                 default-scheduler                                      Successfully assigned default/redis-75b5db55c6-zg7v6 to ip-10-10-178-213.cn-north-1.compute.internal
  Warning  FailedMount         85s (x6 over 12m)   kubelet, ip-10-10-178-213.cn-north-1.compute.internal  Unable to mount volumes for pod "redis-75b5db55c6-zg7v6_default(76088581-76f6-11e9-91d3-021bc1c59f08)": timeout expired waiting for volumes to attach or mount for pod "default"/"redis-75b5db55c6-zg7v6". list of unmounted volumes=[redis-data]. list of unattached volumes=[redis-data default-token-8m7j5]
  Warning  FailedAttachVolume  27s (x15 over 14m)  attachdetach-controller                                AttachVolume.Attach failed for volume "pvc-41771a5f-76f6-11e9-91d3-021bc1c59f08" : error finding instance ip-10-10-178-213.cn-north-1.compute.internal: "instance not found"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.