Could not attach ebs volume more than 50 for one node

[chao007]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.1.355+67859efaec9118-dirty", GitCommit:"67859efaec9118a04e8a42adf6de4a8850b869a0", GitTreeState:"dirty", BuildDate:"2017-02-04T06:10:14Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.1.355+67859efaec9118-dirty", GitCommit:"67859efaec9118a04e8a42adf6de4a8850b869a0", GitTreeState:"dirty", BuildDate:"2017-02-04T06:10:14Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
I create 100 pv, pvc and pods, but 50 pods could not running

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
1.export KUBE_MAX_PD_VOLS=100
2.hack/local_up_cluster.sh to run a k8s env
3.Create 100 pv, 100 pvc and 100 pods
4.Check pods status
cluster/kubectl.sh describe pods mypod90
Name: mypod90
Namespace: default
Node: ip-172-18-9-70.ec2.internal/172.18.9.70
Start Time: Tue, 14 Feb 2017 03:28:09 -0500
Labels: name=frontendhttp
Status: Pending
IP:
Controllers:
Containers:
myfrontend:
Container ID:
Image: aosqe/hello-openshift
Image ID:
Port: 80/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts:
/tmp from aws (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-g3b9l (ro)
Environment Variables:
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
aws:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ebs90
ReadOnly: false
default-token-g3b9l:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-g3b9l
QoS Class: BestEffort
Tolerations:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

55m 55m 1 {default-scheduler } Normal Scheduled Successfully assigned mypod90 to ip-172-18-9-70.ec2.internal
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdik) for parameter device is invalid. /dev/xvdik is not a valid EBS device name.
status code: 400, request id: 5e93cc61-33b6-4e53-9c17-438e18753ba6
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdjc) for parameter device is invalid. /dev/xvdjc is not a valid EBS device name.
status code: 400, request id: 0d1bd5c1-ce6e-4f21-bd32-93f7740a3f78
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdkg) for parameter device is invalid. /dev/xvdkg is not a valid EBS device name.
status code: 400, request id: 13884d3c-6d31-4696-91f2-18b1ae10276f
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdlv) for parameter device is invalid. /dev/xvdlv is not a valid EBS device name.
status code: 400, request id: 07dc85bc-e0a2-42c5-b415-c60d46714f43
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdnd) for parameter device is invalid. /dev/xvdnd is not a valid EBS device name.
status code: 400, request id: 0eb5528c-70c0-4ba0-b317-a55a309236a4
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdpa) for parameter device is invalid. /dev/xvdpa is not a valid EBS device name.
status code: 400, request id: 0f415cbe-2fe3-4684-8d0c-b74c41df1b3b
55m 55m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdqy) for parameter device is invalid. /dev/xvdqy is not a valid EBS device name.
status code: 400, request id: da6b1626-c42e-47c7-92fb-298e25e4f9ce
54m 54m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvdsw) for parameter device is invalid. /dev/xvdsw is not a valid EBS device name.
status code: 400, request id: 42b34100-7406-4bc6-a049-bc42d6ca9c03
53m 53m 1 {attachdetach } Warning FailedMount Failed to attach volume "ebs33" on node "ip-172-18-9-70.ec2.internal" with: Error attaching EBS volume "vol-03c758a43bd51581c" to instance "i-072d9328131bcd9cd": InvalidParameterValue: Value (/dev/xvduu) for parameter device is invalid. /dev/xvduu is not a valid EBS device name.
status code: 400, request id: 4898ba14-a14f-4aff-88f9-43f9c8d55ab1
51m 1m 26 {attachdetach } Warning FailedMount (events with common reason combined)
53m 34s 25 {kubelet ip-172-18-9-70.ec2.internal} Warning FailedMount Unable to mount volumes for pod "mypod90_default(845be9a8-f28f-11e6-9fb2-0ecb7d4ecce4)": timeout expired waiting for volumes to attach/mount for pod "default"/"mypod90". list of unattached/unmounted volumes=[aws]
53m 34s 25 {kubelet ip-172-18-9-70.ec2.internal} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mypod90". list of unattached/unmounted volumes=[aws]

Anything else we need to know:

[gnufied]

This is a problem with new device allocator for AWS. The naming scheme doesn't support those names and actual list of volumes attachable on aws can't be more than 52 - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html

I am working on pushing a fix.

[gnufied]

The naming scheme of AWS does not allow mounting more than 52 volumes on a node, so we can't really change that easily but we have a genuine bug here - as devices are attached and detached even if number of attached volumes is much smaller than 52 - people will run into this bug because code will start creating invalid device names.

[jsafrane]

I'd swear that xvduu was valid when I tested it couple of years ago. It seems that AWS is more strict now :-(.