Failling Test: [sig-storage] Flexvolume expansion tests are flaky

[gnufied]

This is a child issue of #71434

For example - https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/21996

/sig storage

@aniket-s-kulkarni will you be able to look into this?

[AishSundar]

/milestone v1.13
/kind failing-test
/priority critical-urgent

I am addind this to 1.13 milestone as critical-urgent until investigated and triaged otherwise. The Code freeze deadline for the fix is EOD tomorrow, 11/28

/cc @msau42 @jberkus @mariantalla @mortent

[aniket-s-kulkarni]

@gnufied yes I will look at it. At first glance it seems to be timing

This is what the test does

		By("Waiting for file system resize to finish")
		pvc, err = waitForFSResize(pvc, c)
		Expect(err).NotTo(HaveOccurred(), "while waiting for fs resize to finish")

		pvcConditions := pvc.Status.Conditions
		Expect(len(pvcConditions)).To(Equal(0), "pvc should not have conditions")

The waitForFSResize actually checks for .spec.resources.requests and .spec.status.capacity to reflect a successful resize. That one did pass. But the very next step is to expect the PVC conditions to be cleared away didn't

Unfortunately we don't log what condition was there on the PVC, so unsure what was left uncleared. The kubelet updates .spec.status.capacity and clears the conditions in the same patch command so I wouldn't expect the test check getting in between those two.

[gnufied]

@aniket-s-kulkarni I am wondering if kubelet logs have some clues about why condition update failed. I do agree that if pvc.Status.Capacity had correct value then condition should be removed too but apparently something happened.

[gnufied]

I think the root cause of this issue is - same PVC is being considered twice for resizing while fs resizing was pending on the node. The flow is like this:

1. User edits the PVC and requests more size (this add pvc resize progress condition)
2. flexvolume plugin controller side resize is instant and hence it returns success immediately and pv size is updated and PVC gets fs resize condition.
3. On node, kubelet resizes the volume and removes the condition.
4. Now while we wait for resize on the node, same PVC is being considerd again for resizing by the expand-controller loop, because pvc.status.capacity < pvc.spec.capacity . The actual operation generator code actually will reject the second resize request - https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go#L1266 because it considers PV size but what we are seeing is, resize-in-progress condition is being at least added twice.

Here is how we know that volume was considered twice for resize:

1126 13:08:09.979859       1 volume_resize_map.go:115] Adding pvc e2e-tests-mounted-flexvolume-expand-6c7wp/pvc-k7lj2 with Size 6Gi/2Gi for resizing
I1126 13:08:12.090042       1 volume_resize_map.go:115] Adding pvc e2e-tests-mounted-flexvolume-expand-6c7wp/pvc-k7lj2 with Size 6Gi/2Gi for resizing

This flake was not introduced by flexvolume change afaict and existed before. I think timing issue with flex is just triggering it. All resize operation are idempotent and expand controller anyways rejects the resize request by comparing PV size and hence it should not be too bad. In worst case, this WILL cause a condition to be added to the PVC unnecessarily (and that is what is happening here).

I will make a fix for this. But IMO - this should not be release blocker.

[saad-ali]

Thanks for digging in to this @gnufied.

Based on #71434 (comment) we will move this to v1.14 and document it as a known issue for Flex resize.

/milestone v1.14

[AishSundar]

@gnufied @saad-ali is there any perceived user effect if an extra condition gets added to the PVC ? is it just ignored?

[jberkus]

Fix tags:

/kind flake
/remove-kind failing-test

[gnufied]

The condition exists as a messaging to the user about stage of volume resizing, but there aren't any actions taken based on that. So - user will not have any usage issues with PVC or anything, guess just the messaging will be kinda confusing(when resizing operation is over, to end user it will appear as it isn't).

Also - the issue itself is kinda rare. For example, this test flaked only twice between 11/13 and 11/27, so most users will most likely not run into this. But nontheless - I am working on a fix.

[AishSundar]

Thanks @gnufied this is now moved to 1.14

[gnufied]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
This is still failing (it's unclear to me whether it's a flake or a continuous failure from this display) https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=Flexvolume

[gnufied]

Yeah, there is a fix for this available actually - #71611 . I need to sync with @jsafrane and see how we can merge the PR.

[nikopen]

Hey @gnufied , in terms of v1.14 release, how close is the fix for this / what's the current status?

[gnufied]

We missed fixing this in v1.14. We are working on a fix that will land in 1.15.

[spiffxp]

/remove-priority critical-urgent
/priority important-longterm
More representative of how it's actually being treated

[gnufied]

/milestone v1.15

[k8s-ci-robot]

@gnufied: You must be a member of the kubernetes/kubernetes-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@mariantalla: You must be a member of the kubernetes/kubernetes-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gnufied]

This should be fixed now..

/close

[k8s-ci-robot]

@gnufied: Closing this issue.

In response to this:

This should be fixed now..

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.