List of skipped E2E cases

[nilo19]

List of skipped cases:

• cloud-provider-azure-serial

1. should unmount
2. When kubelet restarts
3. regular resource usage tracking resource tracking for
4. validates MaxPods limit number of pods that are allowed to run
5. DNS configMap nameserver
6. Checks that the node becomes unreachable
7. Should test that pv written before kubelet restart is readable after restart

• cloud-provider-azure-multiple-zones

1. Network should set TCP CLOSE_WAIT timeout
2. Mount propagation should propagate mounts to the host
3. PodSecurityPolicy
4. PVC Protection Verify
5. should provide basic identity
6. should adopt matching orphans and release
7. should not deadlock when a pod's predecessor fails
8. should perform rolling updates and roll backs of template modifications with PVCs
9. should perform rolling updates and roll backs of template modifications
10. Services should be able to create a functioning NodePort service
11. volumeMode should not mount / map unused volumes in a pod

[feiskyer]

@ritazh Could you help to check whether some of them could be added into aks-engine, so that we could re-enable those tests?

[feiskyer]

/assign @ritazh

[ritazh]

/assign @chewong

[k8s-ci-robot]

@ritazh: GitHub didn't allow me to assign the following users: chewong.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @chewong

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chewong]

@ritazh @feiskyer I ran all the tests mentioned above and here is the result:

Job: ci-cloud-provider-azure-serial

Test / Test Regex	SIG	Status	Error Messages
should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly]	Storage	Failed	Require agent node to have external IP address so the test can ssh into it
should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly]	Storage	Failed	Require agent node to have external IP address so the test can ssh into it
When kubelet restarts	Storage	Failed	PersistentVolumeClaims [pvc-xxxxxx] not all in phase Bound within 5m0s
regular resource usage tracking - resource tracking for 100 pods per node	Node	Failed	Sep 5 11:58:47.568: Memory usage exceeding limits: node k8s-agentpool1-30531301-vmss000000: container "runtime": expected RSS memory (MB) < 367001600; got 1337942016 node k8s-agentpool1-30531301-vmss000001: container "runtime": expected RSS memory (MB) < 367001600; got 1301487616
resource tracking for 0 pods per node	Node	Failed	Sep 5 12:22:00.106: Memory usage exceeding limits: node k8s-agentpool1-30531301-vmss000001: container "runtime": expected RSS memory (MB) < 131072000; got 231104512 node k8s-master-30531301-0: container "runtime": expected RSS memory (MB) < 131072000; got 153079808 node k8s-agentpool1-30531301-vmss000000: container "runtime": expected RSS memory (MB) < 131072000; got 287764480
validates MaxPods limit number of pods that are allowed to run [Slow]	Scheduling	Failed	[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow] /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:749
DNS configMap nameserver [IPv4]	Network	Pass
DNS configMap nameserver [Feature:Networking-IPv6] [LinuxOnly]	Network	Pass
Checks that the node becomes unreachable	Scheduling	Failed	Sep 5 13:30:55.169: This test is not supported for provider azure and should be disabled
Should test that pv written before kubelet restart is readable after restart	Storage	Failed	Require agent node to have external IP address so the test can ssh into it

Job: ci-cloud-provider-azure-multiple-zones

Test	SIG	Status	Error Messages
Network should set TCP CLOSE_WAIT timeout	Network	Failed	Require agent node to have external IP address so the test can ssh into it
Mount propagation should propagate mounts to the host	Network	Failed	Require agent node to have external IP address so the test can ssh into it
PodSecurityPolicy	Auth	Failed	[Fail] [sig-auth] PodSecurityPolicy [It] should enforce the restricted policy.PodSecurityPolicy /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1357 [Fail] [sig-auth] PodSecurityPolicy [It] should allow pods under the privileged policy.PodSecurityPolicy /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/auth/pod_security_policy.go:120 [Fail] [sig-auth] PodSecurityPolicy [It] should forbid pod creation when no PSP is available /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1357
PVC Protection Verify	Storage	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
should provide basic identity	Apps	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
should adopt matching orphans and release	Apps	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
should not deadlock when a pod's predecessor fails	Apps	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
should perform rolling updates and roll backs of template modifications with PVCs	Apps	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
should perform rolling updates and roll backs of template modifications	Apps	Failed	Failed to create test pod. Pod scheduling error: pod has unbound immediate PersistentVolumeClaims
Services should be able to create a functioning NodePort service	Network	Passed
volumeMode should not mount / map unused volumes in a pod	Storage	Skipped	All of the tests are skipped because Driver local doesn't support DynamicPV or they areonly supported for providers [<other cloud providers>]] (not azure)

[feiskyer]

@chewong Most of them are because of configuration issues. Could you help to enable those in aks-engine?

[chewong]

/assign

[chewong]

Here are the update test results:

Job: cloud-provider-azure-serial

Test Regex	SIG	Status	Note
should unmount	Storage	Passed	Enabled in cloud-provider-azure-serial
When kubelet restarts	Storage	Passed	Enabled in cloud-provider-azure-serial
regular resource usage tracking resource tracking for	Node	Failed	Sep 5 11:58:47.568: Memory usage exceeding limits: node k8s-agentpool1-30531301-vmss000000: container "runtime": expected RSS memory (MB) < 367001600; got 1337942016 node k8s-agentpool1-30531301-vmss000001: container "runtime": expected RSS memory (MB) < 367001600; got 1301487616
validates MaxPods limit number of pods that are allowed to run	Scheduling	Failed	[Fail] [sig-scheduling] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow] /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:749
DNS configMap nameserver	Network	Passed	Enabled in cloud-provider-azure-serial
Checks that the node becomes unreachable	Scheduling	Failed	This test is not supported for provider azure and should be disabled
Should test that pv written before kubelet restart is readable after restart	Storage	Passed	Enabled in cloud-provider-azure-serial

Job: cloud-provider-azure-multiple-zones

Test	SIG	Status	Error Messages
Network should set TCP CLOSE_WAIT timeout	Network	Failed	Require vmss nodes to have external IP address so the test can ssh into it. Was not able to enable enableVMSSNodePublicIP in linux-vmss-multi-zones.json due to the following error during ARM deployment: NicWithPublicIpCannotReferencePoolWithOutboundRule: OutboundRules for VMs with public IpConfigurations (instance level publicIPs) /subscriptions/.../resourceGroups/.../providers/Microsoft.Network/networkInterfaces/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool-95854893-vmss/virtualMachines/0/networkInterfaces/k8s-agentpool-95854893-vmss/ipConfigurations/ipconfig1 are not supported
Mount propagation should propagate mounts to the host	Network	Failed	Please see above.
PodSecurityPolicy	Auth	Failed	[Fail] [sig-auth] PodSecurityPolicy [It] should enforce the restricted policy.PodSecurityPolicy /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1357 [Fail] [sig-auth] PodSecurityPolicy [It] should allow pods under the privileged policy.PodSecurityPolicy /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/auth/pod_security_policy.go:120 [Fail] [sig-auth] PodSecurityPolicy [It] should forbid pod creation when no PSP is available /Users/ernestwong/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1357
PVC Protection Verify	Storage	Passed	Enabled in cloud-provider-azure-multiple-zones
should provide basic identity	Apps	Passed	Enabled in cloud-provider-azure-multiple-zones
should adopt matching orphans and release	Apps	Passed	Enabled in cloud-provider-azure-multiple-zones
should not deadlock when a pod's predecessor fails	Apps	Passed	Enabled in cloud-provider-azure-multiple-zones
should perform rolling updates and roll backs of template modifications with PVCs	Apps	Passed	Enabled in cloud-provider-azure-multiple-zones
should perform rolling updates and roll backs of template modifications	Apps	Passed	Enabled in cloud-provider-azure-multiple-zones
Services should be able to create a functioning NodePort service	Network	Passed	Enabled in cloud-provider-azure-multiple-zones
volumeMode should not mount / map unused volumes in a pod	Storage	Passed	Enabled in cloud-provider-azure-multiple-zones

[ritazh]

I am working on a PR on aks-engine where Azure Disk and Azure File CSI Drivers will be automatically deployed when cloud controller manager is enabled (Azure/aks-engine#1916). After that is merged, I can enable enableVMSSNodePublicIP in linux-vmss-serial.json, and bump the aks-engine version in cloud-provider-azure-config.yaml so we can start enabling the passed tests.

This is a great start! These should help with Network should set TCP CLOSE_WAIT timeout and Mount propagation should propagate mounts to the host.

[chewong]

I can start enabling these tests after Azure/aks-engine#2166 is included in the latest aks-engine release.

[chewong]

With Azure Disk CSI Driver v0.5.0 released, I will open a PR in aks-engine to upgrade the CSI driver version so we can start enabling most of the skipped tests in ci-cloud-provider-azure-multiple-zones.

[feiskyer]

@chewong thanks a lot for the work

[chewong]

kubernetes/test-infra#15981 should re-enable most of the tests described above with aks-engine v0.46.0.

On a separate note, I have compiled a list of skipped tests for all Azure-related presubmit & periodics jobs and noticed that we are skipping a lot of unnecessary test cases because either 1) they started passing again, 2) they are already being skipped with --ginkgo.focus flag, or 3) test cases don't exist anymore. Here is the list and I've crossed out what we should exclude. @feiskyer WDYT?

pull-cloud-provider-azure-e2e

• [Serial]
• [sig-node] Mount propagation
• [sig-network] Network should set TCP CLOSE_WAIT timeout
• [sig-storage] PersistentVolumes-local Stress with local volume provisioner [Serial] should use be able to process many pods and reuse local volumes
• should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
• should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
• [sig-network] Services should be able to create a functioning NodePort service
• [sig-scheduling] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow]
• [sig-storage] Dynamic Provisioning DynamicProvisioner Default should create and delete default persistent volumes [Slow]
• [sig-storage] Dynamic Provisioning DynamicProvisioner should provision storage with different parameters
• [sig-storage] Dynamic Provisioning DynamicProvisioner should test that deleting a claim before the volume is provisioned deletes the volume.
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod.s predecessor fails
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity
• [sig-storage] PersistentVolumes Default StorageClass pods that use multiple volumes should be reschedulable
• [sig-storage] PVC Protection
• [sig-storage] Dynamic Provisioning [k8s.io] GlusterDynamicProvisioner
• [sig-storage] Volumes Azure Disk should be mountable [Slow]
• [sig-apps] Network Partition [Disruptive] [Slow]
• [sig-network] DNS configMap
• [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking resource tracking for 0 pods per node
• [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking resource tracking for 100 pods per node
• Horizontal pod autoscaling (scale resource: CPU)
• [sig-storage] Dynamic Provisioning DynamicProvisioner External should let an external dynamic provisioner create and delete persistent volumes [Slow]
• ESIPP
• [sig-network] Services should preserve source pod IP for traffic thru service cluster IP
• In-tree Volumes
• PersistentVolumes-local
• CSI Volumes
• should write entries to /etc/hosts
• [Slow]
• [Serial]
• [Disruptive]
• [Flaky]
• [Feature:.+]
• [NodeFeature:.+]
• [sig-cli] Kubectl client [k8s.io] Simple pod should support exec using resource / name

cloud-provider-azure-conformance

• [Serial]
• [Slow]

cloud-provider-azure-slow

• [Serial]
• [Disruptive]
• [Flaky]
• [Feature:.+]

cloud-provider-azure-serial

• [Flaky]
• [Feature:.+]
• regular resource usage tracking resource tracking for (rss memory setting varies)
• validates MaxPods limit number of pods that are allowed to run (IsMasterNode function does not apply)
• Checks that the node becomes unreachable (test case not available anymore)

cloud-provider-azure-conformance-vmss

• [Serial]
• [Slow]

cloud-provider-azure-slow-vmss

• [Serial]
• [Disruptive]
• [Flaky]
• [Feature:.+]

cloud-provider-azure-multiple-zones

• [Slow]
• [Serial]
• [Disruptive]
• [Flaky]
• [Feature:.+]
• Delete Grace Period (marked as flaky in [Flaky Test] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance] kubernetes/kubernetes#85762)
• runs ReplicaSets to verify preemption running path (marked as flaky in [Flaky Test] [sig-scheduling] PreemptionExecutionPath runs ReplicaSets to verify preemption running path kubernetes/kubernetes#86068)
• client-go should negotiate (failed due to bad test cases - fixed by improves watch and report e2e  kubernetes/kubernetes#86354)
• should contain custom columns for each resource (failed due to CSINodeInfo / CSIDriver CRDs being installed)
• Network should set TCP CLOSE_WAIT timeout (requires public IP address for worker nodes but it's not possible for vmss + zones cluster)
• PodSecurityPolicy (failed for some unknown reason)

pull-kubernetes-e2e-aks-engine-azure

• [sig-node] Mount propagation
• [sig-network] Network should set TCP CLOSE_WAIT timeout
• [sig-storage] PersistentVolumes-local Stress with local volume provisioner [Serial] should use be able to process many pods and reuse local volumes
• should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]
• should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]
• [sig-network] Services should be able to create a functioning NodePort service
• [sig-scheduling] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow]
• [sig-storage] Dynamic Provisioning DynamicProvisioner Default should create and delete default persistent volumes [Slow]
• [sig-storage] Dynamic Provisioning DynamicProvisioner should provision storage with different parameters
• [sig-storage] Dynamic Provisioning DynamicProvisioner should test that deleting a claim before the volume is provisioned deletes the volume.
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod.s predecessor fails
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs
• [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity
• [sig-storage] PersistentVolumes Default StorageClass pods that use multiple volumes should be reschedulable
• [sig-storage] PVC Protection
• [sig-storage] Dynamic Provisioning [k8s.io] GlusterDynamicProvisioner
• [sig-storage] Volumes Azure Disk should be mountable [Slow]
• [sig-apps] Network Partition [Disruptive] [Slow]
• [sig-network] DNS configMap
• [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking resource tracking for 0 pods per node
• [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] regular resource usage tracking resource tracking for 100 pods per node
• Horizontal pod autoscaling (scale resource: CPU)
• [sig-storage] Dynamic Provisioning DynamicProvisioner External should let an external dynamic provisioner create and delete persistent volumes [Slow]
• ESIPP
• [sig-network] Services should preserve source pod IP for traffic thru service cluster IP
• In-tree Volumes
• PersistentVolumes-local
• CSI Volumes
• should write entries to /etc/hosts
• [Slow]
• [Serial]
• [Disruptive]
• [Flaky]
• [Feature:.+]
• [NodeFeature:.+]

[chewong]

Most of the skipped tests are now enabled (see #224 (comment)). Let's close this issue for now.
/close

[k8s-ci-robot]

@chewong: Closing this issue.

In response to this:

Most of the skipped tests are now enabled (see #224 (comment)). Let's close this issue for now.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.