[k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/2966/

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc420e3c1b0>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gce-etcd3 274
ci-kubernetes-e2e-gci-gce 143
ci-kubernetes-e2e-gci-gke-multizone 299
ci-kubernetes-e2e-gci-gke 197
ci-kubernetes-soak-gke-gci-test 16

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc420f64380>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[saad-ali]

@jingxu97 is this the same kubectl exec issue? If so can you label this as non-blocker for the 1.5 release?

CC @kubernetes/sig-storage

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gke-multizone 457 533

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc4206a4190>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[rootfs]

similar to #37056
probably the same problem with overlayfs

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gce-etcd3 711
ci-kubernetes-e2e-gci-gke-multizone 753

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc420a707d0>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[saad-ali]

Most likely the same issue as #37056 (comment) Marking as non-release blocking

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gce-kubenet 889
ci-kubernetes-e2e-gci-gce-proto 1330
ci-kubernetes-e2e-gci-gce-release-1.5 824
ci-kubernetes-e2e-gci-gke-multizone 1209
ci-kubernetes-e2e-gci-gke-release-1.5 1009 1150
ci-kubernetes-e2e-gci-gke 764 1062

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc420bbeeb0>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/1361/

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
Expected error:
    <*errors.errorString | 0xc4203d1600>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:67

[dims]

@jingxu97 Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1146/

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
Expected error:
    <*errors.errorString | 0xc4203c16a0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:68

[jingxu97]

The last two failure are caused by the error-- connection timed out when nfs clients tried to mount to nfs server.

E1209 21:13:39.049648 1428 mount_linux.go:119] Mount failed: exit status 32
Mounting command: /home/kubernetes/bin/mounter
Mounting arguments: 10.180.2.120:/ /var/lib/kubelet/pods/0725c411-be54-11e6-854e-42010af00002/volumes/kubernetes.io~nfs/nfs-volume nfs [ro]
Output: Running mount using a rkt fly container
run: group "rkt" not found, will use default gid when rendering images
mount.nfs: Connection timed out

The nfs server and client pods seems are running fine. So far no other clue is found about the root cause. Will continue monitor this issue.

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gce-proto 1481
ci-kubernetes-e2e-gci-gce-release-1.5 1269
ci-kubernetes-e2e-gci-gke-release-1.5 1504

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
failed: finding the contents of the mounted file.
Expected error:
    <*errors.errorString | 0xc420fe03c0>: {
        s: "Failed to find \"Hello from NFS!\", last result: \"\"",
    }
    Failed to find "Hello from NFS!", last result: ""
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

[k8s-github-robot]

Builds:
ci-kubernetes-e2e-gci-gce 1208 1228 1234 1252

Failed: [k8s.io] GCP Volumes [k8s.io] NFSv4 should be mountable for NFSv4 {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393
Expected error:
    <*errors.errorString | 0xc4203a4610>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:68

[jeffvance]

@calebamiles @copejon and I have been looking at this issue and cannot find any current nfsv4 failures. Can you point us to the failing tests and we will take a look?

[saad-ali]

Closing old non-recurring flake