e2e manifests #69868

pohly · 2018-10-16T10:49:17Z

/kind cleanup

What this PR does / why we need it:

For CSI testing we want to get away from having to replicate .yaml files in code. Now the CSI drivers and their RBAC rules get created from .yaml files.

Special notes for your reviewer:

My proposal is to review and potentially merge this PR first, then create the rbac.yaml in the individual repos at the URLs linked to here in the readmes.

Release note:

- The builtin system:csi-external-provisioner and system:csi-external-attacher cluster roles
  are deprecated and will not be updated for deployments of CSI sidecar container versions >= 0.4.
  Deployments with the current CSI sidecar containers have to provide their own RBAC
  definitions. The reason is that the rules depend on how the sidecar containers are used,
  which is defined by the deployment.

/sig storage
/sig testing
/cc @msau42

pohly · 2018-10-16T17:00:28Z

@msau42 I went ahead and also rewrote the GCE CSI test, which allowed me to remove a lot of code from csi_objects.go. Can you perhaps check that I didn't break that test? Or is it running in the Kubernetes CI?

pohly · 2018-10-16T17:07:29Z

Note that the csiTestDriver implementation for hostpath and GCE now use almost the same code. The real difference is in the set of .yaml files and in the names of the containers. It seems very feasible now to just have one struct which defines these settings and then fill in that struct from .yaml files, either shipped with the test suite or provided by a user.

Only createGCESecrets() may still need some special handling.

pohly · 2018-10-17T14:50:31Z

/retest

davidz627 · 2018-10-17T18:20:24Z

/cc

pohly · 2018-10-17T18:24:51Z

I found that the gcePD CSI driver test case does run in the CI... and that it currently hangs. I've added more debug output, so perhaps soon I'll know why. I got stuck trying to run it locally because it wasn't obvious how to set up the required service account secret.

pohly · 2018-10-17T19:31:15Z

Some help would be welcome. The gcePD test failed in https://prow.k8s.io/log?job=pull-kubernetes-e2e-gce&id=56220 again because of permission issues. I don't understand what the error messages are trying to tell me, the service account setup looks sane to me:

I1017 18:50:42.646] Oct 17 18:45:34.524: INFO: creating ServiceAccount:
I1017 18:50:42.646] {
I1017 18:50:42.646]   "kind": "ServiceAccount",
I1017 18:50:42.646]   "apiVersion": "v1",
I1017 18:50:42.646]   "metadata": {
I1017 18:50:42.646]     "name": "csi-driver-registrar",
I1017 18:50:42.646]     "namespace": "e2e-tests-csi-volumes-nh7pq",
I1017 18:50:42.646]     "creationTimestamp": null
I1017 18:50:42.646]   }
I1017 18:50:42.646] }
...
I1017 18:50:42.651] Oct 17 18:45:34.531: INFO: creating ClusterRole
I1017 18:50:42.651] :{
I1017 18:50:42.651]   "kind": "ClusterRole",
I1017 18:50:42.651]   "apiVersion": "rbac.authorization.k8s.io/v1",
I1017 18:50:42.651]   "metadata": {
I1017 18:50:42.651]     "name": "driver-registrar-runner-e2e-tests-csi-volumes-nh7pq",
I1017 18:50:42.651]     "creationTimestamp": null
I1017 18:50:42.652]   },
I1017 18:50:42.652]   "rules": [
I1017 18:50:42.652]     {
I1017 18:50:42.652]       "verbs": [
I1017 18:50:42.652]         "get",
I1017 18:50:42.652]         "list",
I1017 18:50:42.652]         "watch",
I1017 18:50:42.652]         "create",
I1017 18:50:42.652]         "update",
I1017 18:50:42.652]         "patch"
I1017 18:50:42.652]       ],
I1017 18:50:42.653]       "apiGroups": [
I1017 18:50:42.653]         ""
I1017 18:50:42.653]       ],
I1017 18:50:42.653]       "resources": [
I1017 18:50:42.653]         "events"
I1017 18:50:42.653]       ]
I1017 18:50:42.653]     },
I1017 18:50:42.653]     {
I1017 18:50:42.653]       "verbs": [
I1017 18:50:42.653]         "get",
I1017 18:50:42.654]         "update",
I1017 18:50:42.654]         "patch"
I1017 18:50:42.654]       ],
I1017 18:50:42.654]       "apiGroups": [
I1017 18:50:42.654]         ""
I1017 18:50:42.654]       ],
I1017 18:50:42.654]       "resources": [
I1017 18:50:42.654]         "nodes"
I1017 18:50:42.654]       ]
I1017 18:50:42.654]     }
I1017 18:50:42.654]   ]
I1017 18:50:42.655] }
...
I1017 18:50:42.657] Oct 17 18:45:34.539: INFO: creating ClusterRoleBinding:
I1017 18:50:42.657] {
I1017 18:50:42.657]   "kind": "ClusterRoleBinding",
I1017 18:50:42.657]   "apiVersion": "rbac.authorization.k8s.io/v1",
I1017 18:50:42.657]   "metadata": {
I1017 18:50:42.657]     "name": "csi-driver-registrar-role-e2e-tests-csi-volumes-nh7pq",
I1017 18:50:42.658]     "creationTimestamp": null
I1017 18:50:42.658]   },
I1017 18:50:42.658]   "subjects": [
I1017 18:50:42.658]     {
I1017 18:50:42.658]       "kind": "ServiceAccount",
I1017 18:50:42.658]       "name": "csi-driver-registrar",
I1017 18:50:42.658]       "namespace": "e2e-tests-csi-volumes-nh7pq"
I1017 18:50:42.658]     }
I1017 18:50:42.658]   ],
I1017 18:50:42.658]   "roleRef": {
I1017 18:50:42.659]     "apiGroup": "rbac.authorization.k8s.io",
I1017 18:50:42.659]     "kind": "ClusterRole",
I1017 18:50:42.659]     "name": "driver-registrar-runner-e2e-tests-csi-volumes-nh7pq"
I1017 18:50:42.659]   }
I1017 18:50:42.659] }
...
I1017 18:45:34.534] Oct 17 18:45:27.723: INFO: At 2018-10-17 18:40:25 +0000 UTC - event for csi-gce-node: {daemonset-controller } FailedCreate: Error creating: pods "csi-gce-node-" is forbidden: unable to validate against any pod security policy: []
I1017 18:45:34.535] Oct 17 18:45:27.723: INFO: At 2018-10-17 18:40:26 +0000 UTC - event for csi-gce-controller: {statefulset-controller } FailedCreate: create Pod csi-gce-controller-0 in StatefulSet csi-gce-controller failed error: pods "csi-gce-controller-0" is forbidden: unable to validate against any pod security policy: []

@liggitt @davidz627

Generated via hack/update-bazel.sh.

pohly · 2018-10-26T22:45:46Z

@msau let me come back to the offer of not renaming the gcePD driver. I had that working before the driver update to gcp-compute-persistent-disk-csi-driver:v0.1.0.alpha, but now I can't get it to work again and my possibilities to debug that are rather limited and slow.

Therefore the latest incarnation of this PR deploys the gcePD driver without renaming, as in current Kubernetes master. If that works, please merge. I'm not going to wait for clean test results myself anymore today, though.

I left in the version update - killing two birds with one stone, basically. Otherwise we would have two conflicting PRs.

/hold cancel

msau42 · 2018-10-26T23:03:08Z

test/e2e/testing-manifests/storage-csi/gce-pd/node_ds.yaml

@@ -16,7 +16,7 @@ spec:
      containers:
        - name: csi-driver-registrar
          imagePullPolicy: Always
-          image: quay.io/k8scsi/driver-registrar:v0.3.0
+          image: quay.io/k8scsi/driver-registrar:v0.4.1


Hm I wonder if the alpha pd driver is going to have issues with the latest sidecars.

Also, unfortunately we haven't published public images of the beta gce pd driver yet. So I think it's best to just leave PD driver image versions the way it was before, and we will update the versions when we're ready.

Also I think we need to add the [Serial] tag to pd so that tests won't be run it parallel.

The test has passed, so it looks like the alpha driver is compatible with the latest sidecar drivers. Do you want to revert that part nonetheless?

Please lets add the [Serial] tag as part of PR #68025 /cc @mkimuram

It has been working so far and this PR doesn't make it worse. From a practical perspective,
I don't see a good way to add it to the current test because of the way how it loops over the different drivers, and that part will be changed anyway:

for driverName, initCSIDriver := range csiTestDrivers { curDriverName := driverName curInitCSIDriver := initCSIDriver Context(fmt.Sprintf("CSI plugin test using CSI driver: %s", curDriverName), func() {

If its passing then it's fine to leave it. It's not a configuration that's officially been tested/supported by us, but I will update it later once we get the latest driver image published.

@msau42 so is this PR ready for merging now?

pohly · 2018-10-27T07:08:24Z

/test pull-kubernetes-integration

pohly · 2018-10-27T07:18:28Z

/test pull-kubernetes-integration

msau42 · 2018-10-27T14:48:13Z

test/e2e/testing-manifests/storage-csi/hostpath/hostpath/csi-hostpath-attacher.yaml

+          image: quay.io/k8scsi/csi-attacher:v0.4.0
+          args:
+            - --v=5
+            - --csi-address=$(ADDRESS)


do you want to simplify the hostpath specs too?

Yes, but let's do that in a separate PR and then also coordinate the update with the original .yaml file in kubernetes-csi/docs.

I've filed kubernetes-csi/docs#68 for this.

msau42 · 2018-10-27T14:58:37Z

test/e2e/testing-manifests/storage-csi/gce-pd/node_ds.yaml

@@ -16,7 +16,7 @@ spec:
      containers:
        - name: csi-driver-registrar
          imagePullPolicy: Always
-          image: quay.io/k8scsi/driver-registrar:v0.3.0
+          image: quay.io/k8scsi/driver-registrar:v0.4.1


If its passing then it's fine to leave it. It's not a configuration that's officially been tested/supported by us, but I will update it later once we get the latest driver image published.

AishSundar · 2018-10-29T00:30:20Z

@pohly is this planned to be merged in 1.13 timeframe? if so please give this a priority and milestone label accordingly. @nikopen in case you need him to apply the milestone label

pohly · 2018-10-29T07:37:13Z

Yes, this is for 1.13.

/milestone v1.13
/priority important-soon

k8s-ci-robot · 2018-10-29T07:37:13Z

@pohly: You must be a member of the kubernetes/kubernetes-milestone-maintainers github team to set the milestone.

In response to this:

Yes, this is for 1.13.

/milestone v1.13
/priority important-soon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pohly · 2018-10-29T07:38:09Z

@nikopen can you please add the v1.13 milestone label to this PR?

nikopen · 2018-10-29T08:50:57Z

/milestone v1.13

test/e2e/framework/create.go

msau42 · 2018-10-29T15:53:01Z

/lgtm

jsafrane · 2018-10-29T16:11:19Z

/assign @saad-ali
for approval

saad-ali · 2018-10-29T22:14:04Z

/approve

k8s-ci-robot · 2018-10-29T22:14:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pohly, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [saad-ali]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gnufied · 2018-10-30T18:37:36Z

test/e2e/framework/create.go

+// - only the latest stable API version for each item is supported
+func (f *Framework) PatchItems(items ...interface{}) error {
+	for _, item := range items {
+		Logf("patching original content of %T:\n%s", item, PrettyPrint(item))


Printing all of the YAMLs is far too noisy - #70448

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Oct 16, 2018

k8s-ci-robot requested a review from msau42 October 16, 2018 10:49

pohly force-pushed the e2e-manifests branch from dda0217 to bbbf7cc Compare October 16, 2018 10:50

pohly mentioned this pull request Oct 16, 2018

Make csi drivers and in-tree drivers share e2e tests #68025

Merged

pohly force-pushed the e2e-manifests branch 2 times, most recently from 7c97bcd to 073e7fa Compare October 16, 2018 16:54

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Oct 16, 2018

pohly force-pushed the e2e-manifests branch 6 times, most recently from 7695f62 to ea44b83 Compare October 17, 2018 14:07

pohly force-pushed the e2e-manifests branch 2 times, most recently from f66c51b to 794000d Compare October 17, 2018 18:04

k8s-ci-robot requested a review from davidz627 October 17, 2018 18:20

pohly force-pushed the e2e-manifests branch from 794000d to b9a48e2 Compare October 17, 2018 19:32

pohly force-pushed the e2e-manifests branch from 4e37314 to 99062b4 Compare October 26, 2018 20:09

pohly added 2 commits October 27, 2018 00:39

e2e CSI: bump all containers to v0.4.1

a56c518

e2e: update bazel BUILD files

248df4a

Generated via hack/update-bazel.sh.

pohly force-pushed the e2e-manifests branch from 99062b4 to 248df4a Compare October 26, 2018 22:40

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 26, 2018

msau42 reviewed Oct 26, 2018

View reviewed changes

msau42 reviewed Oct 27, 2018

View reviewed changes

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Oct 29, 2018

k8s-ci-robot added this to the v1.13 milestone Oct 29, 2018

jsafrane reviewed Oct 29, 2018

View reviewed changes

test/e2e/framework/create.go Show resolved Hide resolved

k8s-ci-robot assigned msau42 Oct 29, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 29, 2018

k8s-ci-robot assigned saad-ali Oct 29, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2018

k8s-ci-robot merged commit 2ecc71d into kubernetes:master Oct 30, 2018

gnufied mentioned this pull request Oct 30, 2018

Test output of installing rbac policies is too noisy #70448

Closed

gnufied reviewed Oct 30, 2018

View reviewed changes

pohly mentioned this pull request Nov 5, 2018

Move CSI RBAC Role definitions from kubernetes/kubernetes to external repos #69379

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e manifests #69868

e2e manifests #69868

pohly commented Oct 16, 2018 •

edited

pohly commented Oct 16, 2018

pohly commented Oct 16, 2018

pohly commented Oct 17, 2018

davidz627 commented Oct 17, 2018

pohly commented Oct 17, 2018

pohly commented Oct 17, 2018

pohly commented Oct 26, 2018

msau42 Oct 26, 2018

davidz627 Oct 26, 2018

pohly Oct 27, 2018

msau42 Oct 27, 2018

pohly Oct 27, 2018

pohly commented Oct 27, 2018

pohly commented Oct 27, 2018

msau42 Oct 27, 2018

pohly Oct 27, 2018

msau42 Oct 27, 2018

AishSundar commented Oct 29, 2018

pohly commented Oct 29, 2018

k8s-ci-robot commented Oct 29, 2018

pohly commented Oct 29, 2018

nikopen commented Oct 29, 2018

msau42 commented Oct 29, 2018

jsafrane commented Oct 29, 2018

saad-ali commented Oct 29, 2018

k8s-ci-robot commented Oct 29, 2018

gnufied Oct 30, 2018

e2e manifests #69868

e2e manifests #69868

Conversation

pohly commented Oct 16, 2018 • edited

pohly commented Oct 16, 2018

pohly commented Oct 16, 2018

pohly commented Oct 17, 2018

davidz627 commented Oct 17, 2018

pohly commented Oct 17, 2018

pohly commented Oct 17, 2018

pohly commented Oct 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Oct 27, 2018

pohly commented Oct 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AishSundar commented Oct 29, 2018

pohly commented Oct 29, 2018

k8s-ci-robot commented Oct 29, 2018

pohly commented Oct 29, 2018

nikopen commented Oct 29, 2018

msau42 commented Oct 29, 2018

jsafrane commented Oct 29, 2018

saad-ali commented Oct 29, 2018

k8s-ci-robot commented Oct 29, 2018

Choose a reason for hiding this comment

pohly commented Oct 16, 2018 •

edited