New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: e2e/storage: testing of external storage drivers #72836

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@pohly
Copy link
Contributor

pohly commented Jan 11, 2019

What type of PR is this?

/kind feature

What this PR does / why we need it:

It is useful to apply the storage testsuite also to "external" (=
out-of-tree) storage drivers. One way of doing that is setting up a
custom E2E test suite, but that's still quite a bit of work.

An easier alternative is to parameterize the Kubernetes e2e.test
binary at runtime so that it instantiates the testsuite for one or
more drivers.

Special notes for your reviewer:

I would prefer to merge this after PR #72434.

Does this PR introduce a user-facing change?:

the test/e2e/e2e.test binary can test arbitrary storage drivers, see the `-storage.testdriver` parameter
e2e/storage: testing of external storage drivers
It is useful to apply the storage testsuite also to "external" (=
out-of-tree) storage drivers. One way of doing that is setting up a
custom E2E test suite, but that's still quite a bit of work.

An easier alternative is to parameterize the Kubernetes e2e.test
binary at runtime so that it instantiates the testsuite for one or
more drivers. Some parameters have to be provided before starting the
test because they define configuration and capabilities of the driver
and its storage backend that cannot be discovered at runtime. This is
done by populating the DriverDefinition with the content of the file
that the new -storage.testdriver parameters points to.

The universal .yaml and .json decoder from Kubernetes is used. It's
flexible, but has some downsides:
- currently ignores unknown fields (see #71589)
- poor error messages when fields have the wrong type

Storage drivers have to be installed in the test cluster before
starting e2e.test. Only tests involving dynamically provisioned
volumes are currently supported.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 11, 2019

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pohly
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: deads2k

If they are not already assigned, you can assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added sig/testing and removed needs-sig labels Jan 11, 2019

@k8s-ci-robot k8s-ci-robot requested a review from msau42 Jan 11, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 11, 2019

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pohly
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: deads2k

If they are not already assigned, you can assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@wongma7

This comment has been minimized.

Copy link
Contributor

wongma7 commented Jan 12, 2019

This feels like it doesn't need to be in-tree...with all the/your efforts refactoring the framework I envisioned the ideal solution would be to develop something analogous to csi-sanity out of tree so that someone has full control over the interface, versioning, etc. of this "shim" between drivers and the existing e2e tests. I think the interface being an arg to e2e.test would be unpleasant to use...I need to checkout kubernetes & do make WHAT=test/e2e/e2e.test, probably building a bunch of unrelated stuff. And what if we someday want to add another args or deprecate one, the versioning of our arg "interface" will be difficult to communicate.

It doesn't mean we can't maintain both though.

Anyway I like very much the general idea of having a generic driver that simply takes yamls so that I don't have to write a TestDriver or any code at all unless I'm doing something complicated, it's exactly what I had in mind also.

@pohly

This comment has been minimized.

Copy link
Contributor

pohly commented Jan 12, 2019

/retest

@pohly

This comment has been minimized.

Copy link
Contributor

pohly commented Jan 12, 2019

@wongma7 I think @msau42 wanted this in-tree because then the e2e.test binary that gets released together with each Kubernetes release (it does get released, right? I'm not sure because I've never used it) can be used as-is. The intended users wouldn't be developers, but rather cluster admins who want to run sanity or conformance tests. For that audience it shouldn't be necessary to check out Kubernetes and to build from source.

Developers of a CSI driver however are better served by including the E2E framework and the storage testsuites package in their own Ginkgo test suite. Then they can run it together with addtional, custom tests.

I had a comment with several questions, but I think I accidentally deleted that yesterday because the comment was shown to me twice. Let me write it down again:

  • How and where do we document this feature?
  • Do we need to support more than just dynamic provisioning?
  • Is it really okay to ignore the fsType parameter? The gcepd driver ignores it, but I wonder whether that means that all tests run with the default filesystem:
    func (g *gcePDCSIDriver) GetDynamicProvisionStorageClass(fsType string) *storagev1.StorageClass {
    ns := g.driverInfo.Config.Framework.Namespace.Name
    provisioner := g.driverInfo.Name
    suffix := fmt.Sprintf("%s-sc", g.driverInfo.Name)
    parameters := map[string]string{"type": "pd-standard"}
    return testsuites.GetStorageClass(provisioner, parameters, nil, ns, suffix)
    }
  • Which storage tests are required for a "conformant" storage driver (if that has been defined at all) and how do we enable running just those?
  • Do we need to execute any code inside the test driver? I don't think so. Skipping unsupported tests can be defined by regular expressions that match against the full Ginkgo test name.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 12, 2019

@pohly: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce a736fa2 link /test pull-kubernetes-e2e-gce
pull-kubernetes-e2e-gce-device-plugin-gpu a736fa2 link /test pull-kubernetes-e2e-gce-device-plugin-gpu
pull-kubernetes-bazel-build a736fa2 link /test pull-kubernetes-bazel-build
pull-kubernetes-e2e-kops-aws a736fa2 link /test pull-kubernetes-e2e-kops-aws
pull-kubernetes-typecheck a736fa2 link /test pull-kubernetes-typecheck
pull-kubernetes-e2e-gce-100-performance a736fa2 link /test pull-kubernetes-e2e-gce-100-performance
pull-kubernetes-kubemark-e2e-gce-big a736fa2 link /test pull-kubernetes-kubemark-e2e-gce-big
pull-kubernetes-bazel-test a736fa2 link /test pull-kubernetes-bazel-test
pull-kubernetes-integration a736fa2 link /test pull-kubernetes-integration
pull-kubernetes-verify a736fa2 link /test pull-kubernetes-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@wongma7

This comment has been minimized.

Copy link
Contributor

wongma7 commented Jan 14, 2019

e2e.test binaries don't seem to get included in binary releases (kubernetes-server-*.tar.gz). I am picturing some place in the kubernetes-csi org that documents how to run the csi e2e tests, like how https://github.com/cncf/k8s-conformance documents how to run sonobuoy to run k8s conformance tests. I think we are going to end up maintaining something out-of-tree like sonobuoy/csi-sanity anyway, because it's not trivial for developers to include the E2E framework and we need to provide an example. Given all that, I'm still unsure if this should be in-tree or if it could live out-of-tree to serve both cluster-admins and developers... i.e. cluster admins could run it as-is (each version of this out-of-tree thing would be pinned to a kubernetes version) and developers could also run it as-is or fork it if they want to write custom tests.

How and where do we document this feature?

I guess somewhere in kubernetes-csi docs and here https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md

Do we need to support more than just dynamic provisioning?

IMO no (assuming you mean GetPersistentVolumeSource)

Is it really okay to ignore the fsType parameter? The gcepd driver ignores it, but I wonder whether that means that all tests run with the default filesystem:

I think you are right, that seems like a bug, it means the dynamic XFS tests aren't actually testing XFS?

// XfsDynamicPV is TestPattern for "Dynamic PV (xfs)"
r.sc = dDriver.GetDynamicProvisionStorageClass(fsType)

Which storage tests are required for a "conformant" storage driver (if that has been defined at all) and how do we enable running just those?

Right, we need to formalize what "conformant" means first, then I think tagging the tests would be okay. At the moment there aren't so many tests that we need to be selective though...

@pohly

This comment has been minimized.

Copy link
Contributor

pohly commented Jan 14, 2019

@msau42

This comment has been minimized.

Copy link
Member

msau42 commented Jan 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment