e2e/storage: testing of external storage drivers #72836

pohly · 2019-01-11T20:40:33Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

It is useful to apply the storage testsuite also to "external" (=
out-of-tree) storage drivers. One way of doing that is setting up a
custom E2E test suite, but that's still quite a bit of work.

An easier alternative is to parameterize the Kubernetes e2e.test
binary at runtime so that it instantiates the testsuite for one or
more drivers.

Special notes for your reviewer:

I would prefer to merge this after PR #72434.

Does this PR introduce a user-facing change?:

the test/e2e/e2e.test binary can test arbitrary storage drivers, see the `-storage.testdriver` parameter

wongma7 · 2019-01-12T04:56:10Z

This feels like it doesn't need to be in-tree...with all the/your efforts refactoring the framework I envisioned the ideal solution would be to develop something analogous to csi-sanity out of tree so that someone has full control over the interface, versioning, etc. of this "shim" between drivers and the existing e2e tests. I think the interface being an arg to e2e.test would be unpleasant to use...I need to checkout kubernetes & do make WHAT=test/e2e/e2e.test, probably building a bunch of unrelated stuff. And what if we someday want to add another args or deprecate one, the versioning of our arg "interface" will be difficult to communicate.

It doesn't mean we can't maintain both though.

Anyway I like very much the general idea of having a generic driver that simply takes yamls so that I don't have to write a TestDriver or any code at all unless I'm doing something complicated, it's exactly what I had in mind also.

pohly · 2019-01-12T13:43:16Z

/retest

pohly · 2019-01-12T14:01:18Z

@wongma7 I think @msau42 wanted this in-tree because then the e2e.test binary that gets released together with each Kubernetes release (it does get released, right? I'm not sure because I've never used it) can be used as-is. The intended users wouldn't be developers, but rather cluster admins who want to run sanity or conformance tests. For that audience it shouldn't be necessary to check out Kubernetes and to build from source.

Developers of a CSI driver however are better served by including the E2E framework and the storage testsuites package in their own Ginkgo test suite. Then they can run it together with addtional, custom tests.

I had a comment with several questions, but I think I accidentally deleted that yesterday because the comment was shown to me twice. Let me write it down again:

How and where do we document this feature?
Do we need to support more than just dynamic provisioning?

Is it really okay to ignore the fsType parameter? The gcepd driver ignores it, but I wonder whether that means that all tests run with the default filesystem:

kubernetes/test/e2e/storage/drivers/csi.go

Lines 309 to 317 in 8feab37

    
           func (g *gcePDCSIDriver) GetDynamicProvisionStorageClass(fsType string) *storagev1.StorageClass { 
        
           	ns := g.driverInfo.Config.Framework.Namespace.Name 
        
           	provisioner := g.driverInfo.Name 
        
           	suffix := fmt.Sprintf("%s-sc", g.driverInfo.Name) 
        
           	parameters := map[string]string{"type": "pd-standard"} 
        
           	return testsuites.GetStorageClass(provisioner, parameters, nil, ns, suffix) 
        
           }

Which storage tests are required for a "conformant" storage driver (if that has been defined at all) and how do we enable running just those?
Do we need to execute any code inside the test driver? I don't think so. Skipping unsupported tests can be defined by regular expressions that match against the full Ginkgo test name.

wongma7 · 2019-01-14T14:15:50Z

e2e.test binaries don't seem to get included in binary releases (kubernetes-server-*.tar.gz). I am picturing some place in the kubernetes-csi org that documents how to run the csi e2e tests, like how https://github.com/cncf/k8s-conformance documents how to run sonobuoy to run k8s conformance tests. I think we are going to end up maintaining something out-of-tree like sonobuoy/csi-sanity anyway, because it's not trivial for developers to include the E2E framework and we need to provide an example. Given all that, I'm still unsure if this should be in-tree or if it could live out-of-tree to serve both cluster-admins and developers... i.e. cluster admins could run it as-is (each version of this out-of-tree thing would be pinned to a kubernetes version) and developers could also run it as-is or fork it if they want to write custom tests.

How and where do we document this feature?

I guess somewhere in kubernetes-csi docs and here https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md

Do we need to support more than just dynamic provisioning?

IMO no (assuming you mean GetPersistentVolumeSource)

Is it really okay to ignore the fsType parameter? The gcepd driver ignores it, but I wonder whether that means that all tests run with the default filesystem:

I think you are right, that seems like a bug, it means the dynamic XFS tests aren't actually testing XFS?

kubernetes/test/e2e/storage/testpatterns/testpattern.go

Line 132 in e41d9f1

// XfsDynamicPV is TestPattern for "Dynamic PV (xfs)"

kubernetes/test/e2e/storage/testsuites/base.go

Line 167 in 3506146

r.sc = dDriver.GetDynamicProvisionStorageClass(fsType)

Which storage tests are required for a "conformant" storage driver (if that has been defined at all) and how do we enable running just those?

Right, we need to formalize what "conformant" means first, then I think tagging the tests would be okay. At the moment there aren't so many tests that we need to be selective though...

pohly · 2019-01-14T16:34:54Z

Matthew Wong <notifications@github.com> writes:

e2e.test binaries don't seem to get included in binary releases (kubernetes-server-*.tar.gz). I am picturing some place in the kubernetes-csi org that documents how to run the csi e2e tests, like how https://github.com/cncf/k8s-conformance documents how to run sonobuoy to run k8s conformance tests.

I thought Sonobuoy was only a simpler way of running the conformance tests, but from https://github.com/cncf/k8s-conformance/blob/master/instructions.md it looks it is more than that. I don't think kubernetes-csi should establish its own "storage conformance" testing. IMHO that belongs into the normal Kubernetes conformance testing and thus needs to be in-tree. For additional features a separate test suite would work.

I think we are going to end up maintaining something out-of-tree like sonobuoy/csi-sanity anyway, because it's not trivial for developers to include the E2E framework and we need to provide an example.

I have a prototype here: https://github.com/pohly/csi-e2e My plan was to use that for a CSI release job in Prow. Publishing the resulting test/e2e.test binary could be added.

Given all that, I'm still unsure if this should be in-tree or if it could live out-of-tree to serve both cluster-admins and developers... i.e. cluster admins could run it as-is (each version of this out-of-tree thing would be pinned to a kubernetes version) and developers could also run it as-is or fork it if they want to write custom tests.

@msau42, what do you think?

> Do we need to support more than just dynamic provisioning? IMO no (assuming you mean GetPersistentVolumeSource)

That's what I meant.

I think you are right, that seems like a bug, it means the dynamic XFS tests aren't actually testing XFS? https://github.com/kubernetes/kubernetes/blob/e41d9f15530d5c5bd116c90a988c8245f0be7974/test/e2e/storage/testpatterns/testpattern.go#L132 https://github.com/kubernetes/kubernetes/blob/35061468cc9c03a377745a731a6ce7151f959948/test/e2e/storage/testsuites/base.go#L167

Looks like it. Can someone from Google check?

msau42 · 2019-01-15T18:18:17Z

Regarding fstype testing:
kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#192
#70822

test/e2e/storage/external/external.go

wongma7 · 2019-01-18T21:28:47Z

test/e2e/storage/external/external.go

+
+	// StorageClass is required for enabling dynamic provisioning tests.
+	// When set to "default", a StorageClass with DriverInfo.Name as provisioner is used.
+	// Everything else is treated like the filename of a .yaml or .json file


the file is searched for in what path? I guess RepoRoot ($GOPATH/src/k8s.io/kubernetes/) ? Should document it here

Strictly speaking, it depends on how the e2e/framework/testfiles package is set up in the E2E test suite that includes this package here. But you are right, I need to mention that here and explain the relationship with the -repo-root parameter.

A safer approach will be to use an absolute file. #71667 was merged, so that works.

mathu97 · 2019-01-18T21:32:49Z

test/e2e/storage/external/external.go

@@ -0,0 +1,217 @@
+/*


Can we add a README under .../storage/external?

+1, an explanation of how to use this and maybe even a link to an example (when there is one) would be very helpful

fyi, @wongma7 and @mathu97 are working on documentation/example

That's good, but I'll also add short README here.

pohly · 2019-01-21T08:12:29Z

@wongma7 I found that e2e.test binaries do get released together with each Kubernetes release. For example: https://dl.k8s.io/v1.13.0/kubernetes-test.tar.gz

There's a KEP pending about reorganizing that a bit (https://github.com/kubernetes/enhancements/pull/701/files) but in principle there is a path of getting a generic test binary into the hand of users, if that's what we want to do (still undecided).

wongma7 · 2019-01-31T20:42:16Z

@pohly OK, sgtm, then I don't really have any objections to pointing users to e2e.test and maintaining this code here.

pohly · 2019-02-12T19:33:44Z

@msau42 did we come to a conclusion whether we want this in-tree?

msau42 · 2019-02-13T01:47:01Z

One advantage of keeping it in-tree is that the tests definitions are tied to the K8s release. You won't have to manage things like K8s adding new tests for new features that may fail on older K8s clusters. And test fixes will automatically get picked up with each release.

A disadvantage is it becomes difficult to manage differences in the driver definition across releases. But if this doesn't change often, then it should be manageable.

Will these test cases try to run in the intree e2e job because they're defined in the binary?

cc @davidz627 if he has ideas on how we would ideally want to do e2e testing in the pd repo.

pohly · 2019-02-13T09:12:30Z

Michelle Au <notifications@github.com> writes:

A disadvantage is it becomes difficult to manage differences in the driver definition across releases. But if this doesn't change often, then it should be manageable.

Agreed.

Will these test cases try to run in the intree e2e job because they're defined in the binary?

The tests are only getting defined if the -storage.testdriver parameter is given, so no. We could rip out the in-tree definition of the gcepd driver and replace it with a pre-test deployment (defined by the Prow job) and execution via -storage.testdriver, but that's a separate change.

davidz627

thanks for working on this! Looks mostly pretty good but I have a couple questions. I also think a README explaining how this works and how to use it would be very valuable.

davidz627 · 2019-02-26T22:23:03Z

test/e2e/storage/external/external.go

@@ -0,0 +1,217 @@
+/*


+1, an explanation of how to use this and maybe even a link to an example (when there is one) would be very helpful

davidz627 · 2019-02-26T22:24:13Z

test/e2e/storage/external/external.go

+		return errors.Errorf("%q: DriverInfo.Name not set", filename)
+	}
+
+	// klog.Infof("Test driver definition from %q: %+v", filename, driver)


nit: uncomment or remove these logs

I haven't fully decided about that yet. I added that because it gives feedback to the user about which parts of his config file really had an effect (because parsing will silently ignore unknown fields) and to make it easier for users to find "their" tests, but then the output is repeated once for each test process in a parallel Ginkgo run, so I commented it out again.

I'll probably remove it.

davidz627 · 2019-02-26T22:27:50Z

test/e2e/storage/external/external.go

+	// When set to "default", a StorageClass with DriverInfo.Name as provisioner is used.
+	// Everything else is treated like the filename of a .yaml or .json file
+	// that defines a StorageClass.
+	StorageClass string


Not sure this variable should be "dual use" here. Seems like there's a concept of StorageClassName and a concept of StorageClassDefinitionFile and they should be separated out with sensible defaults for each.

Turned StorageClass into a struct with two members, FromName and FromFile.

davidz627 · 2019-02-26T22:28:49Z

test/e2e/storage/external/external.go

+
+// Needed only for deserialization which isn't done, therefore the
+// functions below can be empty stubs.
+var _ runtime.Object = &driverDefinition{}


is its "only needed for deserialization which isnt done" then why do we need it at all?

driverDefinition must implement the interface, otherwise it cannot be passed to runtime.DecodeInto. But runtime.DecodeInto then never calls these methods.

If there is a better method for parsing a .yaml or .json file into a normal struct, then I am all ears.

davidz627 · 2019-02-26T22:34:39Z

test/e2e/storage/external/external.go

+
+func (d *driverDefinition) SkipUnsupportedTest(pattern testpatterns.TestPattern) {
+	supported := false
+	// TODO (?): add support for more volume types


I think the volume types supported should also be defined by the external driver. Some may support Block or Snapshots in the future and you'd probably want to be able to toggle those on/off through the DriverInfo definition or some other method.

This isn't about snapshot or blocks. Other pattern.VolType values are currently:

PreprovisionedPV

InlineVolume

I think those cannot be tested with a generic test driver.

kubernetes/test/e2e/storage/testpatterns/testpattern.go

Lines 36 to 46 in 076af3d

// TestVolType represents a volume type to be tested in a TestSuite

type TestVolType string

var (

// InlineVolume represents a volume type that is used inline in volumeSource

InlineVolume TestVolType = "InlineVolume"

// PreprovisionedPV represents a volume type for pre-provisioned Persistent Volume

PreprovisionedPV TestVolType = "PreprovisionedPV"

// DynamicPV represents a volume type for dynamic provisioned Persistent Volume

DynamicPV TestVolType = "DynamicPV"

)

But snapshots is something that can be added. It's not in the PR yet because it was added to the TestDriver interface after my initial version of this PR.

I think block device don't need any special support. It's a capability, and that can be set already.

I've added snapshotting support. I've tried it manually and the test ran, but failed because my cluster lacked some feature gate:

• Failure in Spec Teardown (AfterEach) [365.125 seconds] External Storage [Driver: csi-hostpath] /nvme/gopath/src/k8s.io/kubernetes/test/e2e/storage/external/external.go:78 [Testpattern: Dynamic PV (default fs)] provisioning /nvme/gopath/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:75 should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [AfterEach] /nvme/gopath/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:202 Expected error: <*errors.StatusError | 0xc0005290e0>: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: "PersistentVolumeClaim \"pvc-6xw22\" is invalid: spec.dataSource: Forbidden: VolumeSnapshotDataSource is disabled by feature-gate", Reason: "Invalid", Details: { Name: "pvc-6xw22", Group: "", Kind: "PersistentVolumeClaim", UID: "", Causes: [ { Type: "FieldValueForbidden", Message: "Forbidden: VolumeSnapshotDataSource is disabled by feature-gate", Field: "spec.dataSource", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } PersistentVolumeClaim "pvc-6xw22" is invalid: spec.dataSource: Forbidden: VolumeSnapshotDataSource is disabled by feature-gate not to have occurred

Why not PreprovisionedPV? Like StorageClass, there could be a PersistentVolume field with FromName or FromFile and its source could be returned in GetPersistentVolumeSource. That said, other than maybe for the nfs and iscsi csi drivers IDK if it will be useful

It might be useful for CSI drivers in progress. e.g. I haven't implemented CreateVolume yet but want to make sure publish works.

It gets complicated if we want to add test cases that create multiple volumes. Dynamic provisioning is very useful for writing test cases like that.

But maybe those test cases can just be gated based on if the driver supports Preprovisioned PVs.

We also need to think about if we want to support inline ephemeral CSI volumes now too. Most of our test cases are specific to PVs

davidz627 · 2019-02-26T22:39:35Z

test/e2e/storage/external/external.go

+}
+
+func init() {
+	flag.Var(testDriverParameter{}, "storage.testdriver", "name of a .yaml or .json file that defines a driver for storage testing, can be used more than once")


How can this be used more than once? Do you just define the flag multiple times in the command line and go just figures it out that you want to init multiple versions of this package?

It's not clear to me how to invoke this test package, what kind of parameter to pass in to testDriverParameter since it seems to be just an empty struct.

I think these are all questions that could be answered in a README.

maybe another way to do this would be that testDriverParameter could point to the file that defines the DriverInfo? It seems like maybe thats the idea but i'm not sure why we're using an empty struct that might be implementing an interface (I assume thats where Set comes from even though theres no static interface check). It's just a little bit confusing to me

Yes, -storage.testdriver can be given more than once. Each parameter instantiates the testsuite for another driver. The mechanism for doing that is the testDriverParameter and its implementation of the flag.Value interface. Let me add the interface check and some comments to make this clearer.

If we just stored the parameter instead of handling it right away, then a) the parameter can only be used once (at least when using a plain string value) and more importantly, b) we have no way of hooking into the process init between "parsing parameters" and "test suite starts", i.e. we cannot define the tests for that driver (or drivers).

Added an explanation.

pohly · 2019-03-04T22:33:35Z

@davidz627 @wongma7 please have another look. This should now be ready for merging, or at least as ready as I can make it for 1.14.

I've been using the branch for my kubernetes-csi CI work (kubernetes-csi/csi-driver-host-path#16) and it works for that.

pohly · 2019-03-05T06:41:46Z

/retest

wongma7

lgtm.
some typos to fix later maybe

wongma7 · 2019-03-05T17:01:32Z

test/e2e/storage/external/README.md

@@ -0,0 +1,49 @@
+When a test suite like Kubernete's test/e2e/e2e.test includes this


Kubernetes's?

I think it's "Kubernetes' test/e2e/e2e.test". https://data.grammarbook.com/blog/apostrophes/apostrophes-with-words-ending-in-s/

But let's better avoid that altogether with "like test/e2e/e2e.test from Kubernetes"... done.

wongma7 · 2019-03-05T17:01:56Z

test/e2e/storage/external/README.md

+times to enabling testing of a certain pre-installed storage driver.
+
+The parameter takes as argument the name of a .yaml or .json file. The
+filename can be absolute or relative too --repo-root. The content of


pohly · 2019-03-05T20:16:02Z

One test failed anyway, so I pushed another commit with the typo fixes.

pohly · 2019-03-05T21:45:37Z

/retest

pohly · 2019-03-05T22:18:21Z

@wongma7 @msau42: all test clean now, please lgtm and approve if you think it is ready.

wongma7 · 2019-03-05T22:32:34Z

/lgtm
thank you

msau42 · 2019-03-05T22:49:11Z

test/e2e/storage/external/external.go

+	StorageClass struct {
+		// FromName set to true enables the usage of a
+		// storage class with DriverInfo.Name as provisioner.
+		FromName bool


Can you clarify in the comments that this well generate a storageclass with empty parameters?

Done. I also added that the FromFile option is for the case where parameters are needed.

msau42 · 2019-03-05T22:56:13Z

test/e2e/storage/external/external.go

+	// provisioned volumes. Default is "5GiB".
+	ClaimSize string
+
+	// ClientNodeName selects a specific node for scheduling test pods.


Clarify this is only needed for drivers like hostpath AND do not support topology

Actually do we want to allow this? Can we just require that drivers implement topology if they have such a constraint?

We do need this option right now ourselves for testing the hostpath driver as part of the upcoming kubernetes-csi CI. It's okay to explain that it shouldn't be needed and I have added a comment about it, but we should still have that option.

msau42 · 2019-03-05T22:59:39Z

test/e2e/storage/external/external.go

+
+func (d *driverDefinition) SkipUnsupportedTest(pattern testpatterns.TestPattern) {
+	supported := false
+	// TODO (?): add support for more volume types


It gets complicated if we want to add test cases that create multiple volumes. Dynamic provisioning is very useful for writing test cases like that.

But maybe those test cases can just be gated based on if the driver supports Preprovisioned PVs.

We also need to think about if we want to support inline ephemeral CSI volumes now too. Most of our test cases are specific to PVs

pohly · 2019-03-06T08:11:53Z

We also need to think about if we want to support inline ephemeral CSI volumes now too.

We can always extend this later. Right now it supports everything that can be supported without invoking extra code, doesn't it?

Supporting pre-provisioned volumes would imply that those volumes get created somehow and then are made available for testing. I haven't thought much yet about how to do this in detail. You once mentioned running code shipped in containers. That probably would work.

msau42 · 2019-03-06T17:35:43Z

/approve
thanks so much for working on this!
please squash

k8s-ci-robot · 2019-03-06T17:36:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

It is useful to apply the storage testsuite also to "external" (= out-of-tree) storage drivers. One way of doing that is setting up a custom E2E test suite, but that's still quite a bit of work. An easier alternative is to parameterize the Kubernetes e2e.test binary at runtime so that it instantiates the testsuite for one or more drivers. Some parameters have to be provided before starting the test because they define configuration and capabilities of the driver and its storage backend that cannot be discovered at runtime. This is done by populating the DriverDefinition with the content of the file that the new -storage.testdriver parameters points to. The universal .yaml and .json decoder from Kubernetes is used. It's flexible, but has some downsides: - currently ignores unknown fields (see kubernetes#71589) - poor error messages when fields have the wrong type Storage drivers have to be installed in the test cluster before starting e2e.test. Only tests involving dynamically provisioned volumes are currently supported.

pohly · 2019-03-06T21:07:05Z

@msau42 I've rebased and squashed.

msau42 · 2019-03-06T22:05:33Z

/lgtm
/priority important-soon

davidz627 · 2019-03-06T22:34:43Z

/retest

k8s-ci-robot requested a review from msau42 January 11, 2019 20:44

k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Jan 11, 2019

mathu97 reviewed Jan 18, 2019

View reviewed changes

test/e2e/storage/external/external.go Outdated Show resolved Hide resolved

wongma7 reviewed Jan 18, 2019

View reviewed changes

mathu97 reviewed Jan 18, 2019

View reviewed changes

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 26, 2019

pohly force-pushed the storage-test-plugin branch 3 times, most recently from dcfd365 to de799a4 Compare February 26, 2019 17:29

davidz627 reviewed Feb 26, 2019

View reviewed changes

pohly force-pushed the storage-test-plugin branch from de799a4 to 056ae4d Compare March 1, 2019 20:42

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 4, 2019

pohly force-pushed the storage-test-plugin branch 2 times, most recently from 6754c13 to 27ce623 Compare March 5, 2019 12:16

wongma7 approved these changes Mar 5, 2019

View reviewed changes

k8s-ci-robot assigned wongma7 Mar 5, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 5, 2019

msau42 reviewed Mar 5, 2019

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 6, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2019

pohly force-pushed the storage-test-plugin branch from f3832b4 to 6644db9 Compare March 6, 2019 21:06

k8s-ci-robot assigned msau42 Mar 6, 2019

k8s-ci-robot merged commit 8178fa1 into kubernetes:master Mar 7, 2019

pohly mentioned this pull request Mar 7, 2019

e2e: external snapshotter #75133

Merged

davidz627 mentioned this pull request Mar 9, 2019

Migrate k8s integration test suite to new parameterized suite kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#228

Closed

	// TestVolType represents a volume type to be tested in a TestSuite
	type TestVolType string

	var (
	// InlineVolume represents a volume type that is used inline in volumeSource
	InlineVolume TestVolType = "InlineVolume"
	// PreprovisionedPV represents a volume type for pre-provisioned Persistent Volume
	PreprovisionedPV TestVolType = "PreprovisionedPV"
	// DynamicPV represents a volume type for dynamic provisioned Persistent Volume
	DynamicPV TestVolType = "DynamicPV"
	)

		@@ -0,0 +1,49 @@
		When a test suite like Kubernete's test/e2e/e2e.test includes this

e2e/storage: testing of external storage drivers #72836

e2e/storage: testing of external storage drivers #72836

Conversation

pohly commented Jan 11, 2019

wongma7 commented Jan 12, 2019

pohly commented Jan 12, 2019

pohly commented Jan 12, 2019

wongma7 commented Jan 14, 2019

pohly commented Jan 14, 2019 via email

msau42 commented Jan 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Jan 21, 2019

wongma7 commented Jan 31, 2019

pohly commented Feb 12, 2019

msau42 commented Feb 13, 2019

pohly commented Feb 13, 2019 via email

davidz627 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidz627 Feb 26, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Mar 4, 2019

pohly commented Mar 5, 2019

wongma7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Mar 5, 2019

pohly commented Mar 5, 2019

pohly commented Mar 5, 2019

wongma7 commented Mar 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Mar 6, 2019

msau42 commented Mar 6, 2019

k8s-ci-robot commented Mar 6, 2019

pohly commented Mar 6, 2019

msau42 commented Mar 6, 2019

davidz627 commented Mar 6, 2019

davidz627 Feb 26, 2019 •

edited