remove [Conformance] flag on some e2es

[timothysc]

Downstream distributions that absorb the upstream tests would like to give their customers a standard mechanism to validate their clusters, post setup. As of today [Conformance] works for most things, but there are a known set of tests that vary due to opinionated differences around networking, security, etc... and providing a complete skip list can be cumbersome. To address this, we've simply modified the flag on some tests to [Conformance:Variant]. All existing behavior should be maintained.

Fixes: #34105

---

This change is

[spxtr]

I'm not a big fan. Is anyone against just removing the conformance tag from these tests?

[timothysc]

/cc @kubernetes/sig-testing for comments.

[timothysc]

I'm open to alternatives, but I don't think some of these tests should remove [Conformance] from an upstream perspective. It's just that they're not guaranteed to function downstream given the description above.

[spxtr]

I'm also afraid that we're matching the regex \[Conformance\] which might be a silent problem.

[timothysc]

We've federated out repos, I'm not entirely certain all the places to look.

[timothysc]

@spxtr This is a priority for us, would [Variant] be ok?

[spxtr]

Conformance:Variant or just Variant don't really say anything about the test, though. Are the tests themselves assuming too much about the cluster? If so, we should rework the tests or remove the conformance label, whichever is most appropriate.

We don't actually use the conformance label anywhere in the kubernetes repo, so the change doesn't make a big deal to us, but it smells.

[timothysc]

Are the tests themselves assuming too much about the cluster?

I think there are opinionated differences in deployments that can not easily be resolved. The problem today is upstream only tests/validates 1 configuration, and downstream distributions (e.g. Openshift) have made other choices which have now made the test no longer a [Conformance] test for that environment.

So I'm ok with the following:

1. Remove Conformance for those tests
2. Adding a label of some kind to denote skipping [Variant], [Conformance:Variant], whatever.
3. Some other solution which allows an easy -skip regex

[spxtr]

Can you give an example of an opinionated difference? For instance, should provide unchanging, static URL paths for kubernetes api services seems like it just does a GET request to a few standard paths. Why doesn't that work on Openshift?

[spxtr]

Is it possible to do some sort of SkipIfProviderIs("openshift") or something along those lines?

[timothysc]

Can you give an example of an opinionated difference?

It("should proxy to cadvisor [Conformance:Variant]", func() { nodeProxyTest(f, prefix+"/proxy/nodes/", ":4194/containers/")

We lock down our node tightly to not expose any potential attack surface.

Is it possible to do some sort of SkipIfProviderIs("openshift") or something along those lines?

No, openshift is not a "provider" it is local

Moving the change to [Variant], keeps all the current testing conventions with little/no negative upstream impact, what is the logical argument against it?

[spxtr]

The argument is just that Variant doesn't say anything specific about the test and it will be confusing for everyone else. What we need is some way to say "skip this test if we're on openshift".

[timothysc]

The 3 options provided earlier...

1. Remove Conformance for those tests
2. Adding a label of some kind to denote downstream divergence and document, e.g [Variant], [Conformance:Variant], whatever, I don't really care what the name is. Conventions are whatever we decide, and can be applied to tests however @kubernetes/sig-testing sees fit... What does [Conformance] mean? It only means something to those who bought into the conventions. If we decide the the convention is to call it [ScoobyDoo] then we document it and move along. Obviously we try to pick a name that makes sense... If you don't like variant, please offer a suggestion that seems appropriate.
3. Some other solution which allows an easy -skip regex, provider is not an option.

[spxtr]

Why is provider not an option?

[timothysc]

Why is provider not an option?

Provider in the tests, and other areas, is linked to cloud-provider, which openshift supports many. So provider has an known implementation, and connotation in the code.

[spxtr]

It sounds like we should remove the conformance label, then. This comes up every now and then in @kubernetes/sig-testing.

[timothysc]

It sounds like we should remove the conformance label, then.

Done.

[spxtr]

I'm fine with this. Anyone from @kubernetes/sig-testing disagree? If not I'll LGTM tomorrow.

[spxtr]

@k8s-bot gce e2e test this

#31561

[k8s-ci-robot]

Jenkins GCE e2e failed for commit eb29c61. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

[timothysc]

looks like infra-flakes.

[spxtr]

@k8s-bot test this

kubernetes/test-infra#798

[k8s-ci-robot]

Jenkins GCI GKE smoke e2e failed for commit eb29c61. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

[k8s-github-robot]

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue