fix: remove ExemptFeatures in experimental suite

[Xunzhuo]

What type of PR is this?

/kind bug

What this PR does / why we need it:

In experimental test suite, we did not remove ExemptFeatures. This makes ExemptFeatures in Options does not work.

Which issue(s) this PR fixes:

Fixes #2347

Does this PR introduce a user-facing change?:

Remove ExemptFeatures in experimental suite

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Xunzhuo
Once this PR has been reviewed and has the lgtm label, please assign mlavacca for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• conformance/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Xunzhuo]

/kind bug

[shaneutt]

/cc @mlavacca @sunjayBhatia

[mlavacca]

Thanks, @Xunzhuo! This change makes total sense to me, but I see it as incomplete:

what happens when you want to be conformant with let's say the HTTP profile, but you exempt a mandatory feature like SupportGateway? With this kind of check, I think the conformance will be given, even if the Gateway tests are not executed.

[Xunzhuo]

Thanks, @Xunzhuo! This change makes total sense to me, but I see it as incomplete:

what happens when you want to be conformant with let's say the HTTP profile, but you exempt a mandatory feature like SupportGateway? With this kind of check, I think the conformance will be given, even if the Gateway tests are not executed.

Thanks for comments @mlavacca, let me try to understand this point. You mean we cannot exempt a coreFeatures in profiles? If so, this is a bit werid for me, I think users should be able to exempt features in core or extended. Correct me if I misundstand it.

This is synced with https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/utils/suite/suite.go#L102

[sunjayBhatia]

Thanks, @Xunzhuo! This change makes total sense to me, but I see it as incomplete:
what happens when you want to be conformant with let's say the HTTP profile, but you exempt a mandatory feature like SupportGateway? With this kind of check, I think the conformance will be given, even if the Gateway tests are not executed.

Thanks for comments @mlavacca, let me try to understand this point. You mean we cannot exempt a coreFeatures in profiles? If so, this is a bit werid for me, I think users should be able to exempt features in core or extended. Correct me if I misundstand it.

This is synced with https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/utils/suite/suite.go#L102

I think @mlavacca's point is that if a feature is exempted that is part of one of the conformance profiles specified, then the resulting report should not mark that profile as having passed

e.g. if I use this branch and have a config like this:

	cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
		Options: suite.Options{
...
			SupportedFeatures:    suite.AllFeatures,
			SkipTests: []string{
...
			},
			ExemptFeatures: sets.New(
				suite.SupportMesh,
				suite.SupportHTTPRoutePortRedirect,
				suite.SupportReferenceGrant,
			),
		},
		Implementation: conformance_v1alpha1.Implementation{
...
		},
		ConformanceProfiles: sets.New(
			suite.HTTPConformanceProfileName,
			suite.TLSConformanceProfileName,
		),
	})

I still get a report that says everything passed even though not all the tests were run

        apiVersion: gateway.networking.k8s.io/v1alpha1
        date: "2023-08-15T19:57:19Z"
        gatewayAPIVersion: TODO
        implementation:
...
        kind: ConformanceReport
        profiles:
        - core:
            result: success
            statistics:
              Failed: 0
              Passed: 21
              Skipped: 0
            summary: ""
          extended:
            result: success
            statistics:
              Failed: 0
              Passed: 8
              Skipped: 0
            summary: ""
            supportedFeatures:
            - HTTPRouteHostRewrite
            - HTTPRoutePortRedirect
            - HTTPRoutePathRewrite
            - HTTPRouteRequestMirror
            - HTTPRouteQueryParamMatching
            - HTTPRouteMethodMatching
            - HTTPResponseHeaderModification
            - HTTPRouteSchemeRedirect
            - HTTPRoutePathRedirect
          name: HTTP
        - core:
            result: success
            statistics:
              Failed: 0
              Passed: 6
              Skipped: 0
            summary: ""
          name: TLS

Seems like for completeness we should make sure to set the profile result field to partial or otherwise if an exempt feature is specified that is part of a profile (and probably make sure the skipped tests are recorded as well)

[sunjayBhatia]

left a comment, the change is correct, but probably needs more to make it complete in the context of the conformance profile suite

[youngnick]

Yes, I agree that you should not be able to skip any Core tests in any profile and still have your conformance test marked as "Passing". Allowing that will undermine the value of the Core parts of the tests.

[mlavacca]

I think @mlavacca's point is that if a feature is exempted that is part of one of the conformance profiles specified, then the resulting report should not mark that profile as having passed

Yep, this is 100% what I meant 👍

[Xunzhuo]

Great for clarifications, I will update it to make it complete.

[robscott]

@Xunzhuo do you think you'll have a chance to follow up on this one?

[Xunzhuo]

Yep @robscott

[robscott]

@Xunzhuo thanks for the update here! Unless I'm missing something, it still doesn't address @sunjayBhatia's comment above though: #2279 (comment).

[Xunzhuo]

/close

@arkodg is working on #2515

[k8s-ci-robot]

@Xunzhuo: Closed this PR.

In response to this:

/close

Arko is working on https://github.com/kubernetes-sigs/gateway-api/pull/2515/files

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.