Add CLI script for listing untested conformance behaviors

[Jefftree]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add CLI script for listing conformance behaviors that have not been covered by tests.

Usage: go run test/conformance/kubetestlink/kubetestlink.go

Since we don't have the full list of behaviors yet I'm still not sure about all the features we want to include with this CLI utility.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[Jefftree]

/assign @johnbelamaric

[Jefftree]

I moved this to a separate library to avoid duplication

[johnbelamaric]

Thanks @Jefftree. A couple thoughts come to mind:

1. Rather than another CLI utility, let's rework kubetestgen to be a tool for managing the conformance tests and behaviors. So, combine this and kubetestgen into something like kubeconform. Let's give some thought to the different functions we might need there and perhaps update the behaviors KEP to define the new functions.

2. If we base this on reading conformance.yaml, then we need to keep generating that file. It is a source of conflicts, as we have discussed before. Is it possible to instead use the same code that generates conformance.yaml to navigate the tests directly and get the lists of behaviors?

Along these lines, we need to update the KEP to reflect our desired end state better. That is, we want tooling that eliminates the need for conformance.yaml, and that measures conformance by whether the behaviors have been tested successfully. We want that ideally to come out of the active tests logs, so that it is clear from those logs that that the tests that hit a behavior have run, and have run successfully. This will provide the basis for the profiles work.

[Jefftree]

1. Rather than another CLI utility, let's rework kubetestgen to be a tool for managing the conformance tests and behaviors. So, combine this and kubetestgen into something like kubeconform. Let's give some thought to the different functions we might need there and perhaps update the behaviors KEP to define the new functions.

Sure, will make the change. I'd prefer committing smaller pieces and adding more functionality as we need them, so updating the KEP is definitely helpful.

2. If we base this on reading conformance.yaml, then we need to keep generating that file. It is a source of conflicts, as we have discussed before. Is it possible to instead use the same code that generates conformance.yaml to navigate the tests directly and get the lists of behaviors?

conformance.yaml is a snapshot of the state of conformance behaviors at a point in time. This generating list of behaviors <> test linkage is not committed, I'm not sure how we'd be in a state of conflict? In addition, we have a blocking CI job to ensure that the YAML file is up to date.
I agree that the long term goal is to eventually remove conformance.yaml but at the moment it's a fairly crucial file. I'm open to using the tooling to generate conformance.yaml in kubeconform as we progress, but in the current state I feel that using the YAML file is simpler, quicker to run, and less error prone.

[johnbelamaric]

I am ok with using conformance.yaml for now.

[johnbelamaric]

Oh, and I meant that in general it's a single file everyone has to regenerate when they promote tests, so it creates conflicts in PRs, not that it conflicts with this code.

[k8s-ci-robot]

@Jefftree: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce-csi-serial	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-gce-csi-serial
pull-kubernetes-e2e-gce-storage-slow	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-gce-storage-slow
pull-kubernetes-e2e-azure-disk-vmss	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-azure-disk-vmss
pull-kubernetes-e2e-azure-file-windows	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-azure-file-windows
pull-kubernetes-e2e-azure-file	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-azure-file
pull-kubernetes-e2e-gce-storage-snapshot	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-gce-storage-snapshot
pull-kubernetes-e2e-azure-disk	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-azure-disk
pull-kubernetes-e2e-aks-engine-azure	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-aks-engine-azure
pull-kubernetes-e2e-azure-disk-windows	0230c25fee9bb84567f94443c2dadae2653c8368	link	/test pull-kubernetes-e2e-azure-disk-windows

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[Jefftree]

/retest

[johnbelamaric]

Awesome, thanks. As we define the full set of commands for kubeconform we'll probably want to move this to use the same flag processing and other structure of existing kube CLI tools.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jefftree, johnbelamaric

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/conformance/OWNERS [johnbelamaric]
• test/conformance/behaviors/OWNERS [johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment