Ensure Conformance Tests Do Not Require Beta REST APIs or Features

Table of Contents

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories
    • Story 1
    • Story 2
    • Story 3
• Design Details
  • Graduation Criteria
• Implementation History
• Infrastructure Needed
• References

Release Signoff Checklist

For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., kubernetes/kubernetes, we require the following Release Signoff checklist to be completed.

Check these off as they are completed for the Release Team to track. These checklist items must be updated for the enhancement to be released.

• kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR)
• Design details are appropriately documented
• Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• Graduation criteria is in place
• "Implementation History" section is up-to-date for milestone
• Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
• KEP approvers have set the KEP status to implementable
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io

Note: This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.

Summary

The KEP describes a process for ensuring that the Kubernetes components and the Kubernetes conformance tests have no dependencies on beta REST APIs or features.

Motivation

As the Kubernetes project matures, and is used as the foundation for other projects, production distributions are expected to increase in stability, reliability, and consistency. The Kubernetes conformance project is an important aspect of ensuring consistency across distributions. A production distribution should not be required to enable non-GA features or REST APIs in order to pass conformance tests.

Beta dependencies can be grouped into several categories:

• Kubernetes components requiring beta REST APIs with no stable alternatives (kubelet's use of the beta CertificateSigningRequest endpoints for cert rotation is a borderline example; cert rotation is not a required feature, but all our setup and test tools make use of it)
• Kubernetes components requiring beta REST APIs with stable alternatives (for example, kube-scheduler accidentally switched to using the beta Events REST API in 1.16)
• Kubernetes components requiring behavior enabled by beta feature gates
• Kubernetes conformance tests requiring calls to beta REST APIs
• Kubernetes conformance tests exercising behavior enabled by beta feature gates

For each category, we should identify and resolve existing dependencies, and prevent future occurrences.

Goals

• Identify existing beta REST APIs and features required by Kubernetes components so they can be graduated or the dependencies removed
• Identify existing beta REST APIs and features required by conformance tests so the tests can be rewritten to remove those dependencies
• Prevent new beta REST APIs and features from being required by Kubernetes components or conformance tests
• Demonstrate passing conformance tests on a cluster with all beta REST APIs and features disabled

Non-Goals

• Resolve questions about conformance profiles (see kubernetes/community#2651)
• Resolve questions about GA-but-optional features (see kubernetes/community#3997)
• Resolve code-level dependencies in Kubernetes source code on pre-v1.0.0 components
• Forbid beta REST APIs or features from being enabled in conformant clusters by distributors

Proposal

User Stories

Story 1

As a Kubernetes component developer, I cannot accidentally make changes that would require a production cluster to enable a beta API or feature in order to pass conformance tests.

Attempts to make such a change would be caught/blocked by a CI job.

Story 2

As a Kubernetes conformance test author, I cannot accidentally write (or promote to conformance) tests that require use of beta REST APIs.

Attempts to make such a change would be caught/blocked by a CI job.

Story 3

As a Kubernetes distribution maintainer, I can pass conformance tests without enabling beta REST APIs or features.

A canonical CI job running against Kubernetes master and release branches would demonstrate that it is possible to pass conformance tests with no beta REST APIs or features enabled.

Design Details

1. Make it possible to easily disable all beta REST APIs and features

• Add support for disabling built-in REST API versions of the form v[0-9]+beta[0-9]+ to the Kubernetes API server with --runtime-config api/beta=false
  • Parallels existing use of --runtime-config api/all=false
  • For completeness, we can also add support for disabling built-in REST API versions of the form v[0-9]+alpha[0-9]+ to the Kubernetes API server with --runtime-config api/alpha=false
• Add support for disabling beta feature gates to all components with --feature-gates AllBeta=false
  • Parallels existing use of --feature-gates AllAlpha=false

2. Identify existing beta REST APIs/features required by Kubernetes components or conformance tests

• Iteratively run a conformance test with all beta REST APIs and features disabled
• Identify failures due to uses of beta features/REST APIs
• Open issues for owners of the relevant tests or components to remove the beta dependency
• Construct an exemption list of beta features/REST APIs currently required to pass conformance

3. Prevent introduction of new required beta dependencies in Kubernetes components or conformance tests

• Set up a merge-blocking CI job running conformance tests with all beta REST APIs and features disabled except the exemptions constructed in step 2.

4. Resolve existing dependencies on beta REST APIs/features required by Kubernetes components or conformance tests

• Track issues opened in step 2
• As each dependency is resolved, remove it from the exemption list

Graduation Criteria

Phase 1:

• All beta REST APIs and features required to pass conformance tests are identified
• Blocking presubmit and periodic CI jobs ensure no additional beta dependencies are introduced

Phase 2:

• All identified dependencies on beta REST APIs and features to pass conformance are resolved
• Blocking presubmit and periodic CI jobs ensure no beta dependencies are introduced

Phase 3:

• All GA APIs required to pass conformance tests are identified
• Blocking presubmit and periodic CI jobs ensure no dependencies on optional GA APIs are introduced as required into conformance tests

Implementation History

• 2019-10-23: KEP created
• 2019-11-01: KEP marked implementable

Infrastructure Needed

A pre-submit and periodic CI job per release branch, configured with beta REST APIs and features disabled

References

Existing guidelines that conformance tests should only make use of GA APIs:

• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements

Past issues attempting to diagnose use of beta APIs by conformance tests:

• kubernetes/kubernetes#78605
• kubernetes/kubernetes#78613