Add tooling for publishing OpenShift kubebuilder tools #1774

JoelSpeed · 2024-02-20T23:47:42Z

Envtest, a testing suite used in openshift/api and most controller-runtime based operators leverages archives from a GCS bucket to provide etcd and KAS binaries for local and integration testing.

As of 4.16, we are carrying patches in KAS that bring some upstream API features into OpenShift ahead of their general release in KAS, specifically the CEL net.IP and net.CIDR extensions.

To leverage these, both o/api and others will need binaries provided to be able to test CRDs leveraging these extensions.

This PR adds a go tool/script that extracts the binaries from a payload image, creates the required archives and then uploads them to an openshift GCS bucket.

We will need to run this periodically to create new releases when new binaries are needed, circa once a release.

We may also deprecate support for this when the API features reconverge (4.18), but it's worth keeping the tooling and setup around in case we need to leverage it again in the future

openshift-ci · 2024-02-20T23:47:48Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

JoelSpeed · 2024-02-20T23:47:50Z

/test integration

openshift-ci · 2024-02-20T23:47:51Z

Hello @JoelSpeed! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

JoelSpeed · 2024-02-21T16:46:19Z

/test integration

deads2k · 2024-03-12T13:17:57Z

tools/Makefile

+
+.PHONY: publish-kubebuilder-tools
+publish-kubebuilder-tools:
+	go run ./publish-kubebuilder-tools/main.go -version 1.29.1 -output-dir /tmp/kubebuilder-tools -payload=registry.ci.openshift.org/ocp/release:4.16.0-0.ci-2024-02-20-022620 -pull-secret $(PULL_SECRET)


if memory serves, there's some way to have a repo build produce a change to an imagestream. I'm ok with some ugliness to get our current problem resolved, but a thing to consider for a future update.

Is this a suggestion for automating this? Typically we only update the artifacts version once a release, so given this needs to be run circa 3 times a year, I didn't see too much point investing in further automation, but if you think it would be better to release more frequently, I can look into that.

deads2k · 2024-03-12T13:19:46Z

tools/publish-kubebuilder-tools/main.go

+	}
+
+	// We only expect images from the internal releases
+	if *payload == "" || !strings.HasPrefix(*payload, "registry.ci.openshift.org/ocp/release:") {


there is talk of removing our internal registry. Do we have a fall-back that mostly works?

I could update it to use any registry I guess, I just made some assumptions here that people would always use a nightly. Any timeline for that?

deads2k · 2024-03-12T13:21:00Z

tests/Makefile

@@ -8,7 +8,7 @@ integration: verify-vendor test

 .PHONY: test
 test: ## Run only the tests.


Can we have the old version of this as a different target? Say test-legacy or test-upstream or some such? When this breaks, I'd like to get our repo mostly unstuck quickly.

deads2k · 2024-03-12T13:22:37Z

tools/publish-kubebuilder-tools/main.go

+	flag.Parse()
+
+	if *pullSecretFile == "" {
+		panic("pull-secret is required")


Give me a hint for how to create this. Do I log into a CI cluster and do the oc registry login thing?

Added a note to the makefile where this is being called

deads2k · 2024-03-12T13:22:54Z

tools/publish-kubebuilder-tools/main.go

+
+func main() {
+	pullSecretFile := flag.String("pull-secret", "", "The pull secret to use for the kubebuilder tools")
+	version := flag.String("version", "", "The version of the kubebuilder tools to publish. This should be a Kubernetes version that the build is based upon.")


does it include z-streams or not?

deads2k · 2024-03-12T13:25:24Z

tests/Makefile

@@ -8,7 +8,7 @@ integration: verify-vendor test

 .PHONY: test
 test: ## Run only the tests.
-	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path --bin-dir $(PROJECT_DIR)/bin)" ./hack/test.sh
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path --bin-dir $(PROJECT_DIR)/bin --remote-bucket openshift-kubebuilder-tools)" ./hack/test.sh


can I get a comment with a link to the docs for whatever command has this --remote-bucket arg so should you win the lottery I have some shot at reverse engineering how it works. :)

deads2k · 2024-03-12T13:28:32Z

I understand the goal. I think it'll be fragile over time. I'm ok doing this to resolve our initial problem, but in three releases or so we should try to collapse back.

Some comments. I'd like the legacy-test target created for a rapid fallback.

/lgtm
/hold until we agree on IOU vs update

@JoelSpeed your credit in the area is good. Your call on IOU or pre-merge.

deads2k · 2024-03-12T14:32:13Z

/lgtm
/hold cancel

openshift-ci · 2024-03-12T14:34:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JoelSpeed,deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2024-03-12T15:28:57Z

@JoelSpeed: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

JoelSpeed added 3 commits February 20, 2024 10:22

Update tools vendor

6e353bc

Add publish-kubebuilder-tools tool

07bd45d

Update integration tests to use openshift published artifacts

25df700

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 20, 2024

openshift-ci bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 20, 2024

Update test error outputs for status validations

5d9fcca

JoelSpeed marked this pull request as ready for review February 21, 2024 17:31

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 21, 2024

openshift-ci bot requested review from danwinship and deads2k February 21, 2024 17:35

deads2k reviewed Mar 12, 2024

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 12, 2024

openshift-ci bot assigned deads2k Mar 12, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 12, 2024

Add more documentation to Makefiles

6d5d016

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 12, 2024

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 12, 2024

openshift-merge-bot bot merged commit 16eec10 into openshift:master Mar 12, 2024
9 checks passed

JoelSpeed deleted the publish-kubebuilder-tools branch March 13, 2024 10:39

djoshy mentioned this pull request Apr 25, 2024

NO-ISSUE: API bump & reconcile feature gates move openshift/machine-config-operator#4335

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tooling for publishing OpenShift kubebuilder tools #1774

Add tooling for publishing OpenShift kubebuilder tools #1774

JoelSpeed commented Feb 20, 2024

openshift-ci bot commented Feb 20, 2024

JoelSpeed commented Feb 20, 2024

openshift-ci bot commented Feb 20, 2024

JoelSpeed commented Feb 21, 2024

deads2k Mar 12, 2024

JoelSpeed Mar 12, 2024

deads2k Mar 12, 2024

JoelSpeed Mar 12, 2024

deads2k Mar 12, 2024

JoelSpeed Mar 12, 2024

deads2k Mar 12, 2024

JoelSpeed Mar 12, 2024

deads2k Mar 12, 2024

deads2k Mar 12, 2024

deads2k commented Mar 12, 2024

deads2k commented Mar 12, 2024

openshift-ci bot commented Mar 12, 2024

openshift-ci bot commented Mar 12, 2024

		@@ -8,7 +8,7 @@ integration: verify-vendor test

		.PHONY: test
		test: ## Run only the tests.

Add tooling for publishing OpenShift kubebuilder tools #1774

Add tooling for publishing OpenShift kubebuilder tools #1774

Conversation

JoelSpeed commented Feb 20, 2024

openshift-ci bot commented Feb 20, 2024

JoelSpeed commented Feb 20, 2024

openshift-ci bot commented Feb 20, 2024

JoelSpeed commented Feb 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Mar 12, 2024

deads2k commented Mar 12, 2024

openshift-ci bot commented Mar 12, 2024

openshift-ci bot commented Mar 12, 2024