[release-1.32] Add bash script to downgrade kafka contracts and run as part of downgrade tests

[creydr]

Same as #2553, but includes the linter fixes.

Credits go to @Cali0707!

/cc @Cali0707 @matzew

[mgencur]

Hey, this looks great! It will make testing much easier.

One question: The code modifies a config map in knative-eventing NS. Will this be later overwritten/reconciled by the operator to the original value? Or not. Thanks

[matzew]

That is a GOOD question

[creydr]

One question: The code modifies a config map in knative-eventing NS. Will this be later overwritten/reconciled by the operator to the original value? Or not. Thanks

Not sure if I understood you correctly, but the downgraded kafka-controller knows only the left fields in the contract. Or which component do you mean to reconcile the configmaps?

I am only a bit wondering about the order we execute this. Currently we do this, before we know that the new/old (1.10) operator is fully running:

serverless-operator/test/upgrade/installation/serverless.go

Lines 196 to 206 in ca3472c

	if err := downgradeKafkaContractsWithScript(); err != nil {
	return fmt.Errorf("failed to downgrade contracts: %w", err)
	}
	if _, err := v1alpha1.WaitForKnativeKafkaState(ctx,
	"knative-kafka",
	knativeEventing,
	v1alpha1.IsKnativeKafkaWithVersionReady(strings.TrimPrefix(test.Flags.KafkaVersionPrevious, "v")),
	); err != nil {
	return fmt.Errorf("knative kafka downgrade failed: %w", err)
	}

So IIUC we could have the case, that we update the configmaps while still the 1.11 kafka-controller running and reconciling the configmaps back to include the trustBundles fields 🤔 (but I saw @Cali0707 changing this in 8136b0d 🤷 )

[matzew]

I think the question was, if the Serverless Operator (owing those CMs, b/c applying them), is not reverting the change of the script.

[matzew]

But, didn't we also notice that even on the stuck situation, on a broker cluster, things start to work just fine again, if the reconcilation is triggered? e.g. as we provided manually a new Broker or KafkaSink on our servers?

When we did that the CM was completely untouched. So, I guess, that's fine? But not sure

But we are not in a rush here to merge this

[mgencur]

But, didn't we also notice that even on the stuck situation, on a broker cluster, things start to work just fine again, if the reconcilation is triggered?

That's true. Anyway, the upgrade tests should show if this works. I'd probably run them at least twice on this PR.

[matzew]

@mgencur the kafka-broker-brokers-triggers CM has no ownerReferences: set

[openshift-ci]

@creydr: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/414-upstream-e2e-kafka-aws-414	ca3472c	link	false	/test 414-upstream-e2e-kafka-aws-414

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[matzew]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: creydr, matzew

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [matzew]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Cali0707]

So IIUC we could have the case, that we update the configmaps while still the 1.11 kafka-controller running and reconciling the configmaps back to include the trustBundles fields 🤔 (but I saw @Cali0707 changing this in 8136b0d 🤷 )

@matzew I don't think that commit was necessary - basically the tests were failing for me on the cluster I was using to test and I wasn't sure why, so I was trying some stuff out and I forgot to revert that commit. I don't think that the order change fixed anything though. I guess there is a risk that the timing might work out so that the 1.11 kafka-controller re reconciles the configmap after the script runs