Standalone KafkaSource

[aavarghese]

Signed-off-by: aavarghese avarghese@us.ibm.com

Fixes #2046

Proposed Changes

• Deploy just the KafkaSource v2 controller and new statefulset dispatcher (no receiver) for users of KakfaSource

Release Note

Deploy just the KafkaSource v2 controller and new statefulset dispatcher (no receiver) for users of KakfaSource 

[knative-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Merging #2089 (1287b0f) into main (47eb7f4) will decrease coverage by 0.02%.
The diff coverage is 66.66%.

@@             Coverage Diff              @@
##               main    #2089      +/-   ##
============================================
- Coverage     66.54%   66.52%   -0.03%     
+ Complexity      677      676       -1     
============================================
  Files           142      142              
  Lines          9141     9144       +3     
  Branches        196      196              
============================================
  Hits           6083     6083              
- Misses         2652     2654       +2     
- Partials        406      407       +1     

Flag	Coverage Δ
java-unittests	82.54% <ø> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ontrol-plane/pkg/reconciler/consumer/controller.go	86.11% <66.66%> (-4.80%)	⬇️
...ispatcher/impl/http/WebClientCloudEventSender.java	60.46% <0.00%> (-2.33%)	⬇️
...a/broker/dispatcher/impl/RecordDispatcherImpl.java	90.44% <0.00%> (+0.56%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47eb7f4...1287b0f. Read the comment docs.

[matzew]

/assign @matzew

[pierDipi]

/lgtm

[aavarghese]

Getting this error

ansuvarghese$ k logs knative-kafka-storage-version-migrator--1-kxv4j -n knative-eventing
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.168510135Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.16868271Z","caller":"logging/config.go:117","message":"Logging level set to: info"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.168730257Z","caller":"logging/config.go:79","message":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.174438702Z","logger":"storage-migrator","caller":"migrate/main.go:60","message":"Migrating 2 group resources"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.174738273Z","logger":"storage-migrator","caller":"migrate/main.go:63","message":"Migrating group resource kafkasources.sources.knative.dev"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.277265501Z","logger":"storage-migrator","caller":"migrate/main.go:63","message":"Migrating group resource kafkachannels.messaging.knative.dev"}
{"severity":"EMERGENCY","timestamp":"2022-04-14T19:24:07.301721637Z","logger":"storage-migrator","caller":"migrate/main.go:65","message":"Failed to migrate: unable to fetch crd kafkachannels.messaging.knative.dev - customresourcedefinitions.apiextensions.k8s.io \"kafkachannels.messaging.knative.dev\" not found","stacktrace":"main.main\n\tknative.dev/pkg@v0.0.0-20220411234407-00c122e376d0/apiextensions/storageversion/cmd/migrate/main.go:65\nruntime.main\n\truntime/proc.go:255"}

because KafkaChannel CRD is not installed as part of kafka-source-bundle..... @pierDipi any ideas?

[aavarghese]

/lgtm

@pierDipi Sorry. Needed a few more changes for everything to work (b365c01). All is good with kafkasource now.

But, migrator jobs crashing fatally due to kafkachannel CRD missing in this source only setup (#2089 (comment))

[pierDipi]

Getting this error

ansuvarghese$ k logs knative-kafka-storage-version-migrator--1-kxv4j -n knative-eventing
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.168510135Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.16868271Z","caller":"logging/config.go:117","message":"Logging level set to: info"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.168730257Z","caller":"logging/config.go:79","message":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.174438702Z","logger":"storage-migrator","caller":"migrate/main.go:60","message":"Migrating 2 group resources"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.174738273Z","logger":"storage-migrator","caller":"migrate/main.go:63","message":"Migrating group resource kafkasources.sources.knative.dev"}
{"severity":"INFO","timestamp":"2022-04-14T19:24:07.277265501Z","logger":"storage-migrator","caller":"migrate/main.go:63","message":"Migrating group resource kafkachannels.messaging.knative.dev"}
{"severity":"EMERGENCY","timestamp":"2022-04-14T19:24:07.301721637Z","logger":"storage-migrator","caller":"migrate/main.go:65","message":"Failed to migrate: unable to fetch crd kafkachannels.messaging.knative.dev - customresourcedefinitions.apiextensions.k8s.io \"kafkachannels.messaging.knative.dev\" not found","stacktrace":"main.main\n\tknative.dev/pkg@v0.0.0-20220411234407-00c122e376d0/apiextensions/storageversion/cmd/migrate/main.go:65\nruntime.main\n\truntime/proc.go:255"}

because KafkaChannel CRD is not installed as part of kafka-source-bundle..... @pierDipi any ideas?

Can we patch the storage version migrator to ignore resources that are not found?

[pierDipi]

Here https://github.com/knative/pkg/blob/e325df66cb51d341b5d1142fa42f75539dfe290a/apiextensions/storageversion/migrator.go#L60-L63

[aavarghese]

@pierDipi latest changes for adding DP sourcev2 to bundle + release changes in aa8d0ba

[pierDipi]

LGTM

Verify job is failing https://github.com/knative-sandbox/eventing-kafka-broker/runs/6276506322?check_suite_focus=true due to changes in vendor/k8s.io/code-generator/*

[aavarghese]

LGTM

Verify job is failing https://github.com/knative-sandbox/eventing-kafka-broker/runs/6276506322?check_suite_focus=true due to changes in vendor/k8s.io/code-generator/*

Yeah. Reverting that codegen update to see if it resolves that job for this PR.

[aavarghese]

/retest-required

[pierDipi]

/test upgrade-tests_eventing-kafka-broker_main

[pierDipi]

/test upgrade-tests_eventing-kafka-broker_main

[aavarghese]

/test upgrade-tests_eventing-kafka-broker_main

[pierDipi]

Thanks!

Let's keep an eye on nightly releases (including checking what's inside yamls) so that we can capture any release problem early on.

/lgtm
/approve

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aavarghese, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aavarghese]

Thanks!

Let's keep an eye on nightly releases (including checking what's inside yamls) so that we can capture any release problem early on.

/lgtm /approve

Absolutely. Will do these next few days. Thanks @pierDipi for all the reviews!

[aavarghese]

/unhold