Use sarama config build in source #337

aliok · 2021-01-27T10:14:52Z

Fixes #298

Proposed Changes

Use Sarama field in config-kafka in KafkaSource as defaults

Release Note

- 🎁 Add new feature
It is now possible to define Sarama config defaults for KafkaSource in `config-kafka` configmap with a `sarama` field.

Docs

codecov · 2021-01-27T10:41:24Z

Codecov Report

Merging #337 (cef26ed) into master (70becc9) will decrease coverage by 2.34%.
The diff coverage is 54.20%.

@@            Coverage Diff             @@
##           master     #337      +/-   ##
==========================================
- Coverage   75.27%   72.92%   -2.35%     
==========================================
  Files         124      128       +4     
  Lines        4792     4975     +183     
==========================================
+ Hits         3607     3628      +21     
- Misses        962     1114     +152     
- Partials      223      233      +10

Impacted Files	Coverage Δ
...consolidated/reconciler/controller/kafkachannel.go	`43.43% <0.00%> (ø)`
pkg/common/client/admin_client.go	`0.00% <0.00%> (ø)`
pkg/source/reconciler/source/controller.go	`0.00% <0.00%> (ø)`
pkg/source/reconciler/source/kafkasource.go	`0.00% <ø> (ø)`
pkg/source/adapter/adapter.go	`62.29% <16.66%> (-3.81%)`	⬇️
pkg/source/client/client.go	`39.39% <30.00%> (ø)`
pkg/common/scheduler/statefulset/autoscaler.go	`63.82% <45.45%> (-9.98%)`	⬇️
pkg/source/reconciler/source/config_watcher.go	`66.03% <66.03%> (ø)`
pkg/source/mtadapter/adapter.go	`70.37% <90.00%> (-0.56%)`	⬇️
pkg/common/scheduler/statefulset/snapshot.go	`76.47% <100.00%> (+7.23%)`	⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70becc9...798f36c. Read the comment docs.

aliok · 2021-02-08T10:48:03Z

/test pull-knative-sandbox-eventing-kafka-integration-test-channel-consolidated

slinkydeveloper · 2021-02-09T15:04:41Z

/lgtm

/hold waiting for another review

aliok · 2021-02-10T07:27:45Z

/assign @travis-minke-sap @eric-sap @matzew

knative-prow-robot · 2021-02-10T07:27:46Z

@aliok: GitHub didn't allow me to assign the following users: eric-sap.

Note that only knative-sandbox members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @travis-minke-sap @eric-sap @matzew

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

travis-minke-sap · 2021-02-10T16:33:16Z

Looks good to me! (I'm not super familiar with the Kafka Source though)

Great to see more alignment. I think there might be potential for further consolidation of all the ConfigMapWatchers (and SecretWatchers coming with #373 and #363) but we can address that when they're all in place or maybe during #363 have a look?

@eric-sap is going to review this as well - he's a bit more familiar with the Sarama Config handling ; )

eric-sap

Nothing problematic from the config handling perspective that I see. Made some minor comments as I was looking.

pkg/source/reconciler/source/config_watcher.go

eric-sap · 2021-02-10T17:25:44Z

pkg/source/reconciler/source/config_watcher.go

+	envs = maybeAppendEnvVar(envs, cw.kafkaConfigEnvVar(), cw.KafkaConfig() != nil)
+
+	return envs
+}
+
+// maybeAppendEnvVar appends an EnvVar only if the condition boolean is true.
+func maybeAppendEnvVar(envs []corev1.EnvVar, env corev1.EnvVar, cond bool) []corev1.EnvVar {
+	if !cond {
+		return envs
+	}
+
+	return append(envs, env)
+}


Are there more usages planned for this function? As it is it looks like an overly complicated way of doing this (though nothing technically amiss)

Suggested change

envs = maybeAppendEnvVar(envs, cw.kafkaConfigEnvVar(), cw.KafkaConfig() != nil)

return envs

}

// maybeAppendEnvVar appends an EnvVar only if the condition boolean is true.

func maybeAppendEnvVar(envs []corev1.EnvVar, env corev1.EnvVar, cond bool) []corev1.EnvVar {

if !cond {

return envs

}

return append(envs, env)

}

if cw.KafkaConfig() != nil {

envs = append(envs, cw.kafkaConfigEnvVar())

}

return envs

}

Frankly, it is a copy paste of a private function in eventing. Let me see if I can make that one in eventing public.

Hmm, I will check that after this PR as a follow up.

pkg/source/reconciler/source/config_watcher.go

lionelvillard · 2021-02-11T13:19:53Z

@aliok ignore pull-knative-sandbox-eventing-kafka-integration-test-mt-source. It has been enabled yesterday. I'll take a look at it ASAP.

pkg/source/reconciler/source/config_watcher.go

matzew · 2021-02-12T09:36:13Z

pkg/channel/consolidated/reconciler/controller/kafkachannel.go

+		WithDefaults().
+		WithAuth(kafkaAuthCfg).
+		WithClientId(clientID).
+		FromYaml(kafkaConfig.SaramaSettingsYamlString).


Can we just update the MakeAdminClient to have that option passed in? or not, if nil?

Feels a bit weird to duplicate that code

Not duped anymore. Moved into a common place

matzew · 2021-02-12T09:36:52Z

pkg/source/client/client.go

@@ -111,6 +126,7 @@ func MakeAdminClient(ctx context.Context, clientID string, kafkaAuthCfg *client.
 		WithDefaults().
 		WithAuth(kafkaAuthCfg).
 		WithClientId(clientID).
+		FromYaml(kafkaConfig.SaramaSettingsYamlString).


Isn't that the public MakeAdminClient I talked about above ?

To prevent informer injection happening whenever another package uses this code

…reated before for config Kafka source client

knative-metrics-robot · 2021-02-15T08:04:35Z

The following is the coverage report on the affected files.
Say /test pull-knative-sandbox-eventing-kafka-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/common/client/admin_client.go	Do not exist	0.0%
pkg/source/client/client.go	Do not exist	48.0%
pkg/source/reconciler/source/config_watcher.go	Do not exist	83.3%
pkg/source/reconciler/source/controller.go	Do not exist	0.0%
pkg/source/reconciler/source/kafkasource.go	Do not exist	0.0%

knative-prow-robot · 2021-02-15T08:04:59Z

@aliok: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-knative-sandbox-eventing-kafka-go-coverage	`798f36c`	link	`/test pull-knative-sandbox-eventing-kafka-go-coverage`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

matzew · 2021-02-15T08:36:26Z

pkg/channel/consolidated/reconciler/controller/kafkachannel.go

@@ -592,7 +591,7 @@ func (r *Reconciler) updateKafkaConfig(ctx context.Context, configMap *corev1.Co
 	r.kafkaConfig = kafkaConfig
 	r.kafkaConfigError = err
 	ac, err := kafka.NewAdminClient(ctx, func() (sarama.ClusterAdmin, error) {
-		return source.MakeAdminClient(ctx, controllerAgentName, r.kafkaAuthConfig, kafkaConfig)
+		return client.MakeAdminClient(ctx, controllerAgentName, r.kafkaAuthConfig, kafkaConfig.SaramaSettingsYamlString, kafkaConfig.Brokers)


matzew

/lgtm
/approve

knative-prow-robot · 2021-02-15T08:36:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aliok, matzew

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [aliok,matzew]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aliok · 2021-02-15T10:32:47Z

@slinkydeveloper /unhold?

slinkydeveloper · 2021-02-15T12:45:45Z

/unhold

To reduce the noise for other approvers on automatically requested reviews

knative-prow-robot requested review from grantr and vaikas January 27, 2021 10:15

google-cla bot added the cla: yes Indicates the PR's author has signed the CLA. label Jan 27, 2021

aliok force-pushed the use-sarama-config-build-in-source branch 2 times, most recently from 93cfbd8 to f8833d2 Compare January 27, 2021 10:19

knative-prow-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 27, 2021

knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 2, 2021

aliok force-pushed the use-sarama-config-build-in-source branch from d3d4456 to 38b1e0f Compare February 4, 2021 12:55

knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 4, 2021

aliok force-pushed the use-sarama-config-build-in-source branch from 7ec4112 to fc6ffc4 Compare February 9, 2021 07:58

aliok changed the title ~~[WIP] Use sarama config build in source~~ Use sarama config build in source Feb 9, 2021

knative-prow-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 9, 2021

knative-prow-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 9, 2021

knative-prow-robot assigned slinkydeveloper Feb 9, 2021

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 9, 2021

knative-prow-robot assigned matzew and travis-minke-sap Feb 10, 2021

eric-sap reviewed Feb 10, 2021

View reviewed changes

matzew reviewed Feb 12, 2021

View reviewed changes

pkg/source/reconciler/source/config_watcher.go Show resolved Hide resolved

matzew reviewed Feb 12, 2021

View reviewed changes

aliok added 8 commits February 12, 2021 14:12

Things working - gotta cleanup

3de6db4

Fix build

3066cd3

Compose super type when watching configs

7fd5862

Add unit tests to new Kafka config watcher

5c2a231

Remove ref to source package in consolidated channel

8314e8b

Docs, code improvement, better error handling

b42b809

Address comments

fea4f59

Move the kafkasource client.go into separate package

276bca6

To prevent informer injection happening whenever another package uses this code

aliok force-pushed the use-sarama-config-build-in-source branch from 4aa0bc8 to 276bca6 Compare February 12, 2021 11:13

aliok added 4 commits February 12, 2021 14:16

Add missing headers

8817918

Deduplicate some code, now possible because of the explicit package c…

ac305e7

…reated before for config Kafka source client

Move admin client maker into common package

ea08022

Fix import cycles

798f36c

matzew reviewed Feb 15, 2021

View reviewed changes

matzew approved these changes Feb 15, 2021

View reviewed changes

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 15, 2021

knative-prow-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 15, 2021

knative-prow-robot merged commit 9022563 into knative-extensions:master Feb 15, 2021

aliok deleted the use-sarama-config-build-in-source branch February 16, 2021 07:35

aliok mentioned this pull request Feb 16, 2021

Get rid of maybeAppendVar func #392

Merged

matzew pushed a commit to matzew/eventing-kafka that referenced this pull request Sep 23, 2021

Reduce reviewers to core eventing team (knative-extensions#337)

490c086

To reduce the noise for other approvers on automatically requested reviews

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use sarama config build in source #337

Use sarama config build in source #337

aliok commented Jan 27, 2021

codecov bot commented Jan 27, 2021 •

edited

Loading

aliok commented Feb 8, 2021

slinkydeveloper commented Feb 9, 2021

aliok commented Feb 10, 2021

knative-prow-robot commented Feb 10, 2021

travis-minke-sap commented Feb 10, 2021

eric-sap left a comment

eric-sap Feb 10, 2021

aliok Feb 11, 2021

aliok Feb 11, 2021

aliok Feb 16, 2021

lionelvillard commented Feb 11, 2021

matzew Feb 12, 2021

aliok Feb 15, 2021

matzew Feb 12, 2021

knative-metrics-robot commented Feb 15, 2021

knative-prow-robot commented Feb 15, 2021 •

edited

Loading

matzew Feb 15, 2021

matzew left a comment

knative-prow-robot commented Feb 15, 2021

aliok commented Feb 15, 2021

slinkydeveloper commented Feb 15, 2021

Use sarama config build in source #337

Use sarama config build in source #337

Conversation

aliok commented Jan 27, 2021

Proposed Changes

codecov bot commented Jan 27, 2021 • edited Loading

Codecov Report

aliok commented Feb 8, 2021

slinkydeveloper commented Feb 9, 2021

aliok commented Feb 10, 2021

knative-prow-robot commented Feb 10, 2021

travis-minke-sap commented Feb 10, 2021

eric-sap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lionelvillard commented Feb 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knative-metrics-robot commented Feb 15, 2021

knative-prow-robot commented Feb 15, 2021 • edited Loading

Choose a reason for hiding this comment

matzew left a comment

Choose a reason for hiding this comment

knative-prow-robot commented Feb 15, 2021

aliok commented Feb 15, 2021

slinkydeveloper commented Feb 15, 2021

codecov bot commented Jan 27, 2021 •

edited

Loading

knative-prow-robot commented Feb 15, 2021 •

edited

Loading