Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-8407: Fix validation of class and list configs in connector client overrides #6789

Merged
merged 1 commit into from May 23, 2019

Conversation

@C0urante
Copy link
Contributor

commented May 22, 2019

Jira

Because of how config values are converted into strings in the AbstractHerder.validateClientOverrides() method after being validated by the client override policy, an exception is thrown if the value returned by the policy isn't already parsed as the type expected by the client ConfigDef. A more thorough writeup of how this happens is available in the linked Jira ticket.

The fix here involves parsing client override properties before passing them to the override policy.

A unit test is added to ensure that several different types of configs are validated properly by the herder.

This bug fix should be included in the recently-cut 2.3 branch.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)
@C0urante

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2019

@mageshn, since this was introduced by KIP-458 would you mind taking a look? :)

@mageshn
Copy link
Contributor

left a comment

Nice catch @C0urante. LGTM.

@C0urante

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2019

@rhauch, could you take a look at this when you have a moment?

@rhauch
Copy link
Contributor

left a comment

Thanks, @C0urante. One question below.

@rhauch

rhauch approved these changes May 23, 2019

Copy link
Contributor

left a comment

LGTM. Thanks, @C0urante!

@rhauch rhauch merged commit 5351efe into apache:trunk May 23, 2019

2 checks passed

JDK 11 and Scala 2.12 SUCCESS 11265 tests run, 67 skipped, 0 failed.
Details
JDK 8 and Scala 2.11 SUCCESS 11265 tests run, 67 skipped, 0 failed.
Details

rhauch added a commit that referenced this pull request May 23, 2019

KAFKA-8407: Fix validation of class and list configs in connector cli…
…ent overrides (#6789)

Because of how config values are converted into strings in the `AbstractHerder.validateClientOverrides()` method after being validated by the client override policy, an exception is thrown if the value returned by the policy isn't already parsed as the type expected by the client `ConfigDef`. The fix here involves parsing client override properties before passing them to the override policy.

A unit test is added to ensure that several different types of configs are validated properly by the herder.

Author: Chris Egerton <chrise@confluent.io>
Reviewers: Magesh Nandakumar <magesh.n.kumar@gmail.com>, Randall Hauch <rhauch@gmail.com>

@C0urante C0urante deleted the C0urante:kafka-8407 branch May 23, 2019

haidangdam added a commit to haidangdam/kafka that referenced this pull request May 29, 2019

KAFKA-8407: Fix validation of class and list configs in connector cli…
…ent overrides (apache#6789)

Because of how config values are converted into strings in the `AbstractHerder.validateClientOverrides()` method after being validated by the client override policy, an exception is thrown if the value returned by the policy isn't already parsed as the type expected by the client `ConfigDef`. The fix here involves parsing client override properties before passing them to the override policy.

A unit test is added to ensure that several different types of configs are validated properly by the herder.

Author: Chris Egerton <chrise@confluent.io>
Reviewers: Magesh Nandakumar <magesh.n.kumar@gmail.com>, Randall Hauch <rhauch@gmail.com>

Pengxiaolong added a commit to Pengxiaolong/kafka that referenced this pull request Jun 14, 2019

KAFKA-8407: Fix validation of class and list configs in connector cli…
…ent overrides (apache#6789)

Because of how config values are converted into strings in the `AbstractHerder.validateClientOverrides()` method after being validated by the client override policy, an exception is thrown if the value returned by the policy isn't already parsed as the type expected by the client `ConfigDef`. The fix here involves parsing client override properties before passing them to the override policy.

A unit test is added to ensure that several different types of configs are validated properly by the herder.

Author: Chris Egerton <chrise@confluent.io>
Reviewers: Magesh Nandakumar <magesh.n.kumar@gmail.com>, Randall Hauch <rhauch@gmail.com>

jsancio pushed a commit to jsancio/kafka that referenced this pull request Aug 6, 2019

Merge remote-tracking branch 'confluent/master'
* confluent/master:
  KAFKA-8422; Client should send OffsetForLeaderEpoch only if broker supports latest version (apache#6806)
  MINOR: Fix `toString` NPE in tableProcessorNode (apache#6807)
  KAFKA-8341. Retry Consumer group operation for NOT_COORDINATOR error (apache#6723)
  MINOR: Updated configuration docs with RocksDBConfigSetter#close (apache#6784)
  MINOR: fix Streams version-probing system test (apache#6764)
  MINOR: Fix a few compiler warnings (apache#6767)
  MINOR: add Kafka Streams upgrade notes for 2.3 release (apache#6758)
  MINOR: Update jackson to 2.9.9 (apache#6798)
  MINOR: Add 2.0, 2.1 and 2.2 to broker and client compat tests
  KAFKA-8371: Remove dependence on ReplicaManager from Partition (apache#6705)
  KAFKA-8407: Fix validation of class and list configs in connector client overrides (apache#6789)
  KAFKA-8415: Interface ConnectorClientConfigOverridePolicy needs to be excluded from class loading isolation (apache#6796)
  KAFKA-8417: Remove redundant network definition --net=host when starting testing docker containers (apache#6797)
  MINOR: Remove checking on original joined subscription within handleAssignmentMismatch (apache#6782)
  KAFKA-8229; Reset WorkerSinkTask offset commit interval after task commit (apache#6579)
  MINOR: Remove ControllerEventManager metrics on close (apache#6788)
  CONFLUENT: Update versions in system tests to use CE versions (apache#406)
  CONFLUENT: Disable support.metrics in log_dir_failure_test.py (apache#197)
  KAFKA-8399: bring back internal.leave.group.on.close config for KStream (apache#6779)

Conflicts:
* core/src/main/scala/kafka/cluster/Partition.scala
* core/src/main/scala/kafka/server/KafkaConfig.scala
* core/src/main/scala/kafka/server/ReplicaManager.scala
* core/src/test/scala/unit/kafka/cluster/PartitionTest.scala
* core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala
* core/src/test/scala/unit/kafka/server/epoch/OffsetsForLeaderEpochTest.scala
* gradle/dependencies.gradle

Files with compiler errors:
* MetadataServiceCoordinator
* MultiTenantRequestContextTest
* TierEpochStateReplicationTest
* TierEpochStateRevolvingReplicationTest
* TierUtils
* PartitionTest
* ReplicaManager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.