-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KafakV2SourceConnector #39410
Merged
Merged
KafakV2SourceConnector #39410
Changes from 45 commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
f0c05ac
kafkaV2SourceConnector
7b5fc7f
Merge branch 'main' into KafkaV2SourceConnector
59f2cd4
changes
5ba9587
Merge branch 'main' into KafkaV2SourceConnector
e1a7378
add validator and refactor
bb3cec8
add split merge handling in source task
031ff0c
add tests
f78d910
cspell change
375365f
Merge branch 'main' into KafkaV2SourceConnector
7b42cca
fix
0d39f9d
add tests and resolve few comments
67b0671
add tests
47bcc62
fix compilation
b272054
fix tests
d54ba23
refactor
f574cdc
fix
be4c14a
add config reference config, update changelog and fix spotbugs
034473a
fix
4090abf
fix spell check errors
ceda53d
change
704ec9d
update test.yml
89d6208
Merge branch 'main' into UsingTestContainersForE2ETests
0e61917
add testContainer tests
4ab1e43
Merge branch 'main' into KafkaV2SourceConnector
37b797d
resolve comments
530d418
Merge branch 'KafkaV2SourceConnector' into UsingTestContainersForE2ET…
a53c01f
fix
1fbb8a2
Merge branch 'KafkaV2SourceConnector' into UsingTestContainersForE2ET…
02b3b83
fix compile
1cbb560
refactor
d71cb71
merge from main and resolve conflicts
908b5f2
update version
240d365
change
1b092ae
Merge branch 'main' into UsingTestContainersForE2ETests
283dabf
update external dependencies
ce5d82e
update external dependencies
417a6f3
KafkaV2SourceConnector
9dfe2a0
merge from main
dc1c3a7
remove azure cosmos implementation package dependency
fb5296d
Merge remote-tracking branch 'origin/UsingTestContainersForE2ETests' …
e5ae8a6
fix tests
75e5d8f
Merge branch 'main' into KafkaV2SourceConnector-2
d9d9a25
change to use ImplementationBridgeHelpers pattern
389e20b
fix tests
2ddff13
suppress style check for shaded classes from apache common and guava
9dcd348
remove guava/apachecommon package
10f8bca
update tests.yml file
3b97b2f
refactor
6a25454
fix build issue
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
sdk/cosmos/azure-cosmos-kafka-connect/doc/configuration-reference.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| ## Configuration Reference: | ||
|
|
||
| ## Generic Configuration | ||
| | Config Property Name | Default | Description | | ||
| |:---------------------------------------------|:--------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `kafka.connect.cosmos.accountEndpoint` | None | Cosmos DB Account Endpoint Uri | | ||
| | `kafka.connect.cosmos.accountEndpoint` | None | Cosmos DB Account Key | | ||
| | `kafka.connect.cosmos.useGatewayMode` | `false` | Flag to indicate whether to use gateway mode. By default it is false. | | ||
| | `kafka.connect.cosmos.preferredRegionsList` | `[]` | Preferred regions list to be used for a multi region Cosmos DB account. This is a comma separated value (e.g., `[East US, West US]` or `East US, West US`) provided preferred regions will be used as hint. You should use a collocated kafka cluster with your Cosmos DB account and pass the kafka cluster region as preferred region. See list of azure regions [here](https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.locationnames?view=azure-dotnet&preserve-view=true). | | ||
| | `kafka.connect.cosmos.applicationName` | `""` | Application name. Will be added as the userAgent suffix. | | ||
|
|
||
| ## Source Connector Configuration | ||
| | Config Property Name | Default | Description | | ||
| |:----------------------------------------------------------|:-------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `kafka.connect.cosmos.source.database.name` | None | Cosmos DB database name. | | ||
| | `kafka.connect.cosmos.source.containers.includeAll` | `false` | Flag to indicate whether reading from all containers. | | ||
| | `kafka.connect.cosmos.source.containers.includedList` | `[]` | Containers included. This config will be ignored if kafka.connect.cosmos.source.includeAllContainers is true. | | ||
| | `kafka.connect.cosmos.source.containers.topicMap` | `[]` | A comma delimited list of Kafka topics mapped to Cosmos containers. For example: topic1#con1,topic2#con2. By default, container name is used as the name of the kafka topic to publish data to, can use this property to override the default config | | ||
| | `kafka.connect.cosmos.source.changeFeed.startFrom` | `Beginning` | ChangeFeed Start from settings (Now, Beginning or a certain point in time (UTC) for example 2020-02-10T14:15:03) - the default value is 'Beginning'. | | ||
| | `kafka.connect.cosmos.source.changeFeed.mode` | `LatestVersion` | ChangeFeed mode (LatestVersion or AllVersionsAndDeletes). | | ||
| | `kafka.connect.cosmos.source.changeFeed.maxItemCountHint` | `1000` | The maximum number of documents returned in a single change feed request. But the number of items received might be higher than the specified value if multiple items are changed by the same transaction. | | ||
| | `kafka.connect.cosmos.source.metadata.poll.delay.ms` | `300000` | Indicates how often to check the metadata changes (including container split/merge, adding/removing/recreated containers). When changes are detected, it will reconfigure the tasks. Default is 5 minutes. | | ||
| | `kafka.connect.cosmos.source.metadata.storage.topic` | `_cosmos.metadata.topic` | The name of the topic where the metadata are stored. The metadata topic will be created if it does not already exist, else it will use the pre-created topic. | | ||
| | `kafka.connect.cosmos.source.messageKey.enabled` | `true` | Whether to set the kafka record message key. | | ||
| | `kafka.connect.cosmos.source.messageKey.field` | `id` | The field to use as the message key. | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| connectors/ | ||
| log.txt | ||
|
|
||
| # Exclude all temporary files in resources | ||
| !resources/*example | ||
| resources/sink.properties | ||
| resources/source.properties | ||
| resources/standalone.properties |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Build the Cosmos DB Connectors on top of the Kafka Connect image | ||
| FROM confluentinc/cp-kafka-connect:7.5.0 | ||
|
|
||
| # Install datagen connector | ||
| RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest | ||
|
|
||
| COPY connectors/ /etc/kafka-connect/jars |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Disabling this one is bad (for the entire file) - would it be possible to use this pattern? Directly from the code - scoped to just the method violating the rule?
If not, this is fine...
azure-sdk-for-java/sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/apachecommons/math/exception/util/LocalizedFormats.java
Lines 44 to 45 in 38cf774
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will check
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will track in the next PR