-
Notifications
You must be signed in to change notification settings - Fork 432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add option to use autogenerated id on index requests #510
base: master
Are you sure you want to change the base?
Conversation
@confluentinc It looks like @tvm18860 just signed our Contributor License Agreement. 👍 Always at your service, clabot |
d857080
to
28593ca
Compare
80e22be
to
cc713d6
Compare
Hello! I'm interested in this PR. How can I help to merge it? |
6919a6c
to
7e8f19b
Compare
I think this PR is back in a good state now, I had missed a checkstyle validation error after an earlier rebase. |
We would also find this very useful. Are there any comments from the reviewers about what it would take to get this added? Are there any issues with the approach? Does it just need to be updated again to pass the checks? |
…ssue in Jenkins build (confluentinc#669)
…ssue in Jenkins build (confluentinc#669) (confluentinc#671)
…afka-connect-elasticsearch into feature/autogen-doc-ids
|
Problem
While setting the document ID when indexing does provide exactly once delivery, it does put more load on Elasticsearch and is not necessary for all use cases.
A PR for this issue has been made before (#393). This PR is largely an update to that one, as there were many merge conflicts that needed resolving there as it fell out of date.
Addresses #139 and #97
Solution
Add a new option to use the autogenerated document id on index requests. The new option (
use.autogenerated.ids
) will default tofalse
and only be applicable whenwrite.method
is set toINSERT
.Note that the large diff in the
DataCoverter
class on theconvertRecord
method is a result of having to pull a chunk of that code out into a separate method. The checkstyle plugin was throwing errors when I added an extraif
statement in that the cyclomatic complexity got too high. The more relevant change is on lines 169-173.Does this solution apply anywhere else?
If yes, where?
Test Strategy
Testing done:
We are running live connectors with this change now.
Release Plan