🎉 New Destination: Elasticsearch #7005
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
github-actions
bot
added
area/connectors
|
Amazing @jdbranham we're going to review this contribution tomorrow or max by Friday. Can you fill the checklst so we can see what step you are? |
|
Thanks @marcosmarxm - It's still a WIP, but I'll get the checklist updated by tomorrow, and hope to have the basic functionality working end-to-end |
adds test logging config to inspect http wire begins work for overrwriting existing records
This comment has been minimized.
This comment has been minimized.
.../src/main/java/io/airbyte/integrations/destination/elasticsearch/ConnectorConfiguration.java
Outdated
Show resolved
Hide resolved
.../src/main/java/io/airbyte/integrations/destination/elasticsearch/ConnectorConfiguration.java
Outdated
Show resolved
Hide resolved
...src/main/java/io/airbyte/integrations/destination/elasticsearch/ElasticsearchConnection.java
Outdated
Show resolved
Hide resolved
|
Ready! |
|
@alexandr-shegeda / @VitaliiMaltsev is it pssible to review this contribution again? |
|
Full E2E test with MySql source to Enterprise Elasticsearch instance - |
@marcosmarxm sure, will review it today again |
|
/test connector=connectors/destination-elasticsearch
|
VitaliiMaltsev
approved these changes
Oct 27, 2021
|
@VitaliiMaltsev you'll need to publish from a separate PR that is on the origin repo, this PR is coming from a forked repo. See the Community contribution handbook for more info |
# Conflicts: # airbyte-config/init/src/main/resources/seed/destination_definitions.yaml
lmossman
pushed a commit
that referenced
this pull request
Nov 3, 2021
* feat: adds destination-elasticsearch * feat: adds destination-elasticsearch es server container * refactor: header configuration * update: only call createIndex when preparing the writes * update: reuse container * fix: make index names valid and use namespace * refactor: use bulk process and buffered consumer * refactor: fix bulk process and buffered consumer * chore: update documentation * update: remove ssl reference * fix: bulk indexing adds test logging config to inspect http wire begins work for overrwriting existing records * docs: update for authentication * refactor: simplify config * refactor: cleanup indices, implement auth * update: cleanup equals/toString in Elasticsearch ConnectionConfiguration * chore: use conventions and remove unused code * update: close underlying rest connection * update: enable `supportsNormalization` * refactor: better encapsulate index naming * update: allow upserting * update: use oneOf for auth method * refactor: use encapsulated auth object * chore: pretty * update: simplify auth header creation * chore: remove unused class * update: use boolean as field type * adds: elasticsearch example server * fix: api secret test
schlattk
pushed a commit
to schlattk/airbyte
that referenced
this pull request
Jan 4, 2022
* feat: adds destination-elasticsearch * feat: adds destination-elasticsearch es server container * refactor: header configuration * update: only call createIndex when preparing the writes * update: reuse container * fix: make index names valid and use namespace * refactor: use bulk process and buffered consumer * refactor: fix bulk process and buffered consumer * chore: update documentation * update: remove ssl reference * fix: bulk indexing adds test logging config to inspect http wire begins work for overrwriting existing records * docs: update for authentication * refactor: simplify config * refactor: cleanup indices, implement auth * update: cleanup equals/toString in Elasticsearch ConnectionConfiguration * chore: use conventions and remove unused code * update: close underlying rest connection * update: enable `supportsNormalization` * refactor: better encapsulate index naming * update: allow upserting * update: use oneOf for auth method * refactor: use encapsulated auth object * chore: pretty * update: simplify auth header creation * chore: remove unused class * update: use boolean as field type * adds: elasticsearch example server * fix: api secret test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Adds a new Elasticsearch destination
How
This uses the
elasticsearch-javarest client from the Elasticsearch team -https://github.com/elastic/elasticsearch-java/
Read more in the
bootstrap.mdRecommended reading order
ElasticsearchConnection.javaElasticsearchAirbyteMessageConsumerFactory.javaElasticsearchDestination.javaPre-merge Checklist
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/SUMMARY.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described here