[Flink-26884] Copy elasticsearch connectors to the external repo #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This closes #3112.
…rors. Covered exceptions are: Timeouts, No Master, UnavailableShardsException, bulk queue on node full
…csearchSink This commit fixes both FLINK-5353 and FLINK-5122. It allows users to implement a failure handler to control how failed action requests are dealt with. The commit also includes general improvements to FLINK-5122: 1. Use the built-in backoff functionality in the Elasticsearch BulkProcessor (not available for Elasticsearch 1.x) 2. Migrate the `checkErrorAndRetryBulk` functionality to the new failure handler
This closes #3358.
…a-version.sh Use scala.binary.version as defined in the parent POM and remove the script to swap scala version identifiers. This closes #3800
These ASM dependencies where shaded into the elasticsearch-base module which where then clashing with our newer (also shaded) ASM dependency.
…lasticSearch connector - Removes Serializable from the RequestIndexer, because they are neither required to be serializable (they are created in open()) nor is the main implementation (BulkProcessorIndexer) actually serializable. - Makes BulkFlushBackoffPolicy a static inner class, which avoids adding outer class during serialization and clears various warnings about raw reference to outer class This closes #4298
This commit changes the behaviour of the WebRuntimeMonitor to not longer block serving threads by waiting on the result of futures. Instead the RequestHandler now returns a CompletableFuture<FullHttpResponse> which is written out to the Netty channel upon completion. This will improve the performance of our WebRuntimeMonitor. This closes #4527.
This closes #4865.
Fix typos from the IntelliJ "Typos" inspection. This closes #5242
…tBase The AbstractTestBase fully subsumes the functionality of the StreamingMultipleProgramsTestBase since it now is the most general test base for streaming and batch jobs. As a consequence, we can safely remove the StreamingMultipleProgramsTestBase and let all corresponding tests extend from AbstractTestBase. This closes #4896.
…connector This closes #5124.
This closes #5378. This closes #4847. This closes #5305. This closes #5208. This closes #2192. This closes #2422. This closes #3478.
…ng with Elasticsearch 5.3+ This closes #6043.
…6.x connector This closes #6391.
…ncompatibility This commit fixes the binary incompatibility for UpdateRequests in Elasticsearch. This is due to a binary compatibility issue between the base module (which is compiled against a very old ES version and the current Elasticsearch version). It lets the API call bridge also provide the RequestIndexer version-specific. This closes #6682.
…csearch This commit adds full support for Elasticsearch to be used with Table & SQL API as well as SQL Client. It includes: - Elasticsearch 6 upsert table sink (for append-only and updating queries) - Elasticsearch 6 table factory - Elasticsearch table descriptors & validators - Unit tests, SQL Client end-to-end test - Website documentation This closes #6611.
…-table-common This commit makes the flink-formats module Scala free by introducing a flink-table-common module that is implemented in Java. This module contains all classes that are required across different Maven modules. Additionally, all classes in this module have been annotated with @internal and @PublicEvolving accordingly. Since the methods in Descriptor were declared with private[flink] visibility, they have been migrated to a new toProperties() method that is public. This closes #6958.
Set the system property jna.nosys=true to avoid ElasticsearchSinkITCase test failures due a wrong JNA version.
…on(now() and current_timestamp) in index pattern
…cies to 7.10.2 to keep Apache 2 license
flink-connector-base was previously inconsistently used in connectors (directly shaded in some and transitively pulled in via flink-connector-files which was itself shaded in the table uber jar). FLINK-24687 moved flink-connector-files out from the flink-table uber jar. This commit implements a combined approach for ensuring a smooth transition for both Flink users and for external connector developers, as outlined in this: - all internal Flink connectors that depend on flink-connector-base now shade and relocate it - for compatibility, until external developers implement the same change, flink-connector-base is also included into flink-dist
…antee to AT_LEAST_ONCE
…tor-base to flink-dist
…on to SinkFunction-based one
Since `TableConfig` is a `WritableConfig`, callers should directly call `TableConfig.set()` and avoid going throught `#getConfiguration()` which is there only for advanced internal configuration read value purposes.
…-retry-timeout' option This closes #19202.
… Elasticsearch 6/7 connectors (tests only) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
MartijnVisser
approved these changes
Apr 4, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
With this PR, Elasticsearch connectors will be coped to the external repo. Please refer to the previous draft PR #2 for more information.
Brief change log
flink-connector-elasticsearch-base,flink-connector-elasticsearch6,flink-connector-elasticsearch7flink-sql-connector-elasticsearch6,flink-sql-connector-elasticsearch7Verifying this change
It can be verified by checking if "mvn clean install" works successfully.