From 104240bf173db362cd9cef91ac3af5c8c414fc02 Mon Sep 17 00:00:00 2001 From: Michele Rastelli Date: Wed, 22 Nov 2023 20:12:11 +0100 Subject: [PATCH 1/2] Kafka Connector: batch writes --- .../kafka-connect-arangodb-sink-connector/_index.md | 8 +++----- .../configuration.md | 7 +++++++ .../kafka-connect-arangodb-sink-connector/_index.md | 8 +++----- .../configuration.md | 7 +++++++ .../kafka-connect-arangodb-sink-connector/_index.md | 8 +++----- .../configuration.md | 7 +++++++ 6 files changed, 30 insertions(+), 15 deletions(-) diff --git a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 80031db41f..7bed7b953d 100644 --- a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice) +- ArangoDB 3.11 or higher versions (BTS-615) ## Installation @@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options. ## Limitations - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619) -- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627) -- In case of transient error, the entire Kafka Connect batch is retried (DE-651) - Record values are required to be object-like structures (DE-644) - Auto-creation of ArangoDB collection is not supported (DE-653) - `ssl.cert.value` does not support multiple certificates (DE-655) -- Batch inserts are not guaranteed to be executed serially (FRB-300) -- Batch inserts may succeed for some documents while failing for others (FRB-300) +- Batch writes are not guaranteed to be executed serially (FRB-300) +- Batch writes may succeed for some documents while failing for others (FRB-300) This has two important consequences: - Transient errors might be retried and succeed at a later point - Data errors might be asynchronously reported to the DLQ diff --git a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index ef3583ca62..3f2062244a 100644 --- a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -173,6 +173,13 @@ to `update`: - `true`: objects are merged - `false`: existing document fields are overwritten +### batch.size + +- type: _int_ +- default: `3_000` + +Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. + ### insert.timeout.ms - type: _int_ diff --git a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 80031db41f..7bed7b953d 100644 --- a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice) +- ArangoDB 3.11 or higher versions (BTS-615) ## Installation @@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options. ## Limitations - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619) -- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627) -- In case of transient error, the entire Kafka Connect batch is retried (DE-651) - Record values are required to be object-like structures (DE-644) - Auto-creation of ArangoDB collection is not supported (DE-653) - `ssl.cert.value` does not support multiple certificates (DE-655) -- Batch inserts are not guaranteed to be executed serially (FRB-300) -- Batch inserts may succeed for some documents while failing for others (FRB-300) +- Batch writes are not guaranteed to be executed serially (FRB-300) +- Batch writes may succeed for some documents while failing for others (FRB-300) This has two important consequences: - Transient errors might be retried and succeed at a later point - Data errors might be asynchronously reported to the DLQ diff --git a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index ef3583ca62..3f2062244a 100644 --- a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -173,6 +173,13 @@ to `update`: - `true`: objects are merged - `false`: existing document fields are overwritten +### batch.size + +- type: _int_ +- default: `3_000` + +Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. + ### insert.timeout.ms - type: _int_ diff --git a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 80031db41f..7bed7b953d 100644 --- a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice) +- ArangoDB 3.11 or higher versions (BTS-615) ## Installation @@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options. ## Limitations - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619) -- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627) -- In case of transient error, the entire Kafka Connect batch is retried (DE-651) - Record values are required to be object-like structures (DE-644) - Auto-creation of ArangoDB collection is not supported (DE-653) - `ssl.cert.value` does not support multiple certificates (DE-655) -- Batch inserts are not guaranteed to be executed serially (FRB-300) -- Batch inserts may succeed for some documents while failing for others (FRB-300) +- Batch writes are not guaranteed to be executed serially (FRB-300) +- Batch writes may succeed for some documents while failing for others (FRB-300) This has two important consequences: - Transient errors might be retried and succeed at a later point - Data errors might be asynchronously reported to the DLQ diff --git a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index ef3583ca62..3f2062244a 100644 --- a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -173,6 +173,13 @@ to `update`: - `true`: objects are merged - `false`: existing document fields are overwritten +### batch.size + +- type: _int_ +- default: `3_000` + +Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. + ### insert.timeout.ms - type: _int_ From e3bddd29c55d2f5edc7cea973f99656baea03d1c Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 23 Nov 2023 17:12:17 +0100 Subject: [PATCH 2/2] Review --- .../kafka-connect-arangodb-sink-connector/_index.md | 2 +- .../kafka-connect-arangodb-sink-connector/configuration.md | 3 ++- .../kafka-connect-arangodb-sink-connector/_index.md | 2 +- .../kafka-connect-arangodb-sink-connector/configuration.md | 3 ++- .../kafka-connect-arangodb-sink-connector/_index.md | 2 +- .../kafka-connect-arangodb-sink-connector/configuration.md | 3 ++- 6 files changed, 9 insertions(+), 6 deletions(-) diff --git a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 7bed7b953d..8b21e9d99b 100644 --- a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- ArangoDB 3.11 or higher versions (BTS-615) +- ArangoDB 3.11.1 and higher versions ## Installation diff --git a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index 3f2062244a..5ae7e5a239 100644 --- a/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -178,7 +178,8 @@ to `update`: - type: _int_ - default: `3_000` -Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. +Specifies how many records to attempt to batch together for insertion or deletion +into the destination collection. ### insert.timeout.ms diff --git a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 7bed7b953d..8b21e9d99b 100644 --- a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- ArangoDB 3.11 or higher versions (BTS-615) +- ArangoDB 3.11.1 and higher versions ## Installation diff --git a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index 3f2062244a..5ae7e5a239 100644 --- a/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -178,7 +178,8 @@ to `update`: - type: _int_ - default: `3_000` -Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. +Specifies how many records to attempt to batch together for insertion or deletion +into the destination collection. ### insert.timeout.ms diff --git a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md index 7bed7b953d..8b21e9d99b 100644 --- a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md +++ b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md @@ -18,7 +18,7 @@ This connector is compatible with: - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions) - JDK 8 and higher versions -- ArangoDB 3.11 or higher versions (BTS-615) +- ArangoDB 3.11.1 and higher versions ## Installation diff --git a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md index 3f2062244a..5ae7e5a239 100644 --- a/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md +++ b/site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md @@ -178,7 +178,8 @@ to `update`: - type: _int_ - default: `3_000` -Specifies how many records to attempt to batch together for insertion or deletion into the destination collection. +Specifies how many records to attempt to batch together for insertion or deletion +into the destination collection. ### insert.timeout.ms