Repository navigation
Releases
Releases · GreptimeTeam/greptimedb-kafka-connect
v0.1.0
First release of the GreptimeDB sink connector for Kafka Connect. It writes records from Kafka topics to GreptimeDB tables over gRPC.
Features
- Field mapping. Choose the time index (
timestamp.field, or the Kafka record timestamp), tag columns (tag.fields), and field columns (field.include,field.exclude).table.name.formatroutes each topic to a table, with${topic}as a placeholder. - Record formats.
Structvalues from Avro, Protobuf, or JSON Schema, with column types taken from the schema, and schemaless JSON. - Automatic schema. GreptimeDB creates tables and adds columns on write. Tags become the primary key.
- Batched writes. Each batch is written before offsets are committed, so delivery is at-least-once. Replayed rows overwrite rows with the same tags and time index.
- Retries. Transient errors, such as an unreachable server or rate limiting, are retried up to
max.retriestimes,retry.backoff.msapart. The task fails after that; batches are never dropped. - Bad records. Records that can't be mapped, and rows that GreptimeDB rejects, follow
errors.toleranceand go to the dead letter queue when one is configured. When GreptimeDB rejects a request, the connector finds the rejected rows and still writes the rest. - Security. Username and password authentication, and TLS with an optional CA certificate.
Install
Download greptimedb-kafka-connect-0.1.0.zip and its checksum, verify it, and extract it into a directory in the worker's plugin.path:
curl -LO https://github.com/GreptimeTeam/greptimedb-kafka-connect/releases/download/v0.1.0/greptimedb-kafka-connect-0.1.0.zip
curl -LO https://github.com/GreptimeTeam/greptimedb-kafka-connect/releases/download/v0.1.0/greptimedb-kafka-connect-0.1.0.zip.sha512
sha512sum -c greptimedb-kafka-connect-0.1.0.zip.sha512
unzip greptimedb-kafka-connect-0.1.0.zip -d /opt/connectorsThen restart the worker. See the README for a quick start and the full configuration reference.
Compatibility
- Built for Java 8 and later.
- Tested with Kafka Connect 3.9 and GreptimeDB v1.3.
- Uses GreptimeDB Java Ingester 0.16.0.
Known limitations
- Nested values (
STRUCT,MAP,ARRAY) aren't supported. Exclude them, or flatten structs and maps with theFlattentransform. - String timestamps aren't supported. Convert them with the
TimestampConvertertransform. - Records are only inserted. Tombstones and deletes aren't applied to GreptimeDB.
- In schemaless JSON,
20is an integer and20.5is a float. Write floating-point values with a decimal point, or GreptimeDB rejects them once the column type is set. - Tables created with
append_mode = 'true'keep duplicates written on retry. - The archive is about 46 MB, mostly dependencies of the ingester (gRPC, Netty, and Arrow).