Skip to content

Releases

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 17 Nov 01:30

First release of the GreptimeDB sink connector for Kafka Connect. It writes records from Kafka topics to GreptimeDB tables over gRPC.

Features

  • Field mapping. Choose the time index (timestamp.field, or the Kafka record timestamp), tag columns (tag.fields), and field columns (field.include, field.exclude). table.name.format routes each topic to a table, with ${topic} as a placeholder.
  • Record formats. Struct values from Avro, Protobuf, or JSON Schema, with column types taken from the schema, and schemaless JSON.
  • Automatic schema. GreptimeDB creates tables and adds columns on write. Tags become the primary key.
  • Batched writes. Each batch is written before offsets are committed, so delivery is at-least-once. Replayed rows overwrite rows with the same tags and time index.
  • Retries. Transient errors, such as an unreachable server or rate limiting, are retried up to max.retries times, retry.backoff.ms apart. The task fails after that; batches are never dropped.
  • Bad records. Records that can't be mapped, and rows that GreptimeDB rejects, follow errors.tolerance and go to the dead letter queue when one is configured. When GreptimeDB rejects a request, the connector finds the rejected rows and still writes the rest.
  • Security. Username and password authentication, and TLS with an optional CA certificate.

Install

Download greptimedb-kafka-connect-0.1.0.zip and its checksum, verify it, and extract it into a directory in the worker's plugin.path:

curl -LO https://github.com/GreptimeTeam/greptimedb-kafka-connect/releases/download/v0.1.0/greptimedb-kafka-connect-0.1.0.zip
curl -LO https://github.com/GreptimeTeam/greptimedb-kafka-connect/releases/download/v0.1.0/greptimedb-kafka-connect-0.1.0.zip.sha512
sha512sum -c greptimedb-kafka-connect-0.1.0.zip.sha512
unzip greptimedb-kafka-connect-0.1.0.zip -d /opt/connectors

Then restart the worker. See the README for a quick start and the full configuration reference.

Compatibility

  • Built for Java 8 and later.
  • Tested with Kafka Connect 3.9 and GreptimeDB v1.3.
  • Uses GreptimeDB Java Ingester 0.16.0.

Known limitations

  • Nested values (STRUCT, MAP, ARRAY) aren't supported. Exclude them, or flatten structs and maps with the Flatten transform.
  • String timestamps aren't supported. Convert them with the TimestampConverter transform.
  • Records are only inserted. Tombstones and deletes aren't applied to GreptimeDB.
  • In schemaless JSON, 20 is an integer and 20.5 is a float. Write floating-point values with a decimal point, or GreptimeDB rejects them once the column type is set.
  • Tables created with append_mode = 'true' keep duplicates written on retry.
  • The archive is about 46 MB, mostly dependencies of the ingester (gRPC, Netty, and Arrow).