Skip to content

joker1007/embulk-input-kafka

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Kafka input plugin for Embulk

Apache Kafka input plugin for Embulk

Overview

  • Plugin type: input
  • Resume supported: no
  • Cleanup supported: no
  • Guess supported: no

Configuration

  • broker: kafka broker host and port (array(string), required)
  • topics: target topic name (array(string), required)
  • topic_column: use column value as target topic (string, default: null)
  • schema_registry_url: Schema Registy URL that is needed for avro format (string, default: null)
  • serialize_format: use column value as target topic (enum, required, json or avro_with_schema_registry)
  • key_column_name: use column value as record key. If columns option has same key, plugin set the record key to the column (string, default: _key)
  • partition_column_name: use column value as partition id. If columns option has same key, plugin set the partition id of the record to the column (string, default: _partition)
  • fetch_max_wait_ms: Max wait time for polling (integer, default: 30000)
  • max_empty_pollings: If the result of pollings is empty over this value times, plugin give up to poll the partition (integer, default: 2)
  • other_consumer_configs: other consumer configs (json, default: {})
  • value_subject_name_strategy: Set SchemaRegistry subject name strategy (string, default: null, ex. io.confluent.kafka.serializers.subject.RecordNameStrategy)
  • columns: embulk column configs (array(column config))

Example

type: kafka
topics:
  - "json-complex-topic"
serialize_format: json
brokers:
  - "localhost:9092"
fetch_max_wait_ms: 10000
columns:
  - {name: id, type: string}
  - {name: int_item, type: long}
  - {name: varchar_item, type: string}
  - {name: time, type: timestamp, format: "%Y-%m-%dT%H:%M:%SZ"}
  - {name: array, type: json}
  - {name: data, type: json}
  - {name: _key, type: string}
  - {name: _partition, type: long}

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

About

Apache Kafka input plugin for Embulk

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published