Skip to content

JakkuSakura/kafcat

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

59 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

kafcat

Kafcat is a Rust fully async rewrite of kafkacat.

I was trying to copy some data from kafka on a remote server to localhost for testing purpose. Kafkacat was the only usable tool to do so. However, it is written in C and does not support binary data at all. I was working on a pull request, but it haven't been merged yet, and the code was hard to maintain and error prone. So I wrote my own kafcat.

Features

It mimicks kafkacat's cli usage, but with extra features. It can

  • read from stdin and send to kafka topic
  • read from kafka to stdin
  • copy from one topic to another(even on different servers)
  • import/export data in json format
  • swappable backend(only librdkafka for now, kafka-rust coming soon)

Drawbacks

Kafcat is still on early development, so

  • does not support TLS yet

  • does not support Apache AVRO

  • does not support complex input/output format

Basic Usage

  1. Compile kafcat. You can find the binary at target/release/kafcat

    cargo build --release
  2. Make sure Kafka is running (We assume at localhost:9092). You can usedocker-compose -f tests/plaintext-server.yml up(note that kafka in docker on MacOS is problematic, I couldn't even setup the proper testing environment via docker)

  3. Setup the listener, assuming kafcat is in current directory

    ./kafcat -C --topic test
  4. Setup the producer in another terminal,

    ./kafcat -P --topic test
  5. Copy a topic. the format is ./kafcat copy <from> -- <to>, from and to are exactly as used consumer and producer

    ./kafcat copy --topic test -- --topic test2
    
  6. Type any key and value with the default delimiter :. For example, hello:world.

  7. You should see hello:world in the consumer terminal.

  8. Detailed help can be found at ./kafcat --help, ./kafcat -C --help, ./kafcat -P --help, etc. You can also check out kafkacat for reference.

    kafcat-consume 
    
    USAGE:
        kafcat {consume, -C} [FLAGS] [OPTIONS] --topic <topic>
    
    FLAGS:
        -e, --exit
                Exit successfully when last message received
    
        -h, --help
                Prints help information
    
        -V, --version
                Prints version information
    
    
    OPTIONS:
        -b, --brokers <brokers>
                Broker list in kafka format [default: localhost:9092]
    
        -s, --format <format>
                Serialize/Deserialize format [default: text]
    
        -G, --group-id <group-id>
                Consumer group id. (Kafka >=0.9 balanced consumer groups) [default: kafcat]
    
        -K <key-delimiter>
                Delimiter to split input key and message [default: :]
    
        -D <msg-delimiter>
                Delimiter to split input into messages(currently only supports '\n') [default: 
                ]
    
        -o <offset>
                Offset to start consuming from:
                                     beginning | end | stored |
                                     <value>  (absolute offset) |
                                     -<value> (relative offset from end)
                                     s@<value> (timestamp in ms to start at)
                                     e@<value> (timestamp in ms to stop at (not included))[default:
                beginning]
    
        -p, --partition <partition>
                Partition
    
        -t, --topic <topic>
                Topic
    kafcat-produce 
    
    USAGE:
        kafcat {produce, -P} [OPTIONS] --topic <topic>
    
    FLAGS:
        -h, --help       Prints help information
        -V, --version    Prints version information
    
    OPTIONS:
        -b, --brokers <brokers>        Broker list in kafka format [default: localhost:9092]
        -s, --format <format>          Serialize/Deserialize format [default: text]
        -G, --group-id <group-id>      Consumer group id. (Kafka >=0.9 balanced consumer groups)
                                       [default: kafcat]
        -K <key-delimiter>             Delimiter to split input key and message [default: :]
        -D <msg-delimiter>             Delimiter to split input into messages(currently only supports
                                       '\n') [default: 
                                       ]
        -p, --partition <partition>    Partition
        -t, --topic <topic>            Topic
    kafcat-copy 
    Copy mode accepts two parts of arguments <from> and <to>, the two parts are separated by [--].
    <from> is the exact as Consumer mode, and <to> is the exact as Producer mode.
    
    USAGE:
        kafcat copy <from>... [--] <to>...
    
    ARGS:
        <from>...    
        <to>...      
    
    FLAGS:
        -h, --help       Prints help information
        -V, --version    Prints version information
    
    
    

Programming Style

  • git rebase onto master branch when possible
  • Squash commits before push