Skip to content

sevagh/pq

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

pq license Crates.io

protobuf to json deserializer, written in Rust

pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited/i32be streams, and it can connect to Kafka.

pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.

Download

pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.

Usage

new You can now pass in a proto file and have pq compile it on the fly using protoc:

$ pq --protofile ./tests/protos/dog.proto  --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "breed": "gsd",
  "age": 3,
  "temperament": "excited"
}

Use PROTOC and PROTOC_INCLUDE to control the executed protoc binary and configure the -I=/proto/path argument (design copied from prost).

To set up, put your *.fdset files in ~/.pq, /etc/pq, or a different directory specified with the FDSET_PATH env var:

$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/

You can specify additional fdset directories or files via options:

$ pq --msgtype com.example.dog.Dog --fdsetdir ./tests/fdsets <./tests/samples/dog
$ pq --msgtype com.example.dog.Dog --fdsetfile ./tests/fdsets/dog.fdset <./tests/samples/dog

Pipe a single compiled protobuf message:

$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "age": 4,
  "breed": "poodle",
  "temperament": "excited"
}

Pipe a varint or leb128 delimited stream:

$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Consume from a Kafka stream:

$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Convert a Kafka stream to varint-delimited:

$ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
> pq --msgtype com.example.dog.Dog --stream varint
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Pipe kafkacat output to it:

$ kafkacat -b 192.168.0.1:9092 -C -u -q -f "%R%s" -t my_topic |\
> pq --msgtype=com.example.dog.Dog --stream i32be
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Compile without kafka

To compile pq without kafka support, run:

$ cargo build --no-default-features

Compile for Windows

  1. Install Visual Studio Installer Community edition
  2. Run the installer and install Visual Studio Build Tools 2019. You need the C++ Build Tools workload. Note that you can't just install the minimal package, you also need MSVC C++ x64/86 Build Tools and Windows 10 SDK.
  3. Open x64 Native Tools Command Prompt from the start menu.
  4. Download and run rustup-init.exe
  5. Close and reopen your terminal (so cargo will be in your path)
  6. Run cargo install --no-default-features pq

Note that this will disable the Kafka feature, which is not currently supported on Windows.