Merge pull request #87 from mailgun/maxim/develop

Add support for gRPC

.travis.yml

@@ -1,6 +1,6 @@
 language: go
 go:
-  - 1.6.3
+  - 1.7.5
 
 env:
   global:
@@ -18,16 +18,6 @@ before_install:
   - vendor/github.com/Shopify/sarama/vagrant/install_cluster.sh
   - vendor/github.com/Shopify/sarama/vagrant/boot_cluster.sh
   - vendor/github.com/Shopify/sarama/vagrant/create_topics.sh
-  # Install coveralls.io dependencies
-  - go get github.com/axw/gocov/gocov
-  - go get github.com/mattn/goveralls
-  - go get golang.org/x/tools/cmd/cover
-  - go get github.com/pierrre/gotestcover
-
-script:
- - $HOME/gopath/bin/gotestcover -v -covermode=atomic -parallelpackages=1 -coverprofile=cover.out -timeout=5m ./... -check.vv
- - $HOME/gopath/bin/goveralls -coverprofile=cover.out -service=travis-ci
- - make vet
- - make fmt
 
+script: ./scripts/travis_build.sh
 sudo: false

CHANGELOG.md

@@ -2,21 +2,22 @@
 
 #### Version 0.12.0 (TBD)
 
+Implemented:
+
 * [#81](https://github.com/mailgun/kafka-pixy/pull/81) Add capability to
   proxy to multiple Kafka/ZooKeeper clusters.
-* [#83](https://github.com/mailgun/kafka-pixy/pull/83) Panic in partition
-  multiplexer.
-* [#85](https://github.com/mailgun/kafka-pixy/pull/85) Panic in partition
-  multiplexer.
-* [#16](https://github.com/mailgun/kafka-pixy/issues/16) A YAML configuration
-  file can be passed to Kafka-Pixy with `--config` command line parameter. A
-  [default](https://github.com/mailgun/kafka-pixy/blob/master/default.yaml)
-  configuration file is provided for reference.
+* [#16](https://github.com/mailgun/kafka-pixy/issues/16) A YAML
+  configuration file can be passed to Kafka-Pixy with `--config` command
+  line parameter. Default configuration file is provided for reference.
+* [#87](https://github.com/mailgun/kafka-pixy/pull/87) Add support for
+  gRPC API.
 
 Fixed:
 
-* [#83](https://github.com/mailgun/kafka-pixy/issues/83) Panic in partition
-  multiplexer.
+* [#83](https://github.com/mailgun/kafka-pixy/issues/83) Panic in
+  partition multiplexer.
+* [#85](https://github.com/mailgun/kafka-pixy/pull/85) Another panic in
+  partition multiplexer.
 
 #### Version 0.11.1 (2016-08-11)
 

README.md

@@ -1,35 +1,59 @@
-# Kafka-Pixy (HTTP Proxy)
+# Kafka-Pixy (gRPC/REST Proxy for Kafka)
 
 [![Build Status](https://travis-ci.org/mailgun/kafka-pixy.svg?branch=master)](https://travis-ci.org/mailgun/kafka-pixy) [![Go Report Card](https://goreportcard.com/badge/github.com/mailgun/kafka-pixy)](https://goreportcard.com/report/github.com/mailgun/kafka-pixy) [![Coverage Status](https://coveralls.io/repos/mailgun/kafka-pixy/badge.svg?branch=master&service=github)](https://coveralls.io/github/mailgun/kafka-pixy?branch=master)
 
-Kafka-Pixy is a local aggregating HTTP proxy to [Kafka](http://kafka.apache.org/documentation.html)
-with automatic consumer group control. It is designed to hide the complexity of
-the Kafka client protocol and provide a stupid simple HTTP API that is trivial
-to implement in any language.
-
-Kafka-Pixy works with Kafka **0.8.2.x** and **0.9.0.x**. It uses the Kafka 
-[Offset Commit/Fetch API](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI)
-to keep track of consumer offsets and ZooKeeper to manage distribution of
-partitions among consumer group members.
-
-You can jump to [Quick Start](README.md#quick-start) if you are anxious to give it a try.
-
-## Aggregation
-Kafka works best when messages are read/written in batches, but from application
-standpoint it is easier to deal with individual message read/writes. Kafka-Pixy
-provides message based API to clients, but internally it collects them in
-batches and submits them the way Kafka likes it the best. This behavior plays
-very well with the microservices architecture, where there are usually many tiny
-assorted service instances running on one beefy physical host. So Kafka-Pixy
-installed on that host would aggregate messages from all the service instances,
-herby effectively using the network bandwidth.
-
-## Locality
-Kafka-Pixy is intended to run on the same host as the applications using it.
-Remember that it provides only message based API - no batching, therefore using
-it over network is suboptimal. To make local usage even more efficient
-Kafka-Pixy provides an option to serve its API via a Unix Domain Socket
-alongside a TCP socket (**0.0.0.0:19092** by default).
+Kafka-Pixy is a dual API (gRPC and REST) proxy for [Kafka](http://kafka.apache.org/documentation.html)
+with automatic consumer group control. It is designed to hide the
+complexity of the Kafka client protocol and provide a stupid simple
+API that is trivial to implement in any language.
+
+Kafka-Pixy supports Kafka **0.8.2.x** and **0.9.0.x**. It uses the
+Kafka [Offset Commit/Fetch API](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI)
+to keep track of consumer offsets and ZooKeeper to manage distribution
+of partitions among consumer group members.
+
+You can jump to [Quick Start](README.md#quick-start) if you are anxious
+to give it a try.
+
+#### Key Features:
+
+- **Dual API**: Kafka-Pixy provides two types of API:
+  - [gRPC](http://www.grpc.io/docs/guides/)
+  ([Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview)
+  over [HTTP/2](https://http2.github.io/faq/)) recommended to
+  produce/consume messages;
+  - REST (JSON over HTTP) intended for for testing and operations
+  purposes, although you can use it to produce/consume messages too;
+- **Muli-Cluster Support**: One Kafka-Pixy instance can proxy to
+  several Kafka clusters. You just need to define them in the [config
+  file](https://github.com/mailgun/kafka-pixy/blob/master/default.yaml)
+  and then address clusters by name given in the config file in your
+  API requests.
+- **Aggregation**: Kafka works best when messages are read/written in
+  batches, but from application standpoint it is easier to deal with
+  individual message read/writes. Kafka-Pixy provides message based API
+  to clients, but internally it collects them in batches and submits
+  them the way Kafka likes it the best. This behavior plays very well
+  with the microservices architecture, where there are usually many
+  tiny assorted service instances running on one beefy physical host.
+  So Kafka-Pixy installed on that host would aggregate messages from
+  all the service instances, herby effectively using the network
+  bandwidth.
+- **Locality**: Kafka-Pixy is intended to run on the same host as the
+  applications using it. Remember that it provides only message based
+  API - no batching, therefore using it over network is suboptimal.
+
+## gRPC API
+
+[gRPC](http://www.grpc.io/docs/guides/) is an opens source framework
+that is using [Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview)
+as interface definition language and [HTTP/2](https://http2.github.io/faq/)
+as transport protocol. Kafka-Pixy defines producer/consumer API in
+[grpc.proto](https://github.com/mailgun/kafka-pixy/blob/master/grpc.proto)
+file. Client stubs for Golang and Python are pre-generated, but you can
+easily generated for dozens of other languages. Please refer to the
+gRPC [documentation](http://www.grpc.io/docs/) for information on the
+language of your choice.
 
 ## HTTP API
 
@@ -284,8 +308,9 @@ Command line parameters that Kafka-Pixy accepts are listed below:
  config         | Path to a YAML configuration file.
  kafkaPeers     | Comma separated list of Kafka brokers. Note that these are just seed brokers. The rest brokers are discovered automatically. (Default **localhost:9092**)
  zookeeperPeers | Comma separated list of ZooKeeper nodes followed by optional chroot. (Default **localhost:2181**)
- tcpAddr        | TCP interface where the HTTP API should listen. (Default **0.0.0.0:19092**)
- unixAddr       | Unix Domain Socket that the HTTP API should listen on. If not specified then the service will not listen on a Unix Domain Socket. 
+ grpcAddr       | TCP address that the HTTP API should listen. (Default **0.0.0.0:19092**)
+ tcpAddr        | TCP address that the HTTP API should listen. (Default **0.0.0.0:19092**)
+ unixAddr       | Unix Domain Socket that the HTTP API should listen on. If not specified then the service will not listen on a Unix Domain Socket.
  pidFile        | Name of a pid file to create. If not specified then a pid file is not created.
 
 You can run `kafka-pixy -help` to make it list all available command line

config/config.go

@@ -16,22 +16,24 @@ import (
 // App defines Kafka-Pixy application configuration. It mirrors the structure
 // of the JSON configuration file.
 type App struct {
-	// TCP address that Kafka-Pixy should listen at.
+	// TCP address that gRPC API server should listen on.
+	GRPCAddr string `yaml:"grpc_addr"`
+
+	// TCP address that HTTP API server should listen on.
 	TCPAddr string `yaml:"tcp_addr"`
 
-	// Unix domain socket address that Kafka-Pixy should listen at. Listening
-	// on a unix domain socket is disabled by default.
+	// Unix domain socket address that HTTP API server should listen on.
+	// Listening on a unix domain socket is disabled by default.
 	UnixAddr string `yaml:"unix_addr"`
 
 	// An arbitrary number of proxies to different Kafka/ZooKeeper clusters can
 	// be configured.
-	Proxies map[string]*Proxy
+	Proxies map[string]*Proxy `yaml:"proxies"`
 
-	// Default proxy, the one to be used in API calls that do not start with
-	// prefix `/proxy/<alias>`. If config is initialized from a YAML file then
-	// default proxy is the one that is mentioned in the `Proxies` section
-	// first.
-	DefaultProxy string
+	// Default proxy is the one to be used in API calls that do not start with
+	// prefix `/proxy/<alias>`. If it is not explicitly provided, then the one
+	// mentioned in the `Proxies` section first is assumed.
+	DefaultProxy string `yaml:"default_proxy"`
 }
 
 // Proxy defines configuration of a proxy to a particular Kafka/ZooKeeper
@@ -211,6 +213,7 @@ func (p *Proxy) validate() error {
 
 func newApp() *App {
 	appCfg := &App{}
+	appCfg.GRPCAddr = "0.0.0.0:19091"
 	appCfg.TCPAddr = "0.0.0.0:19092"
 	appCfg.Proxies = make(map[string]*Proxy)
 	return appCfg

default.yaml

@@ -1,8 +1,11 @@
-# TCP address that Kafka-Pixy should listen at.
+# TCP address that gRPC API server should listen on.
+grpc_addr: 0.0.0.0:19091
+
+# TCP address that RESTful API server should listen on.
 tcp_addr: 0.0.0.0:19092
 
-# Unix domain socket address that Kafka-Pixy should listen at. Listening on
-# a unix domain socket is disabled by default.
+# Unix domain socket address that RESTful API server should listen on.
+# Listening on a unix domain socket is disabled by default.
 # unix_addr: "/var/run/kafka-pixy.sock"
 
 # An arbitrary number of proxies to different Kafka/ZooKeeper clusters can be