Merge 5a1023a into 9648065

.travis.yml

@@ -1,6 +1,6 @@
 language: go
 go:
-- 1.x
+- '1.14'
 services: docker
 
 env:
@@ -14,10 +14,7 @@ before_install:
 - docker-compose -f kafka.yml up -d
 - PATH="$PATH:$GOPATH/bin"
 # Download the dependencies
-- export GO111MODULE=on
-- go mod download
-- go mod vendor
-- go mod verify
+- go mod download && go mod verify
 - go get github.com/mattn/goveralls
 - go get golang.org/x/tools/cmd/cover
 

README.md

@@ -17,23 +17,18 @@ Frizzle is a magic message (`Msg`) bus designed for parallel processing w many g
 
 ## Prereqs / Build instructions
 
-### Go mod
-
-As of Go 1.11, frafka uses [go mod](https://github.com/golang/go/wiki/Modules) for dependency management.
-
 ### Install librdkafka
 
-Frafka depends on C library `librdkafka` (>=`v0.11.6`). For Debian 9+ (which includes golang docker images),
-it has to be built from source. Fortunately, there's a script for that.
+The underlying kafka library,
+[confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go#installing-librdkafka)
+has some particularly important nuances:
 
-```sh
-  # Install librdkafka
-  - curl --silent -OL https://raw.githubusercontent.com/confluentinc/confluent-kafka-go/v0.11.4/mk/bootstrap-librdkafka.sh
-  - bash bootstrap-librdkafka.sh v0.11.4 /usr/local
-  - ldconfig
-```
+* alpine builds (e.g. `FROM golang-1.14-alpine` should run all go commands with `-tags musl`
+  * e.g. `go test -tags musl ./...`
+* all builds producing an executable should run with `CGO_ENABLED=1`
+  * not necessary for libraries, however.
 
-Once that is installed, should be good to go with
+Otherwise, should be good to go with
 
 ```sh
 go get github.com/qntfy/frafka
@@ -86,13 +81,20 @@ use a prefix before the below values.
 | KAFKA_TOPICS | source | topic(s) to read from |  |
 | KAFKA_CONSUMER_GROUP | source | consumer group value for coordinating multiple clients |  |
 | KAFKA_CONSUME_LATEST_FIRST | source (optional) | start at the beginning or end of topic | earliest |
-| KAFKA_MAX_BUFFER_KB | optional | How large a buffer to allow for prefetching and batch produing kafka message* | 16384 |
+| KAFKA_COMPRESSION | sink (optional) | set a compression format, equivalent to `compression.type` |  |
+| KAFKA_MAX_BUFFER_KB | optional | How large a buffer to allow for prefetching and batch produing kafka message | 16384 |
+| KAFKA_CONFIG | optional | Add librdkafka client config, format `key1=value1 key2=value2 ...` |  |
+
+### Configuration Notes
 
-*`KAFKA_MAX_BUFFER_KB` is passed through to librdkafka. Default is 16MB.
+* `KAFKA_MAX_BUFFER_KB` is passed through to librdkafka. Default is 16MB.
 Corresponding librdkafka config values are `queue.buffering.max.kbytes` (Producer) and `queued.max.messages.kbytes`
 (Consumer). Note that librdkafka creates one buffer each for the Producer (Sink) and for each topic+partition
 being consumed by the source. E.g. with default 16MB default, if you are consuming from 4 partitions and also
 producing then the theoretical max memory usage from the buffer would be `16*(4+1) = 80` MB.
+* `KAFKA_CONFIG` allows setting arbitrary
+[librdkafka configuration](https://github.com/edenhill/librdkafka/blob/v1.4.2/CONFIGURATION.md)
+such as `retries=10 max.in.flight=1000 delivery.report.only.error=true`
 
 ## Async Error Handling
 

go.mod

@@ -2,11 +2,26 @@ module github.com/qntfy/frafka
 
 require (
 	github.com/confluentinc/confluent-kafka-go v1.4.2
-	github.com/gofrs/uuid v3.2.0+incompatible
-	github.com/pkg/errors v0.8.1
-	github.com/qntfy/frizzle v0.5.0
-	github.com/spf13/viper v1.2.1
-	github.com/stretchr/testify v1.2.2
+	github.com/fsnotify/fsnotify v1.4.9 // indirect
+	github.com/gofrs/uuid v3.3.0+incompatible
+	github.com/magiconair/properties v1.8.3 // indirect
+	github.com/mitchellh/mapstructure v1.3.3 // indirect
+	github.com/pelletier/go-toml v1.8.1 // indirect
+	github.com/pkg/errors v0.9.1
+	github.com/qntfy/frizzle v0.5.2
+	github.com/spf13/afero v1.4.0 // indirect
+	github.com/spf13/cast v1.3.1 // indirect
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/spf13/viper v1.7.1
+	github.com/stretchr/objx v0.3.0 // indirect
+	github.com/stretchr/testify v1.6.1
+	go.uber.org/multierr v1.6.0 // indirect
+	go.uber.org/zap v1.16.0 // indirect
+	golang.org/x/sys v0.0.0-20200917073148-efd3b9a0ff20 // indirect
+	gopkg.in/ini.v1 v1.61.0 // indirect
+	gopkg.in/yaml.v2 v2.3.0 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 // indirect
 )
 
-go 1.13
+go 1.14