Fetches messages from SQS and forwards them to Kafka.
Given a properly setup go develop environment, just run
go get -d -t .
go build
The resulting binary should be named sqs-to-kafka
(or sqs-to-kafka.exe
on Windows). For further detail, e.g. cross-platform builds and custom build
parameters, please refer to the go-documentation.
sqs-to-kafka
takes the following commandline parameters:
--aws-access-key=STRING
: AWS access key--aws-secret-key=STRING
: AWS secret key--aws-region=STRING
: AWS region--aws-profile=STRING
: AWS profile--aws-read-config
: read AWS configuration from~/.aws/config
--aws-endpoint=STRING
: URL of the AWS endpoint--sqs-url=STRING
: URL of the SQS queue for incomming messages--kafka-brokers=STRING
: list of Kafka brokers used for bootstrapping--kafka-topic=STRING
: Kafka topic for outgoing messages--metrics-address=HOST:PORT
: Listening address to serve metrics
sqs-url
, kafka-brokers
and kafka-topic
are mandatory, everything else is
optional. When metrics-address
is given sqs-to-kafka
binds that address to
export prometheus compatible metrics. Parameters may set via environment
variables as well
AWS_ACCESS_KEY_ID
: AWS access keyAWS_SECRET_ACCESS_KEY
: AWS secret keyAWS_REGION
: AWS regionAWS_ENDPOINT
: URL of the AWS endpointAWS_PROFILE
: AWS profileAWS_SDK_LOAD_CONFIG
: if set to1
read AWS configuration from~/.aws/config
SQS_URL
: URL of the SQS queue for incomming messagesKAFKA_BROKERS
: list of Kafka brokers used for bootstrappingKAFKA_TOPIC
: Kafka topic for outgoing messagesMETRICS_ADDRESS
: Listening address to serve metrics
When both are specified, commandline parameters take precedence over environment variables.
In addition to that there a couple of AWS SDK specific parameters may be passed via environment variables, see the AWS SDK documentation for a complete list.
A docker-ized variant is available here meteogroup/sqs-to-kafka
. Metrics are
exposed on port 8080
. To run use
docker run -P meteogroup/sqs-to-kafka <additional commandline arguments>
Copyright © 2017 MeteoGroup Deutschland GmbH, all the files in this repository are released under the terms of Apache License 2.0.