HTTP log transport.
Go Makefile
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cmd/log-shuttle Add length prefixed rfc5424 support + cleanups Mar 2, 2018
misc
vendor
.gitignore
.godir
.travis.yml
Dockerfile Update Dockerfile comment Apr 26, 2016
Makefile
Procfile Use the binary installed by the platform if it exists Jun 21, 2013
batch.go
batcher.go
changelog.md
cleanurl.go Address feedback via @freeformz Jan 25, 2018
config.go Add length prefixed rfc5424 support + cleanups Mar 2, 2018
counter.go Move the "library" into the main dir. Dec 3, 2014
counter_test.go Move the "library" into the main dir. Dec 3, 2014
formatter.go
gzip_formatter.go Go {lint,vet} fixes Mar 24, 2015
gzip_formatter_test.go cleanup debugging + fix lint Apr 26, 2016
http_outlet.go
http_outlet_test.go
kinesis_formatter.go KinesisFormatter.shards isn't necessary Apr 26, 2016
kinesis_formatter_test.go Protect from data races in test code Aug 13, 2015
kinesis_types.go
kinesis_types_test.go
log_line_reader_test.go
logline.go Add length prefixed rfc5424 support + cleanups Mar 2, 2018
logplex_formatter.go
logplex_formatter_test.go Add length prefixed rfc5424 support + cleanups Mar 2, 2018
reader.go pass time taken to batch to delivery function Jul 6, 2016
readme.md Add length prefixed rfc5424 support + cleanups Mar 2, 2018
shuttle.go cleanup debugging + fix lint Apr 26, 2016
shuttle_test.go

readme.md

Log Shuttle

Travis Releases GoDoc

Log-shuttle is an open source UNIX program that delivers messages from applications and daemons to log routers and processors via HTTPs.

One of the motivations behind log-shuttle is to provide a simpler form of encrypted & authenticated log delivery. Using HTTPs & Basic Authentication is simpler than the techniques described in RFC5425. TLS transport mapping for Syslog requires that you maintain both client & server certificates for authentication. In multi-tenant environments, the maintenance of certificate management can be quite burdensome.

When using log-shuttle with logplex it is recommended that you spawn 1 log-shuttle per logplex token. This will isolate data between customers and ensure a good QoS. Log-shuttle accepts input from stdin in a newline (\n) delimited format.

When using log-shuttle with Amazon's Kinesis, all the details for the region, stream and access credentials are supplied in the -logs-url (or $LOGS_URL env variable). See the Kinesis section of this document.

To block as little as possible, log-shuttle will drop outstanding batches if it accumulates > -back-buff amount.

Kinesis

log-shuttle sends data into Kinesis using the PutRecords API call. Each Kinesis record is encoded as length prefixed rfc5424 encoded logs as per rfc6587 (this is the same format logplex accepts). One record per log line.

Log-shuttle expects the following encoding of -logs-url when using Amazon Kinesis:

```
https://<AWS_KEY>:<AWS_SECRET>@kinesis.<AMAZON_REGION>.amazonaws.com/<STREAM NAME>
```

See the Amazon Endpoints documentation for supported regions and hostnames.

Kinesis Caveats

Things that should be handled better/things you should know:

  1. AWS_SECRET, AWS_KEY, AMAZON_REGION & STREAM NAME need to be properly url encoded.
  2. log-shuttle assumes a 200 response means everything is good. Kinesis can return a 200, meaning the http request was good, but include per record errors in the response body.
  3. The maximum number of records in a PutRecords requests is 500, so set the batch size no higher than 498 (500 - 2 records for possible drops / lost).
  4. Logplex max line length is 10k, Kinesis max record size is 50k of base64 encoded data. A -max-line-length of somewhere less than 37500 should work for Kinesis w/o causing errors.
  5. Kinesis does not support the -gzip option as that option compresses the body of the request.
  6. Even with -kinesis-shards, no guarantees can be made about writing to unique shards.

Install

go get -u github.com/heroku/log-shuttle/...

After that $GOPATH/bin/log-shuttle should be available.

Making Debs

Requires:

  • dpkg (see also brew install dpkg)
  • go 1.6+
make debs

Docker

There is a Makefile target named docker that can be used to build a docker image.

Hacking on log-shuttle

Fork the repo, hack, submit PRs.

Testing

go test -v ./...

Submitting Code

  • Open an issue on GitHub.
  • Keep changes in a feature branch
  • Submit PR

License

Copyright (c) 2013-15 Heroku Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.