-
Notifications
You must be signed in to change notification settings - Fork 0
License
Doist/s3logger
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Program s3logger collects json messages over TCP, stores them into gzip-compressed files split by time and uploads these files to AWS s3 bucket. Once started, service accepts TCP connections and expects clients to send streams of json objects over such connections. s3logger only closes connection if it encounters malformed json or single object size exceeds 4 MiB limit. s3logger only reads data from the client. s3logger accumulates received messages over predefined time window (-t flag) to a temporary log file creating new ones as needed; previous files are uploaded to s3 bucket in background and removed after successful upload. Optionally maximum size of input read can be specified in megabytes (-mb flag) to rotate file before reaching predefined time. Program only writes to a single temporary log file at a time, so json messages received from multiple concurrent connections are interleaved into a single json stream. It does its best not to lose messages, but can still drop them if they're coming faster than could be saved on disk or there's any disk write error. Stored messages are separated by new line (0xa). s3logger uploads files to a specified bucket using predefined s3 object naming scheme: dt=2018-02-09/20180209T213803_df718a7818e53243.json.gz It uses dt=YYYY-MM-DD "directories", object name base starting with date and time when log file was created (UTC) followed by hex-encoded 64-bit random value and .json.gz suffix. See https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables on how to configure s3 bucket access credentials. s3logger does not use TLS for its listener at the moment as it is expected to run on localhost or inside trusted network. Usage of s3logger: -addr string address to listen (default "localhost:8080") -bucket string s3 bucket to upload logs -dir string directory to keep unsent files (default "/var/spool/s3logger") -mb int megabytes of input read until file is rotated (0 to disable) (default 512) -prefix string s3 object name prefix (directory in a bucket) -t duration time to use single file (min 1m) (default 5m0s)