s3logtools: S3 Logs utility module

This script allows handling of S3 logs as incoming streams to be either piped and colored on the fly from the console, or to consolidate them into a file

It resorts to long polling to update the log as AWS doesn't ship in real time.

It's designed to work with logs produced by s3-streamlogger and winston.

Prerequisites

You need to have an AWS S3 bucket and it's recommended to set a new AWS IAM user with these permissions: then store his access keys in your ~/.aws/credentials.

{
    "Version": "2012-10-17",
    "Statement": [
        
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": ["arn:aws:s3:::muzetestlogs",
                    "arn:aws:s3:::your-bucket-name/*"]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::your-bucket-name",
                    "arn:aws:s3:::your-bucket-name/*"]
        }
    ]
}

Installation and Usage

npm install -g s3logtools

Ensure that the AWS credentials to access your bucket are set in your environment (for example by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables) or in the ~/.aws/credentials.

s3logtools --bucket=your-log-bucket-name --watch

More complex operations are possible:

s3logtools --bucket=your-log-bucket-name --ago='30 days' --output='myfile.csv' --delimiter='|'

Full list:

Options	Use
-v	prints the version number
-o, --output <path/filename>	compacts the response into a correctly formatted file (.json/.csv)
-w, --watch	'Live tails using polling'
-s, --start	starts the logging output from this time (momentjs compatible)')
-e, --end	ends the logging output after this time (momentjs compatible)')
-a, --ago	starts the logging output time ago (momentjs compatible)')
--prefix	fetch only logs with the specified prefix')
-b, --beautify	beautify the output')
-r, --raw	outputs the logs as is from s3')
-p, --profile	AWS profile name to use from ~/.aws/credentials')
-f, --filter	filters only the logs matching the specified pattern')
-l, --loglevel	what level of logs to report: silent, error, minimal, warn, info, verbose, silly (default: warn)

License

ISC: equivalent to 2-clause BSD.