AWS Serverless logging
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

AWS Structured Serverless logging

Cheap structured serverless logging system that scales automatically.

Create AWS infrastructure

The following script will ask for the required input and then proceeds to create the necessary resources.


Send test log messages

To send 3000 log message to the log sytem use the

python3 <stream-name> 3000

The test messages will be delivered to S3, buffered and compressed.


The architecture is quite simple.

  1. The provided logger logs messages into the Kinesis Firehose delivery stream
  2. The stream buffers and compresses the log messages and delivers it to S3
  3. You can query the log messages with EMR (Spark, MR, etc..), Athena


We provide an out-of-the-box logger with Import the Logger class and use it to log structured messages.

# Initiate the logger with the stream/logging name
logger = KinesisLogger("datapao-logging-3")

# Simple message log{ "message": "Failed to run batch job" })

# Batch log messages
    { "level": "INFO", "message": "Failed to run batch job" },
    { "level": "DEBUG", "origin": "testing", "sensor": "temp-0002xb" }

The logger automatically adds an ISO8601 UTC timestamp to every message. It also offers convenient logging functions that adds a Level attribute.


The development tasks are tracked in Projects


Feel free to send any feedback, PR or unneccesary GIFs.

Created with ❤️ at