Welcome to Crato
Crato is an open source framework for small applications to easily deploy centralized logging. Crato is built with Node.js, Rsyslog, Apache Kafka, InfluxDB, and AWS S3. This repo contains the core Crato project that receives, transforms, and stores log data for further analysis.
Table of Contents
After installing the above dependencies, run
npm install &&
npm link which will make the
crato command available.
Here are a list of Crato commands:
||Provides a listing and description of all of Crato's services|
||Starts up Crato system|
||Stops Crato system and all services|
||Starts a specific Crato service|
||Stops a specific Crato service|
||Installs Kafka cluster and textlogs & jsonlogs topics|
||Displays Docker container logs for a specific service|
||See all external logs streaming into Crato. Press Ctrl-C to exit|
||Attaches a shell for a specific Crato service|
||Displays the status of all of Crato's services|
Crato uses some environment variables for its Node.js consumer app mainly to archive logs to Amazon Web Services (AWS) Simple Storage Service (S3).
AWS_S3_BUCKET_NAME to authenticate for AWS S3. We also require the
QUEUE_MAX_SIZE to determine the number of logs to queue, before being uploaded to AWS S3.
These environment variables should be written on your local machine and Docker will automatically read them if they are. See this great guide by Twilio for how to set environment variables on different operating systems.
To install Crato, clone this repo. Then run
docker-compose pull to download the necessary Docker images to build the service containers. The install can be finished via Crato CLI commands or Docker or Docker-Compose CLI commands.
Using Crato CLI
Using Docker-Compose CLI
docker-compose up -d zookeeper
docker-compose up -d kafka
docker-compose exec kafka kafka-topics --create --zookeeper zookeeper:8092 --replication-factor 3 --partitions 6 --topic textlogs
docker-compose exec kafka kafka-topics --create --zookeeper zookeeper:8092 --replication-factor 3 --partitions 6 --topic jsonlogs
docker-compose up -d
crato status or
docker ps -a to check the status of the service containers that have been created.
Crato exposes ports
10514 for receiving log data into the system. On your own host machine where Crato is deployed, make sure these are available.
Show your support