- Logger
- Config
- Client
- Server
- Retry manager
- Store queue
- Buffering before storing
- Exponential backoff (I am not sure if we need it here.)
- HTTP API for querying logs from server
- Filter and Search on server side
- Filter wrapper for filtering logs
- HTTP/GRPC endpoint for getting logs from other service
I am not going to create a query algorithm or query api because this is unneccessary, complex and time consuming. So my focus is to make it more like Fluentd. There will be plugin in future for data visualizing tools/platforms. eg: Elasticsearch,datadog,logstash etc. (If you have a solution, create an issue and I am open to discussion.)
Make its architecture like flutend.
go get -u github.com/itzmanish/go-log-aggregator
Make sure $GOPATH/bin directory is in your path
$ go-log-aggregator -h
log-aggregator is a log aggregating tool which provides a server and agent.
Server command is used to start a server. Whereas agent command is used to start
an agent so that logs can be sent from host machine to server and stored in File system or S3.
Usage:
log-aggregator [flags]
log-aggregator [command]
Available Commands:
agent Log aggregator agent for collecting logs and sending to server.
help Help about any command
server Log aggregator server to collect logs from agent and process it.
Flags:
--config string config file (default is $HOME/.config/.log-aggregator.yaml)
-h, --help help for log-aggregator
-v, --version version for log-aggregator
Use "log-aggregator [command] --help" for more information about a command.
Note This is just a practice exercise and not production ready.
If anyone wants to make this project real or have some good features/roadmap in mind. Feel free to raise a issue. Every contributions to this project is ❤️ welcome.
This project is licence under Apache Licence 2.0