GO Framework to create an input/filter/ouput pipeline.
The goal is to provide a framework that allows to model generic ETLs inspired by Logstash.
The framework provides a tick-channel, ticking along every once so often (5s by default).
Data channel moves messages from collectors to handlers and allows any number of filters in between.
Each plugin is its own golang project. Thus, it is easily written and interchangeable. Furthermore the plan is to allow the use of GOLANG plugins, so that each plugin can be build as shared object and dynamically loaded, without the need to compile it into the resulting daemon.
- collector: Input plugin producing messages
- filter: plugin to refine/alter messages from collectors or other filters
- handler: output plugin to send/output the data
The following plugins are available.
- docker-events Hooks into moby's
/eventsAPI endpoint and parses incoming events like
contianer.createor `network.attach. For now SWARM events are not provided, but there is already a PR against moby (former called docker) on github.
- docker-stats For each incoming
docker-eventabout a started container, this collector will spawn a goroutine to stream the /container//stats` API call. Thus, the collector gets (as close as possible) real-time metrics for a container.
- docker-log Similar to the
docker-statscollector, a goroutine is started for each container, which subscribes the stream of stdout/stderr output via the docker API.
- gelf Collector for the GELF log-driver of the docker-engine. As it needs a container to be started using the
docker-logcontainer is easier to handle.
- tcp Opens a TCP port which should be used by a container to send messages like AppMetrics.
By using the
inventoryfilter the metadata will be added according to the remote-IP used by the container.
- file Simple collector to tail a file.
- internal Samples internal metrics of the GO program (like counting the goroutines, memory usage and such).
- statsq Caches StatsQ (StatsD plus tags) metrics and flushes them as metrics.
- health Keeps track of health messages within the framework and provides
- inventory Listens to
docker-eventsand keeps an inventory of all containers. Can be queried by other plugins sending
ContainerRequests down theData` channel.
- id Relays the message - might be droped as it was used for reversing events.
- grok Allows for matching messages with GROK patterns (typed RegEx, much nicer to use then RegExp).
- docker-stats Potential filter to aggregate or transform metrics comming from the
- metrics Assumes the keys
tagsand transforms it to a
- log Outputs to stdout of the daemon.
- influxdb Forwards metrics to an InfluxDB server.
- elasticsearch FOrwards