extract whitebox monitoring data from application logs for collection in a timeseries database
jaqx0r Delete data from metrics after an expiration time.
This adds to the language a form `del .. after`, which indicates that after a
period of time has passed, if the datum has not been updated, then it should be
removed from memory.

This form is more convenient when the close signal event can be missed, leading
to extra memory usage.

Issue: #84, #172
Latest commit 6e1ac92 Oct 14, 2018
Permalink
Failed to load latest commit information.
.circleci Fix typo in the circle config. Sep 5, 2018
docs Update docs. Oct 15, 2018
emgen Use for-range instead of for-select form in emgen. Feb 9, 2018
examples Improve apache metrics histogram example Oct 2, 2018
exporter Prevent text metrics from being exposed on endpoints that don't suppo… Jun 16, 2018
fuzz More renames missed by tests. Mar 5, 2015
hooks add post_push tagging for semantic versioning Apr 19, 2018
logline Move logline, and start a new file implementation. Sep 5, 2018
metrics Emit expire code and execute in VM. Oct 15, 2018
mtail Run an expiration loop every hour. Sep 23, 2018
tailer Make the log_glob_test actually use a glob. Sep 22, 2018
testdata Let stringy.mtail be loaded with other programs (due to variable name… Jun 16, 2018
tests clean up log_glob_test. Oct 15, 2018
testutil Create a new metric type Text for storing text. Jun 16, 2018
vm Merge branch 'master' into dellater Oct 15, 2018
watcher Apply a change recommended by megacheck lint. Jun 10, 2018
.gitattributes Check in the generated files now we can exclude them from review. Aug 24, 2018
.gitignore Remove verbosity Oct 15, 2018
.gitlab-ci.yml Fix gitlab-ci stage test failing May 28, 2018
.travis.yml No deploy from Travis. Sep 5, 2018
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Sep 20, 2017
CONTRIBUTING.md Update contribution guide to indicate response time. Apr 22, 2018
Dockerfile Install dependencies when building Docker image Jul 26, 2018
ISSUE_TEMPLATE.md Create Github ISSUE_TEMPLATE.md Feb 18, 2018
LICENSE Add CONTRIBUTING and LICENSE. Mar 2, 2015
Makefile Adds support for go-fuzz to the compiler. Sep 10, 2018
README.md Update README for building from source. Jul 2, 2018
TODO Update todo Sep 23, 2018
bench_test.go Create separate directories for the metric reload test, due to a race in May 21, 2018
benchmark_results.csv Read up to 4K from each log. Sep 9, 2013
ex_test.go Fix test issues with running the end to end test. Jun 16, 2018
log_rewriter.awk Fix an output formatting bug in the rewriter script. Apr 5, 2015
logo.png Adds project logo. Sep 20, 2017
main.go Fix error formatting. Sep 19, 2018
mtail-mode.el Rename from emtail to mtail, restoring harmony in the universe. May 28, 2014

README.md

mtail

mtail - extract whitebox monitoring data from application logs for collection into a timeseries database

GoDoc CircleCI Build Status Coverage Status

mtail is a tool for extracting metrics from application logs to be exported into a timeseries database or timeseries calculator for alerting and dashboarding.

It aims to fill a niche between applications that do not export their own internal state, and existing monitoring systems, without patching those applications or rewriting the same framework for custom extraction glue code.

The extraction is controlled by mtail programs which define patterns and actions:

# simple line counter
counter line_count
/$/ {
  line_count++
}

Metrics are exported for scraping by a collector as JSON or Prometheus format over HTTP, or can be periodically sent to a collectd, StatsD, or Graphite collector socket.

Read the programming guide if you want to learn how to write mtail programs.

Mailing list: https://groups.google.com/forum/#!forum/mtail-users

Installation

There are various ways of installing mtail.

Precompiled binaries

Precompiled binaries for released versions are available in the Releases page on Github. Using the latest production release binary is the recommended way of installing mtail.

Windows, OSX and Linux binaries are available.

Building from source

To build mtail from the source code yourself you need to have a working Go environment with version 1.9 or greater installed, also some additional tools like goyacc. You can't go get the software and have it build straight away. Sorry.

See the Build instructions for more details.

A Dockerfile is included in this repository for local development as an alternative to installing Go in your environment, and takes care of all the build dependency installation, if you don't care for that.

Deployment

mtail works best when it paired with a timeseries-based calculator and alerting tool, like Prometheus.

So what you do is you take the metrics from the log files and you bring them down to the monitoring system?

It deals with the instrumentation so the engineers don't have to! It has the extraction skills! It is good at dealing with log files!!