beefheart

a natural black angus beef heart slow-cooked in a fiery sriracha honey wasabi BBQ sauce and set on a rotating lazy Susan.

Beefheart is a tool that monitors Fastly real-time analytics data and indexes it into Elasticsearch.

quickstart

Prerequisites:

• Build
  • stack
• Run or Demo
  • Fastly API key available in environment variable FASTLY_KEY
  • Docker

The following command will look for your Fastly API key in the environment variable FASTLY_KEY and, by default, send metrics documents to a local Elasticsearch listener at http://localhost:9200.

$ make run

Testing (Docker is required)

$ make test

See available command shortcuts to build, run a demo, and so on.

$ make help

Beefheart comes with a Makefile target to quickly demonstrate its features if a) you have a Fastly account and b) your API key that can read realtime metrics is available in your shell environment under the name FASTLY_KEY. With the key exported to your environment, run the demo target in a shell:

$ make demo

And open http://localhost:5601 to begin creating dashboards from your Fastly metrics. Use Ctrl-C to stop the demo, which will cleanup any Docker containers.

If you'd like to run the application with custom flags or settings, invoke the command with stack, passing in arguments in the following format:

$ stack exec beefheart -- --help
<help output>

Why?

Fastly provides a real-time analytics API that exposes a variety of metrics across services for a given account that include useful insights like transmitted bytes, number of hits, and number of requests. While the default stats dashboard provided in the Fastly management portal is useful, extracting and storing these metrics can provide another level of utility, such as for use cases like monitoring and custom dashboards. Beefheart extracts all (as of the time of this writing) available metrics from this API and stores them in Elasticsearch which permits you to drill down into Fastly service performance deeply. For example, you might want to create a Kibana dashboard that highlights services pushing the most traffic, or visualize what the cache hit to cache miss ratio looks like.

Implementation

As part of Elastic's Space, Time engineering principles, Beefheart was written in Haskell as a learning opportunity. The core operations of a tool like this are not excessively esoteric (that is, consuming metrics from a JSON API, transforming them, and sending them along to another REST API), which provided a good neutral ground to explore principles of Haskell software development.

Note that significant effort has been taken to document the code extensively and use techniques and patterns that are friendly to newcomers. Although the code comments and documentation should help lead a novice through the program's logic, the bulk of this project was written by a newcomer to Haskell, so contributions are welcome!

Beefheart provides useful real-world Haskell examples for:

• Software transactional memory (STM)
• Concurrent programming through the use of async
• HTTP request and response via req and JSON serialization via aeson
• Monitoring threadsafe values via ekg

Maintenance

This project is provided as a utility to the Elastic community and is not an officially supported product. Responsiveness to contributions, bugfixes, and features will be collaboratively supported on a best-effort basis.

Licensing

This project is licensed under the Apache 2 open source license.

FAQ

Why not use Bloodhound instead of req?

Elasticsearch communication is done over plain HTTP libraries instead of Bloodhound because there are a few missing pieces like ILM support from the library. Adding support for missing features to Bloodhound and using it in Beefheart would be great!

Why is this project named "Beefheart"?

The Stack Infrastructure team at Elastic has a tradition of naming some projects after our favorite celebrities. Guy Fieri is one of our honored team mascots, and the name for this project was picked from a satirical restaurant menu.

Wishlist

Beefheart tries to avoid overly complex abstractions, so if you're new to the language and are itching to try some Haskell, please do! Contributions are welcome. Here are some ideas:

• Standardized bracket or json log formatting everywhere in the code (for example, picking out any simple printed output)
• Add the ability to send EKG metrics to Elasticsearch
• Enrich point-of-presence codes with human-readable names using the datacenter API