Skip to content
The Heroic Time Series Database
Java Kotlin Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.cache [build] move checkstyle cache to own directory Dec 15, 2015
.github Make the stale issue bot less aggressive. (#513) Jul 10, 2019
aggregation Deprecate events and remove the partial events implementation (#566) Oct 10, 2019
assets [assets] add assets and logo May 26, 2016
consumer
debian [debian] fix build process (#473) May 3, 2019
discovery/simple
docs
example Fix a couple of issues with Heroic minimal setup (#529) Aug 21, 2019
gradle/wrapper [usagetracking] Create a module for tracking basic events. (#582) Nov 20, 2019
heroic-component [tracing] fix NPE (#587) Nov 27, 2019
heroic-core [usagetracking] Send an uptime event every 24 hours. (#584) Nov 20, 2019
heroic-dist [usagetracking] Create a module for tracking basic events. (#582) Nov 20, 2019
heroic-elasticsearch-utils [component] Partially remove lombok. (#543) Sep 20, 2019
heroic-test Add tracing to the query path for bigtable (#569) Oct 18, 2019
idea [idea] update code style to include layout table Jan 18, 2017
metadata Take resource tags into account when generating the hash tag and crea… Oct 7, 2019
metric Add tracing to the query path for bigtable (#569) Oct 18, 2019
repackaged/datastax [core,elasticsearch] Stop repackaging folsom. (#528) Aug 9, 2019
rpc Add tracing to the query path for bigtable (#569) Oct 18, 2019
src/main/resources initial commit Jun 3, 2015
statistics/semantic Add tracing to the query path for bigtable (#569) Oct 18, 2019
suggest Take resource tags into account when generating the hash tag and crea… Oct 7, 2019
tools
usage-tracking [usage-tracking] set timeout and close body (#588) Nov 27, 2019
.dockerignore Add git to the docker build. (#578) Nov 6, 2019
.editorconfig [all] pom.xml files should have 2 space indentation Dec 14, 2016
.gitignore Setup local config for serving documentation pages. (#447) Apr 2, 2019
.travis.yml Update dependencies to support Java 9+. (#559) Oct 7, 2019
CONTRIBUTING.md Add updated contributing guidelines (#461) Apr 16, 2019
Dockerfile Add git to the docker build. (#578) Nov 6, 2019
LICENSE Fix copyright info in LICENSE file (#510) Jun 24, 2019
README.md
build.gradle Fallback to DOCKER_TAG when setting version. (#585) Nov 20, 2019
checkstyle.xml Switch build/tests to use gradle instead of maven (#429) Mar 18, 2019
codecov.yml Disable codecov status checks. (#484) May 10, 2019
gradlew Update dependencies to support Java 9+. (#559) Oct 7, 2019
guide-to-dagger2.md
logo.42.png [README] add logo May 26, 2016
rfcs.md Move static assets from gh-pages branch. (#438) Mar 22, 2019
run-heroic.sh fix path to java executable in the openjdk:8 container (#509) Jun 24, 2019
settings.gradle [usagetracking] Create a module for tracking basic events. (#582) Nov 20, 2019
suppressions.xml Switch build/tests to use gradle instead of maven (#429) Mar 18, 2019

README.md

Heroic Heroic

Build Status Codecov License

A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. Go to https://spotify.github.io/heroic/ for documentation.

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

Stability Disclaimer: Heroic is an evolving project, and should in its current state be considered unstable. Do not use in production unless you are willing to spend time with it, experiment and contribute. Not doing so might result in losing your data to goblins. It is currently not on a release schedule and is not versioned. At Spotify we rely on multiple release forks that we actively maintain and flip between.

Building

Java 11 is required.

There are some repackaged dependencies that you have to make available, you do this by running tools/install-repackaged.

$ tools/install-repackaged
Installing repackaged/x
...

After this, the project is built using Gradle:

# full build, runs all tests and builds the shaded jar
./gradlew build

# only compile
./gradlew assemble

# build a single module
./gradlew heroic-metric-bigtable:build

The heroic-dist module can be used to produce a shaded jar that contains all required dependencies:

./gradlew heroic-dist:shadowJar

Running

After building, the entry point of the service is com.spotify.heroic.HeroicService. The following is an example of how this can be run:

./gradlew heroic-dist:runShadow <config>

which is the equivalent of doing:

java -jar $PWD/heroic-dist/build/libs/heroic-dist-0.0.1-SNAPSHOT-shaded.jar <config>

For help on how to write a configuration file, see the Configuration Section of the official documentation.

Heroic has been tested with the following services:

Building/Running via docker

A docker container with the shaded jar is now available. To build the container:

$ docker build -t heroic:latest .

This is a multi-stage build and will first build Heroic via a ./gradlew clean build and then copy the resulting shaded jar into the runtime container.

Running heroic via docker can be done:

$ docker run -d -p 8080:8080 -p 9091:9091 -v /path/to/config.yml:/heroic.yml spotify/heroic:latest

Logging

Logging is captured using SLF4J, and forwarded to Log4j.

To configure logging, define the -Dlog4j.configurationFile=<path> parameter. You can use docs/log4j2-file.xml as a base.

Testing

We run tests with Gradle:

# Start pubsub emulator
gcloud beta emulators pubsub start

# run unit tests
./gradlew test

# run integration tests
./gradlew integrationTest

or to run a more comprehensive set of checks:

./gradlew check

This will run:

It is strongly recommended that you run the full test suite before setting up a pull request, otherwise it will be rejected by Travis.

Remote Integration Tests

Integration tests are configured to run remotely depending on a set of system properties.

Elasticsearch
Property Description
-D elasticsearch.version=<version> Use the given client version when building the project
-D it.elasticsearch.remote=true Run Elasticsearch tests against a remote database
-D it.elasticsearch.seed=<seed> Use the given seed (default: localhost)
-D it.elasticsearch.clusterName=<clusterName> Use the given cluster name (default: elasticsearch)
Datastax
Property Description
-D datastax.version=<version> Use the given client version when building the project
-D it.datastax.remote=true Run Datastax tests against a remote database
-D it.datastax.seed=<seed> Use the given seed (default: localhost)
Bigtable
Property Description
-D bigtable.version=<version> Use the given client version when building the project
-D it.bigtable.remote=true Run Bigtable tests against a remote database
-D it.bigtable.project=<project> Use the given project
-D it.bigtable.zone=<zone> Use the given zone
-D it.bigtable.instance=<instance> Use the given instance
-D it.bigtable.credentials=<credentials> Use the given credentials file

The following is an example Elasticsearch remote integration test:

$> mvn -P integration-tests \
    -D elasticsearch.version=5.6.0 \
    -D it.elasticsearch.remote=true \
    clean verify
PubSub

PubSub relies on having the PUBSUB_EMULATOR_HOST environment variable set instead of a system property. Detailed instructions are available in the Google PubSub emulator docs.

Full Cluster Tests

Full cluster tests are defined in heroic-dist/src/test/java.

This way, they have access to all the modules and parts of Heroic.

The JVM RPC module is specifically designed to allow for rapid execution of integration tests. It allows multiple cores to be defined and communicate with each other in the same JVM instance.

Coverage

Coverage

There's an ongoing project to improve test coverage. Clicking the above graph will bring you to codecov.io, where you can find areas to focus on.

Building a Debian Package

This project does not provide a single debian package, this is primarily because the current nature of the service (alpha state) does not mesh well with stable releases.

Instead, you are encouraged to build your own using the provided scripts in this project.

First run the prepare-sources script:

$ debian/bin/prepare-sources myrel 1

myrel will be the name of your release, it will be part of your package name debian-myrel, it will also be suffixed to all helper tools (e.g. heroic-myrel).

For the next step you'll need a Debian environment:

$ dpkg-buildpackage -uc -us

If you encounter problems, you can troubleshoot the build with DH_VERBOSE:

$ env DH_VERBOSE=1 dpkg-buildpackage -uc -us

Contributing

Guidelines for contributing can be found here.

Module Orientation

The Heroic project is split into a couple of modules.

The most critical one is heroic-component. It contains interfaces, value objects, and the basic set of dependencies necessary to glue different components together.

Submodules include metric, suggest, metadata, and aggregation. The first three contain various implementations of the given backend type, while the latter provides aggregation methods.

heroic-core contains the com.spotify.heroic.HeroicCore class which is the central building block for setting up a Heroic instance.

heroic-elasticsearch-utils is a collection of utilities for interacting with Elasticsearch. This is separate since we have more than one backend that needs to talk with elasticsearch.

Anything in the repackaged directory is dependencies that include one or more Java packages that must be relocated to avoid conflicts. These are exported under the com.spotify.heroic.repackaged groupId.

Finally there is heroic-dist, a small project that depends on all module. Here is where everything is bound together into a distribution — a shaded jar. It also provides the entry-point for services, namely com.spotify.heroic.HeroicService or through an interactive shell com.spotify.heroic.HeroicShell. The shell can either be run standalone or connected to an existing Heroic instance for administration.

Bypassing Validation

To bypass automatic formatting and checkstyle validation you can use the following stanza:

// @formatter:off
final List<String> list = ImmutableList.of(
   "Welcome to...",
   "... The Wild West"
);
// @formatter:on

To bypass a FindBugs error, you should use the @SupressFBWarnings annotation.

@SupressFBWarnings(value="FINDBUGS_ERROR_CODE", justification="I Know Better Than FindBugs")
public class IKnowBetterThanFindbugs() {
    // ...
}

HeroicShell

Heroic comes with a shell that contains many useful tasks, these can either be run in a readline-based shell with some basic completions and history, or standalone.

You can use the following helper script to run the shell directly from the project.

$ tools/heroic-shell [opts]

There are a few interesting options available, most notably is --connect that allows the shell to connect to a remote heroic instance.

See -h for a full listing of options.

You can run individual tasks in standalone mode, giving you a bit more options (like redirecting output) through the following.

$ tools/heroic-shell <heroic-options> -- com.spotify.heroic.shell.task.<task-name> <task-options>

There are also profiles that can be activated with the -P <profile> switch, available profiles are listed in --help.

Repackaged Dependencies

These are third-party dependencies that has to be repackaged to avoid binary incompatibilities with dependencies.

Every time these are upgraded, they must be inspected for new conflicts. The easiest way to do this, is to build the project and look at the warnings for the shaded jar.

$> ./gradlew clean assemble
...
[WARNING] foo-3.5.jar, foo-4.5.jar define 10 overlapping classes:
[WARNING]   - com.foo.ConflictingClass
...

This would indicate that there is a package called foo with overlapping classes.

You can find the culprit using the dependencies task.

$> ./gradlew <module>:dependencies
You can’t perform that action at this time.