Java Other
Clone or download
nabam and nezdolik [aggregation] Delta per second, non negative (#365)
New aggregations:

* Calculate delta per second

* NotNegative aggregation
Latest commit be5f2d0 Jul 5, 2018
Permalink
Failed to load latest commit information.
.cache [build] move checkstyle cache to own directory Dec 15, 2015
aggregation [aggregation] Delta per second, non negative (#365) Jul 5, 2018
assets [assets] add assets and logo May 26, 2016
consumer [all] Add metric for consumer retries Jun 22, 2017
debian [all] disable heavier tests by default May 17, 2016
discovery/simple [all] A shard with no discovered nodes is an error Mar 31, 2017
docs [docs/rfc] fix link May 19, 2017
heroic-all [all] introduce sub-system for conditional features Jun 30, 2017
heroic-component-test [test-base,component-test] move test-base -> component-test Feb 2, 2017
heroic-component [component] fix calculation of common tags and resources Jan 11, 2018
heroic-core [core] Make host field optional in Spotify100 Feb 23, 2018
heroic-dist [aggregation] Delta per second, non negative (#365) Jul 5, 2018
heroic-elasticsearch-test [all] break test modules into heroic-test-base, heroic-test-it May 31, 2016
heroic-elasticsearch-utils [metadata,suggest,elasticsearch-utils] actually block in rate-limiter Jan 9, 2018
heroic-loading [component,loading] use standard serialization method for MetricColle… Sep 21, 2017
heroic-parser [parser] add test for simple string starting with number Sep 18, 2016
heroic-shell HeroicShell should return exit code 1 in case of errors Sep 18, 2016
heroic-test-it [all] Add resource identifiers Jan 11, 2018
heroic-test [test] introduce FakeModuleLoader (fixes #110) Feb 2, 2017
idea [idea] update code style to include layout table Jan 18, 2017
metadata [all] report cache sizes Jan 9, 2018
metric [bigtable] Use default credentials Feb 9, 2018
repackaged Revert "[bigtable] Bump BT client version to 0.9.7.1" Nov 9, 2017
reporting [rpc/nativerpc] deprecate module Oct 9, 2016
rpc Revert "[rpc/grpc] Bump grpc version to 1.3.0" Nov 9, 2017
src/main/resources initial commit Jun 3, 2015
statistics/semantic [all] Add resource identifiers Jan 11, 2018
suggest [all] report cache sizes Jan 9, 2018
tools [all] Add preAggregationSampleSize in query log May 18, 2017
.editorconfig [all] pom.xml files should have 2 space indentation Dec 14, 2016
.gitignore [gitignore] Ignore *.p12 Feb 8, 2018
.travis.yml [travis] enable container-based build Dec 14, 2016
LICENSE initial commit Jun 3, 2015
README.md Typo on string #45 Jun 11, 2018
checkstyle.xml [build] move checkstyle cache to own directory Dec 15, 2015
logo.42.png [README] add logo May 26, 2016
pom.xml [statistics/semantic] More MinMaxSlidingTimeReservoir tests Jul 5, 2017
suppressions.xml [all] fix checkstyle violations Nov 16, 2015

README.md

Heroic Heroic

Build Status Codecov License Join the chat at https://gitter.im/spotify/heroic

A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. Go to https://spotify.github.io/heroic/ for documentation, please join #heroic at Freenode if you need help or want to chat.

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

Stability Disclaimer: Heroic is an evolving project, and should in its current state be considered unstable. Do not use in production unless you are willing to spend time with it, experiment and contribute. Not doing so might result in losing your data to goblins. It is currently not on a release schedule and is not versioned. At Spotify we rely on multiple release forks that we actively maintain and flip between.

Building

Java 8 is required.

There are some repackaged dependencies that you have to make available, you do this by running tools/install-repackaged.

$ tools/install-repackaged
Installing repackaged/x
...

After this, the project is built using Maven:

$ mvn package

This will cause the heroic-dist module to produce a shaded jar that contains all required dependencies.

Running

After building, the entry point of the service is com.spotify.heroic.HeroicService. The following is an example of how this can be run:

$ java -cp $PWD/heroic-dist/target/heroic-dist-0.0.1-SNAPSHOT-shaded.jar com.spotify.heroic.HeroicService <config>

For help on how to write a configuration file, see the Configuration Section of the official documentation.

Heroic has been tested with the following services:

Logging

Logging is captured using SLF4J, and forwarded to Log4j.

To configure logging, define the -Dlog4j.configurationFile=<path> parameter. You can use docs/log4j2-file.xml as a base.

Testing

We run unit tests with Maven:

$ mvn test

A more comprehensive test suite is enabled with the environment=test property.

$ mvn -D environment=test verify

This adds:

It is strongly recommended that you run the full test suite before setting up a pull request, otherwise it will be rejected by Travis.

Remote Integration Tests

Integration tests are configured to run remotely depending on a set of system properties.

Elasticsearch
Property Description
-D elasticsearch.version=<version> Use the given client version when building the project
-D it.elasticsearch.remote=true Run Elasticsearch tests against a remote database
-D it.elasticsearch.seed=<seed> Use the given seed (default: localhost)
-D it.elasticsearch.clusterName=<clusterName> Use the given cluster name (default: elasticsearch)
Datastax
Property Description
-D datastax.version=<version> Use the given client version when building the project
-D it.datastax.remote=true Run Datastax tests against a remote database
-D it.datastax.seed=<seed> Use the given seed (default: localhost)
Bigtable
Property Description
-D bigtable.version=<version> Use the given client version when building the project
-D it.bigtable.remote=true Run Bigtable tests against a remote database
-D it.bigtable.project=<project> Use the given project
-D it.bigtable.zone=<zone> Use the given zone
-D it.bigtable.instance=<instance> Use the given instance
-D it.bigtable.credentials=<credentials> Use the given credentials file

The following is an example Elasticsearch remote integration test:

$> mvn -P integration-tests \
    -D elasticsearch.version=1.7.5 \
    -D it.elasticsearch.remote=true \
    clean verify

Full Cluster Tests

Full cluster tests are defined in heroic-dist/src/test/java.

This way, they have access to all the modules and parts of Heroic.

The JVM RPC module is specifically designed to allow for rapid execution of integration tests. It allows multiple cores to be defined and communicate with each other in the same JVM instance.

Coverage

Coverage

There's an ongoing project to improve test coverage. Clicking the above graph will bring you to codecov.io, where you can find areas to focus on.

Speedy Building

For a speedy build without tests and checks, you can run:

$ mvn -D maven.test.skip=true package

Building a Debian Package

This project does not provide a single debian package, this is primarily because the current nature of the service (alpha state) does not mesh well with stable releases.

Instead, you are encouraged to build your own using the provided scripts in this project.

First run the prepare-sources script:

$ debian/bin/prepare-sources myrel 1

myrel will be the name of your release, it will be part of your package name debian-myrel, it will also be suffixed to all helper tools (e.g. heroic-myrel).

For the next step you'll need a Debian environment:

$ dpkg-buildpackage -uc -us

If you encounter problems, you can troubleshoot the build with DH_VERBOSE:

$ env DH_VERBOSE=1 dpkg-buildpackage -uc -us

Contributing

Fork the code at https://github.com/spotify/heroic

Make sure you format the code using the provided formatter in idea. Even if you disagree with the way it is formatted, consistency is more important. For special cases, see Bypassing Validation.

If possible, limit your changes to one module per commit. If you add new, or modify existing classes. Keep that change to a single commit while maintaing backwards compatible behaviour. Deprecate any old APIs as appropriate with @Deprecated and add documentation for how to use the new API.

The first line of the commit should be formatted with [module1,module2] my message.

module1 and module2 are paths to the modules affected with any heroic- prefix stripped. So if your change affects heroic-core and metric/bigtable, the message should say [core,metric/bigtable] did x to y.

If more than 3 modules are affected by a commit, use [all]. For other cases, adapt to the format of existing commit messages.

Before setting up a pull request, run the comprehensive test suite as specified in Testing.

Module Orientation

The Heroic project is split into a couple of modules.

The most critical one is heroic-component. It contains interfaces, value objects, and the basic set of dependencies necessary to glue different components together.

Submodules include metric, suggest, metadata, and aggregation. The first three contain various implementations of the given backend type, while the latter provides aggregation methods.

heroic-core contains the com.spotify.heroic.HeroicCore class which is the central building block for setting up a Heroic instance.

heroic-elasticsearch-utils is a collection of utilities for interacting with Elasticsearch. This is separate since we have more than one backend that needs to talk with elasticsearch.

heroic-parser provides an Antlr4 implementation of com.spotify.heroic.grammar.QueryParser, which is used to parse the Heroic DSL.

heroic-shell contains com.spotify.heroic.HeroicShell, a shell capable of either running a standalone, or connecting to an existing Heroic instance for administration.

heroic-all contains dependencies and references to all modules that makes up a Heroic distribution. This is also where profiles are defined since they need to have access to all dependencies.

Anything in the repackaged directory is dependencies that include one or more Java packages that must be relocated to avoid conflicts. These are exported under the com.spotify.heroic.repackaged groupId.

Finally there is heroic-dist, a small project that depends on heroic-all, heroic-shell, and a logging implementation. Here is where everything is bound together into a distribution — a shaded jar. It also provides the entry-point for services, namely com.spotify.heroic.HeroicService.

Bypassing Validation

To bypass automatic formatting and checkstyle validation you can use the following stanza:

// @formatter:off
final List<String> list = ImmutableList.of(
   "Welcome to...",
   "... The Wild West"
);
// @formatter:on

To bypass a FindBugs error, you should use the @SupressFBWarnings annotation.

@SupressFBWarnings(value="FINDBUGS_ERROR_CODE", justification="I Know Better Than FindBugs")
public class IKnowBetterThanFindbugs() {
    // ...
}

HeroicShell

Heroic comes with a shell that contains many useful tasks, these can either be run in a readline-based shell with some basic completions and history, or standalone.

You can use the following helper script to run the shell directly from the project.

$ tools/heroic-shell [opts]

There are a few interesting options available, most notably is --connect that allows the shell to connect to a remote heroic instance.

See -h for a full listing of options.

You can run individual tasks in standalone mode, giving you a bit more options (like redirecting output) through the following.

$ tools/heroic-shell <heroic-options> -- com.spotify.heroic.shell.task.<task-name> <task-options>

There are also profiles that can be activated with the -P <profile> switch, available profiles are listed in --help.

Repackaged Dependencies

These are third-party dependencies that has to be repackaged to avoid binary incompatibilities with dependencies.

Every time these are upgraded, they must be inspected for new conflicts. The easiest way to do this, is to build the project and look at the warnings for the shaded jar.

$> mvn clean package -D maven.test.skip=true
...
[WARNING] foo-3.5.jar, foo-4.5.jar define 10 overlapping classes:
[WARNING]   - com.foo.ConflictingClass
...

This would indicate that there is a package called foo with overlapping classes.

You can find the culprit using the dependency plugin.

$> mvn package dependency:tree