cedar -- Build Data Warehouse

Overview

Cedar is a collection of data-focused tools for data generated by and used in Evergreen builds. It is not a part of evergreen, but a companion service and tool kit. In particular, Cedar focuses on the "offline" data processing cases, and providing access to data to support external user interfaces (e.g. cli tools, data visualizations, etc.)

Build operations generate data, and Evergreen stores most of this data directly: the outcome of tasks, the outcomes of specific tests, artifacts produced by builds. There's other data, specific to particular projects, data that is less naturally structured or less specifically linked to specific execution events. Additionally, cedar provides a platform to collect and analyze data about an evergreen deployment in the large.

Components

Cedar has three core components:

• a simple JSON-only REST web service to data storage and access. (See the rest sub-package.)
• a command line tool that provides a REST client for common operations, as well as independent data collection and exploration tools. (See the operations sub-package.)
• an offline task processing framework based on amboy for application-level data aggregation operations. (See the units sub-package.)

Data Types

• a simple log storage system, as a proof of concept. Logs stream from the client to the cedar application which stores metadata about the logs and then saves data to S3. Out of band, jobs can process log data.
• MongoDB Core Server internal dependency data which stores data produced by the scons dagger target.
• Cost spending data, which is collected from Evergreen and hosting providers' APIs (or static information as needed.)

Use

1. Download and compile cedar. I prefer something like: :

   go get github.com/evergreen-ci/cedar
   cd $GOPATH/src/github.com/evergreen-ci/cedar
   make build
   pushd /usr/local/bin
   ln -s $GOPATH/src/github.com/evergreen-ci/cedar/build/cedar
   popd

2. Explore the help menus in the cedar binary: :

   cedar --help
   cedar cost --help
   cedar dagger --help
   cedar service --help
   cedar client --help
   cedar worker --help

3. Enjoy!

Development

The cedar project uses a makefile to coordinate testing. Use the following command to build the cedar binary: :

make build

The artifact is at build/cedar. The makefile provides the following targets:

test

Runs all tests, sequentially, for all packages.

test-<package>

Runs all tests for a specific package

race, race-<package>

As with their test counterpart, these targets run tests with the race detector enabled.

lint, lint-<package>

Installs and runs the gometaliter with appropriate settings to lint the project.