cedar -- Build Data Warehouse

Overview

Cedar is a collection of data-focused tools for data generated by and used in Evergreen builds. It is not a part of Evergreen, but serves as a companion service and tool kit. In particular, Cedar focuses on the "offline" data processing cases, and providing access to data to support external user interfaces (e.g. cli tools, data visualizations, etc.)

Build operations generate data, and Evergreen stores most of this data directly (e.g. the outcome of tasks, artifacts produced by builds). There's other data, specific to particular projects, that is less naturally structured or less specifically linked to specific execution events. Additionally, Cedar provides a platform to collect and analyze data about an Evergreen deployment in the large.

Components

• A simple JSON-only REST web service for data access (see the rest package).
• A gRPC service for data storage (see the rpc package).
• A command line tool that provides a REST client for common operations, as well as independent data collection and exploration tools (see the operations package).
• An offline task processing framework based on amboy for application-level data aggregation operations (see the units package).

Data Types ~~~~~~~~~~

• Logs for Evergreen tasks.
• Test result logs and metadata for Evergreen tasks.
• A simple log storage system, as a proof of concept. Logs stream from the client to the Cedar application which stores metadata about the logs and then saves data to S3. Out of band, jobs can process log data.

Use ---

1. Download and compile Cedar. I prefer something like:

   git clone https://github.com/evergreen-ci/cedar
   cd cedar
   make cedar
   

2. Explore the help menus in the cedar binary:

   cedar --help
   cedar service --help
   cedar client --help
   cedar worker --help
   

3. Enjoy!

Development -----------

The Cedar project uses a makefile to coordinate testing. Use the following command to build the Cedar binary:

make cedar

The artifact is at build/cedar. The makefile provides the following targets:

test Runs all tests, sequentially, for all packages. Some of the tests require a replica set to be running under http://127.0.0.1:27017.

test-<package> Runs all tests for a specific package

lint, lint-<package> Installs and runs the gometaliter with appropriate settings to lint the project.