WIP: A dashboard for CI
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.artifacts build: Improve utilization of Black and pre-commit (#116) Jun 8, 2018
.vscode build: Improve utilization of Black and pre-commit (#116) Jun 8, 2018
bin fix: Dont recursively chown repo root Jul 24, 2018
config test: Add tests for UserBuildList Jul 30, 2018
kubernetes/zeus ref: Remove unneeded json call Jul 24, 2018
public initial skeleton Jul 3, 2017
scripts test: Correct locale on remaining date usages Jan 6, 2018
templates feat: Add footer with build/github Jul 24, 2018
tests feat: Various improvements to statistics Oct 2, 2018
webapp feat: Link active repos Oct 18, 2018
zeus feat: Various improvements to statistics Oct 2, 2018
.babelrc feat: Expand tests Jan 5, 2018
.dockerignore feat: Use pipenv for dev/test Oct 1, 2018
.editorconfig meta: Add prettierrc to match prettier hook Nov 24, 2017
.envrc feat: Use pipenv for dev/test Oct 1, 2018
.eslintignore add prettier and eslint pre-commit hooks Jul 12, 2017
.gitignore feat: Use pipenv for dev/test Oct 1, 2018
.nvmrc Switch to Node 8 Jul 24, 2018
.pre-commit-config.yaml ref: Expand pre-commit checks Jul 24, 2018
.prettierignore feat: Expand tests Jan 5, 2018
.prettierrc meta: Add prettierrc to match prettier hook Nov 24, 2017
.travis.yml feat: Use pipenv for dev/test Oct 1, 2018
Dockerfile bypass '/dev/tty' gpg bug Nov 16, 2018
LICENSE lint the world Jul 8, 2017
Makefile feat: Use pipenv for dev/test Oct 1, 2018
Pipfile feat: Use pipenv for dev/test Oct 1, 2018
Pipfile.lock fix: Incorrect version of Flask in Pipfile.lock Oct 1, 2018
README.md feat: Use pipenv for dev/test Oct 1, 2018
cloudbuild.yaml feat: Add cloudbuild configuration Jan 5, 2018
conftest.py build: Apply Black linter Apr 23, 2018
docker-compose.yml initial docker compose file Aug 11, 2017
package.json ref: Upgrade jest, babel jest Jul 30, 2018
requirements-base.txt fix: Bump Flask to 0.12.3+ Oct 1, 2018
requirements-dev.txt feat: Switch to Python 3.7 Jul 23, 2018
requirements-test.txt test: Upgrade base test requirements Oct 5, 2017
setup.cfg build: Improve utilization of Black and pre-commit (#116) Jun 8, 2018
setup.py build: Apply Black linter Apr 23, 2018
webpack.config.js initial pass at cleaning up webpack Aug 15, 2017
yarn.lock ref: Upgrade jest, babel jest Jul 30, 2018

README.md

Zeus

This project is under development.

Zeus is a frontend and analytics provider for CI solutions. It is inspired by the work done at Dropbox on Changes.

User Guide

Currently Zeus publicly supports GitHub.com as well as easy integration with Travis CI.

To add a new project:

  1. Add a repository (via settings).
  2. Go to the repository's settings and generate a new Hook.
  3. Bind ZEUS_HOOK_BASE as a secret environment variable in Travis.
  4. Update your .travis.yml to include the Zeus webhook.
  5. (Optional) Update your .travis.yml to include artifact upload.
  6. (Optional, not yet recommended) Update your .travis.yml to disable Travis' native email notifications.

Once you've added a project Zeus will automatically update with details from any builds you've run.

Some quick caveats:

  • The project is still pretty early on, and may break/change without warning.
  • travis-ci.com and GitHub Enterprise are not yet supported.
  • Notifications will only be triggered for users which have authenticated against Zeus.

If you want to use Zeus with a build system that's not currently supported, see the details on "Hooks" in the documentation.

Supported Artifact Types

While you can upload any kind of Artifact to zeus (e.g. .html output), the platform has knowledge of certain types and will grant additional functionality if they're present.

The recommended way to support artifacts is to configure a post-build step (on both failure and success) to do something similar to the following:

npm install -g @zeus-ci/cli
$(npm bin -g)/zeus upload -t "application/x-junit+xml" jest.junit.xml
$(npm bin -g)/zeus upload -t "application/x-cobertura+xml" coverage.xml

Code Coverage

  • application/x-clover+xml
  • application/x-cobertura+xml

xUnit

  • application/x-bitten+xml
  • application/x-junit+xml
  • application/x-xunit+xml

Style Checks

  • application/x-checkstyle+xml
  • text/x-pep8
  • text/x-pycodestyle
  • test/x-pylint

Webpack Stats

Webpack stats can be generated with:

webpack --profile --json > webpack-stats.json

They should be submitted with the application/x-webpack-stats+json type.

Contributing

Requirements

  • Python 3.7
  • Node
  • Postgres 9.4+

Setup

# install pipenv
brew update && brew install pipenv

# load dependencies
make

# initialize config
pipenv run zeus init

Note, before running any future Python commands (including zeus), you'll need to activate the environment:

pipenv shell

You can also setup direnv to automatically activate the environment.

Once dependencies are resolved, bootstrap the database (see Makefile for details):

make reset-db

Finally, launch the webserver:

zeus devserver

# or alternatively, with workers:
zeus devserver --workers

Managing Python dependencies

We use Pipenv for environment management, but due to the way Docker layering works, we dont declare dependencies inside of Pipfile. This means adding a new dependency has an extra step.

  • Add the dependency to the appropriate requirements-[env].txt file
  • Run pipenv update

Getting some data

$ zeus repos add https://github.com/getsentry/zeus.git

Once you've authenticated, give yourself access to the repository:

$ zeus repos access add https://github.com/getsentry/zeus.git [you@example.com]

Additionally, you can generate some mock data:

$ zeus mocks load-all

Layout

zeus
├── setup.py                // server dependencies
├── zeus                    // server code
|   ├── artifacts           // artifact handlers
|   ├── api
|   |   ├── resources       // api endpoints/resources
|   |   └── schemas         // api serializer/schemas
|   ├── cli                 // command line utilities
|   ├── models              // database schema
|   ├── storage             // file storage implementations
|   ├── tasks               // async task definitions
|   ├── vcs                 // version control system implementations
|   └── web                 // server-rendered web views
├── templates               // server-rendered templates
├── public                  // general static assets
├── package.json            // web client dependencies
└── webapp                  // web client
    ├── actions             // redux actions
    ├── components          // react components
    ├── reducers            // redux reducers
    ├── routes.js           // routes (react-router)
    └── pages.js            // react components (pages)

Data Model

  • Most models contain a GUID (UUID) primary key.
  • Some generalized models (such as ItemStat) are keyed by GUID, and do not contain backrefs or constraints.
  • Access is controlled at the repository level, and is generally enforced if you use the {ModelClass}.query utilities.
zeus
├── ApiToken
|   └── ApiTokenRepositoryAccess
├── Hook
├── Repository
|   ├── RepositoryAccess
|   ├── ItemOption
|   ├── Build
|   |   ├── ItemStat
|   |   ├── Source
|   |   ├── FileCoverage
|   |   └── Job
|   |       ├── Artifact
|   |       ├── ItemStat
|   |       └── TestCase
|   |           ├── Artifact
|   |           └── ItemStat
|   ├── ChangeRequest
|   |   └── Revision
|   └── Source
|       ├── Author
|       ├── Patch
|       └── Revision
|           └── Author
└── User
    ├── Email
    └── Identity

Hooks

A subset of APIs are exposed using simple hook credentials. These credentials are coupled to a provider (e.g. travis-ci) and a single repository.

To create a new hook:

zeus hooks add https://github.com/getsentry/zeus.git travis-ci

Using the subpath, you'll be able to access several endpoints:

  • {prefix}/builds/{build-external-id}
  • {prefix}/builds/{build-external-id}/jobs/{job-external-id}
  • {prefix}/builds/{build-external-id}/jobs/{job-external-id}/artifacts

The prefix will be generated for you as part of the a new hook, and is made up of the Hook's GUID and it's signature:

http://example.com/hooks/{hook-id}/{hook-signature}/{path}

Each endpoint takes an external ID, which is used as a unique query parameter. The constraints are coupled to the parent object. For example, to create or patch a build:

POST http://example.com/hooks/{hook-id}/{hook-signature}/builds/abc

This will look for a Build object with the following characteristics:

  • provider={Hook.provider}
  • external_id=abc
  • repository_id={Hook.repository_id}

If a match is found, it will be updated with the given API parameters. If it isn't found, it will be created. All of these operations are treated like a standard UPSERT (UPDATE IF EXISTS or INSERT).

The process for publishing data generally looks like this:

  1. if applicable, upsert a change request and its source association
  2. upsert the build's basic parameters
  3. upsert the detailed job parameters
  4. publish artifacts

These actions can be also performed manually (without using the native webhooks) with zeus-cli (recommended) or curl.

Updating data with zeus-cli

More information (installation instructions, documentation) about zeus-cli can be found on its project's page: https://github.com/getsentry/zeus-cli

zeus-cli is a command line tool that facilitates interaction with Zeus API for actions such as updating jobs or uploading artifacts.

The following command creates a build and a job for a given git revision:

zeus job update -b $MY_BUILD_ID -j $MY_JOB_ID  --ref=$MY_REF_ID

And here's how you upload an artifact:

zeus upload -b $MY_BUILD_ID -j $MY_JOB_ID -t 'text/xml+coverage' coverage.xml

Updating data with curl

Here's an example of how you can publish job details without the native webhooks with curl from Travis:

#!/bin/bash -eu
if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then
    BUILD_LABEL="PR #${TRAVIS_PULL_REQUEST}"
else
    BUILD_LABEL=""
fi

# ensure the build exists
curl $ZEUS_HOOK_BASE/builds/$TRAVIS_BUILD_NUMBER \
    -X POST \
    -H 'Content-Type: application/json' \
    -d "{\"label\": \"${BUILD_LABEL}\", \"ref\": \"$TRAVIS_COMMIT\", \"url\": \"https://travis-ci.org/${TRAVIS_REPO_SLUG}/builds/${TRAVIS_BUILD_ID}\"}"

# upsert current job details
curl $ZEUS_HOOK_BASE/builds/$TRAVIS_BUILD_NUMBER/jobs/$TRAVIS_JOB_NUMBER \
    -X POST \
    -H 'Content-Type: application/json' \
    -d "{\"status\": \"$1\", \"result\": \"$2\", \"url\": \"https://travis-ci.org/${TRAVIS_REPO_SLUG}/jobs/${TRAVIS_JOB_ID}\", \"allow_failure\": ${TRAVIS_ALLOW_FAILURE}}"

From there you can submit artifacts using zeus-cli and its standard mechanisms.