Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-10261: Create dispatch_verify.py CLI for uploading a verification job #10

Merged
merged 8 commits into from
Apr 28, 2017
Merged

DM-10261: Create dispatch_verify.py CLI for uploading a verification job #10

merged 8 commits into from
Apr 28, 2017

Commits on Apr 28, 2017

  1. Create SQUASH HTTP client 

    This code is adapted from https://github.com/lsst-sqre/post-qa

The idea is to provide a client library to support HTTP GET, POST, etc.
requests against the SQUASH API. This handles getting endpoints,
handling API version headers, logging, etc.

Data objects can use this API to implement data upload and download
methods. The idea is that the squash.py method itself will remain data
agnostic.

Adds `requests` as an EUPS dependency.

responses (https://github.com/getsentry/responses) is an optional
dependency to support unittests. Tests will skip is responses cannot be
imported. Eventually we shall try to ensure responses is systematically
available in the CI environment.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    be31126 View commit details
    Browse the repository at this point in the history

  2. Add Job.dispatch: proposal for uploading to SQUASH 

    Job.dispatch() uploads the job to the SQUASH database via its HTTP API.
This API doesn't exist yet, so this method should be considered a
proof-of-concept.

Metrics and specifications should be uploaded separately since they're
not changing per-job, and can be considered separate resources.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    32996b2 View commit details
    Browse the repository at this point in the history

  3. Add Manifest manifest.txt metadata info 

    The new lsst.verify.metadata sub-package will host tools for gathering
metadata from environments.

metadata/eupsmanifest.py is the first one, which parses the manifest.txt
file found in lsstsw builds.

A sample manifest.txt is included in tests/data/lsstsw/build (emulates
the directory structure of lsstsw).
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    4c87454 View commit details
    Browse the repository at this point in the history

  4. Add lsstsw repos.py interface 

    Port from https://github.com/lsst-sqre/post-qa.

LsstswRepos provides an interface to repos.yaml and git repository
metadata in lsstsw. This includes the Git origin url of packages and the
currently-checked out Git branch.

Note I tried to mock the GitPython package to properly test the
LsstswRepo.get_package_branch() method, but couldn't get it to work yet.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    de043bf View commit details
    Browse the repository at this point in the history

  5. Add jenkinsci metadata source 

    get_jenkins_env() gather job metadata from the Jenkins CI execution
environment.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    fd96a87 View commit details
    Browse the repository at this point in the history

  6. Add in-place addition and update() to set classes 

    MetricSet, SpecificationSet and MeasurementSet gain __iadd__() and
update() methods. This provides a simple API for merging sets.

Also add Job.__iadd__ to merge Jobs

This provides a convenient API for merging Jobs, which comes up when
collecting several Jobs output from individual tasks for upload to
SQUASH.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    2d81345 View commit details
    Browse the repository at this point in the history

  7. Add metric/specification package refresh 

    These are a convenience APIs for inserting Metric instances from a
MetricSet into the Measurement.metric attribute of a measurement.

Job.reload_metrics_package adds metrics to Measurement instances, as
well as augmenting Job.metrics and Job.specs.

Adding metrics to measurement instances is useful because it helps us
normalize the units of measurements when serialized.

Add Job.reload_metrics_package
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    2811edf View commit details
    Browse the repository at this point in the history

  8. Create dispatch_verify.py command 

    dispatch_verify.py is responsible for gathering Job JSON files, enriching
them with environment metadata, and either printing, writing, or HTTP
POSTing those Job documents to SQUASH.
    @jonathansick
    jonathansick committed Apr 28, 2017
    Configuration menu
    Copy the full SHA
    795ae0f View commit details
    Browse the repository at this point in the history