Skip to content

Commit

Permalink
Merge pull request mozilla-releng#320 from escapewindow/remove-gpg
Browse files Browse the repository at this point in the history
bug 1534506 - remove gpg support from chain of trust
  • Loading branch information
escapewindow committed Mar 27, 2019
2 parents 646d478 + 3e7acb7 commit bc321cc
Show file tree
Hide file tree
Showing 74 changed files with 388 additions and 4,101 deletions.
29 changes: 29 additions & 0 deletions CHANGELOG.md
Expand Up @@ -2,6 +2,35 @@
All notable changes to this project will be documented in this file.
This project adheres to [Semantic Versioning](http://semver.org/).

## [23.0.0] - 2019-03-27
### Added
- added `CODE_OF_CONDUCT.md`.
- `verify_cot` now has a `--verify-sigs` option to test level 3 chains of trust with signature verification on.
- added a `verify_ed25519_signature` endpoint helper script.

### Changed
- Updated documentation to reflect the new ed25519-only chain of trust world.
- `docker/run.sh` no longer points `/dev/random` to `/dev/urandom`, and no longer has hacks to install an old version of gpg.
- `public/chain-of-trust.json` is now a mandatory artifact in cot verification. `public/chain-of-trust.json.sig` is mandatory if signature verification is on. `public/chainOfTrust.json.asc` is no longer used.
- similarly, `public/chainOfTrust.json.asc` is no longer generated or uploaded by scriptworker.
- `add_enumerable_item_to_dict` now uses `setdefault` instead of `try/except`.

### Fixed
- added missing modules to the source documentation.
- restored missing test branch coverage.
- `get_all_artifacts_per_task_id` now returns a sorted, unique list of artifacts, preventing duplicate concurrent downloads of the same file.
- `test_verify_production_cot` now tests win64 repackage-signing instead of linux64 repackage-signing because linux64 stopped running repackage-signing. We also test an esr60 index.

### Removed
- removed gpg support from chain of trust verification.
- removed `scriptworker.gpg` module and associated tests.
- removed the `defusedxml`, `pexpect`, and `python-gnupg` dependencies.
- removed the `create_gpg_keys.py` and `gpg_helper.sh` helper scripts.
- removed gpg-specific config.
- removed `ScriptWorkerGPGException`
- removed the `rebuild_gpg_homedirs` endpoint.
- removed the `check_pubkeys.py` and `gen1000keys.py` test scripts.

## [22.1.0] - 2019-03-19
### Added
- `event.repository.full_name` and `event.pull_request.base.repo.full_name` on `cot_verify` (for GitHub repos)
Expand Down
23 changes: 6 additions & 17 deletions README.rst
@@ -1,6 +1,6 @@
===============================
===================
Scriptworker Readme
===============================
===================

.. image:: https://travis-ci.org/mozilla-releng/scriptworker.svg?branch=master
:target: https://travis-ci.org/mozilla-releng/scriptworker
Expand Down Expand Up @@ -34,9 +34,11 @@ Credentials can live in ``./scriptworker.yaml``, ``./secrets.json``, ``~/.script

* Launch: ``scriptworker [config_path]``

-------
.. _build the docker image:

-----------------------
Building a docker image
-------
-----------------------

First, create a `secrets.json`. For integration testing, you'll need to define the `integration_credentials`; to do any other authenticated work, you'll need to define `credentials`.

Expand All @@ -49,8 +51,6 @@ Then::
Testing
-------

Note: GPG tests require gpg 2.0.x!

Without integration tests, install tox, then

``NO_TESTS_OVER_WIRE=1 tox -e py36``
Expand Down Expand Up @@ -81,14 +81,3 @@ then
It's also possible to test in docker. First, `build the docker image`_, making sure to add integration credentials to `secrets.json`. Then::

docker run -i scriptworker-test-$PY_DOT_VERSION

GPG Homedir testing
^^^^^^^^^^^^^^^^^^^

Sometimes it's nice to be able to test things like ``rebuild_gpg_homedirs``. To do so, `build the docker image`_, then::

docker run -i scriptworker-test-$PY_DOT_VERSION gpg
# in the docker shell,
rebuild_gpg_homedirs gnupg.yaml

.. _build the docker image: #building-a-docker-image
19 changes: 2 additions & 17 deletions docker/Dockerfile.test
@@ -1,39 +1,24 @@
ARG PY_DOT_VERSION
FROM python:${PY_DOT_VERSION:-3.6}

# gnupg 2.0.x
RUN echo "deb http://http.us.debian.org/debian/ oldstable non-free contrib main" >> /etc/apt/sources.list

RUN apt-get update && apt-get install -y --allow-downgrades \
git \
gnupg2=2.0.26-6+deb8u2 \
gnupg-agent=2.0.26-6+deb8u2 \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*

RUN mkdir -p /builds/test
RUN mkdir -p /builds/scriptworker

RUN ln -s /usr/bin/gpg2 /usr/bin/gpg
COPY ./requirements/ /builds/scriptworker/requirements
COPY setup.py /builds/scriptworker/
COPY setup.cfg /builds/scriptworker/
COPY scriptworker/ /builds/scriptworker/scriptworker
COPY .git /builds/scriptworker/.git
COPY .coveragerc /builds/scriptworker/
COPY tox.ini /builds/scriptworker/
COPY MANIFEST.in /builds/scriptworker/
COPY version.json /builds/scriptworker/

# allow for rebuild_gpg_homedirs testing
COPY docker/gnupg.yaml /builds/test/gnupg.yaml
COPY scriptworker/test/data/gpg/keys/scriptworker@example.com.pub /builds/test/my_pubkey.asc
COPY scriptworker/test/data/gpg/keys/scriptworker@example.com.sec /builds/test/my_privkey.asc
COPY secrets.json /builds/test
COPY docker/run.sh /builds/test
RUN bash -c "echo $PY_DOT_VERSION | sed -e 's/\.//' > /builds/test/pyver"
# get trusted gpg pubkeys
RUN git clone https://github.com/mozilla-releng/cot-gpg-keys.git /builds/key_repo
RUN git clone https://github.com/mozilla-releng/build-puppet.git /builds/puppet

# test-specific installs
WORKDIR /builds/scriptworker/
Expand Down
19 changes: 0 additions & 19 deletions docker/gnupg.yaml

This file was deleted.

5 changes: 0 additions & 5 deletions docker/run.sh
Expand Up @@ -16,15 +16,10 @@ test_env() {

if [ $1 == "unittest" ]; then
test_env
rm /dev/random && ln -s /dev/urandom /dev/random # speed up gpg tests
PYVER=`cat /builds/test/pyver`
tox -e py$PYVER
rc=$?
exit $rc
elif [ $1 == "gpg" ] ; then
test_env
cp /builds/test/gnupg.yaml .
exec /bin/bash -il
else
echo "unknown mode: $1"
exit 1
Expand Down
112 changes: 112 additions & 0 deletions docs/_static/chain-of-trust.json
@@ -0,0 +1,112 @@
{
"artifacts": {
"public/build/setup.exe": {
"sha256": "5aaee8c1950f33ecbad672395831462d30d7aff80f2fdf7c6860718005aa6789"
},
"public/build/target.zip": {
"sha256": "71eb5d7008dc790dbf003ba45811d0071c86a7dd1249d0ada68fe9f0531497ed"
},
"public/logs/chain_of_trust.log": {
"sha256": "6613dca735b7f6c8166d42a608d0cbfe4ff46882a07d9e94a6a77f537072020b"
},
"public/logs/live_backing.log": {
"sha256": "c159a5ebc188633941046afc83e62980b144296de465731337b8f7ea33e6d23a"
},
"public/logs/precomplete.diff": {
"sha256": "3ae983b00741c62364ac756c3e2607742ede423bb6470fdc0298e3abf04b352a"
}
},
"chainOfTrustVersion": 1,
"environment": {},
"runId": 0,
"task": {
"created": "2019-03-09T22:02:23.215Z",
"deadline": "2019-03-10T22:02:23.215Z",
"dependencies": [
"EHZBfHRZRoiRf3yVeh_Q8g"
],
"expires": "2020-03-08T22:02:23.215Z",
"extra": {
"index": {
"rank": 1552168399
},
"parent": "bu-mLf3PR3m716VoXgYX1w",
"treeherder": {
"collection": {
"opt": true
},
"jobKind": "build",
"machine": {
"platform": "windows2012-64"
},
"symbol": "Ns",
"tier": 1
},
"treeherder-platform": "windows2012-64/opt"
},
"metadata": {
"description": "Initial Signing for locale 'en-US' for build 'win64-nightly/opt' ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=30385b68bea1a7f52c109b28a67f0ea611d88534))",
"name": "build-signing-win64-nightly/opt",
"owner": "cron@noreply.mozilla.org",
"source": "https://hg.mozilla.org/mozilla-central/file/30385b68bea1a7f52c109b28a67f0ea611d88534/taskcluster/ci/build-signing"
},
"payload": {
"maxRunTime": 3600,
"upstreamArtifacts": [
{
"formats": [
"sha2signcode"
],
"paths": [
"public/build/setup.exe"
],
"taskId": "EHZBfHRZRoiRf3yVeh_Q8g",
"taskType": "build"
},
{
"formats": [
"sha2signcode",
"widevine"
],
"paths": [
"public/build/target.zip"
],
"taskId": "EHZBfHRZRoiRf3yVeh_Q8g",
"taskType": "build"
}
]
},
"priority": "medium",
"provisionerId": "scriptworker-prov-v1",
"requires": "all-completed",
"retries": 5,
"routes": [
"index.gecko.v2.mozilla-central.signed-nightly.nightly.latest.firefox.win64-opt",
"index.gecko.v2.mozilla-central.signed-nightly.nightly.2019.03.09.revision.30385b68bea1a7f52c109b28a67f0ea611d88534.firefox.win64-opt",
"index.gecko.v2.mozilla-central.signed-nightly.nightly.2019.03.09.latest.firefox.win64-opt",
"index.gecko.v2.mozilla-central.signed-nightly.nightly.revision.30385b68bea1a7f52c109b28a67f0ea611d88534.firefox.win64-opt",
"index.gecko.v2.mozilla-central.signed-nightly.revision.30385b68bea1a7f52c109b28a67f0ea611d88534.firefox-l10n.win64-opt.en-US",
"index.gecko.v2.mozilla-central.signed-nightly.pushdate.2019.03.09.20190309215319.firefox-l10n.win64-opt.en-US",
"index.gecko.v2.mozilla-central.signed-nightly.pushlog-id.35673.firefox-l10n.win64-opt.en-US",
"index.gecko.v2.mozilla-central.signed-nightly.latest.firefox-l10n.win64-opt.en-US",
"tc-treeherder.v2.mozilla-central.30385b68bea1a7f52c109b28a67f0ea611d88534.35673"
],
"schedulerId": "gecko-level-3",
"scopes": [
"project:releng:signing:cert:nightly-signing",
"project:releng:signing:format:sha2signcode",
"project:releng:signing:format:widevine"
],
"tags": {
"createdForUser": "cron@noreply.mozilla.org",
"kind": "build-signing",
"label": "build-signing-win64-nightly/opt"
},
"taskGroupId": "bu-mLf3PR3m716VoXgYX1w",
"workerType": "signing-linux-v1"
},
"taskId": "Qt3GKapxQmWDJId8m5o8Yg",
"workerGroup": "signing-linux-v1",
"workerId": "signing-linux-12",
"workerType": "signing-linux-v1"
}

0 comments on commit bc321cc

Please sign in to comment.