Verify in-toto's supply chain with in-toto

[lukpueh]

Description of issue or feature request:

As per our roadmap we

plan on documenting our release process more thoroughly. For this, we also intend to add in-toto metadata to all of our releases to ensure that all the practices outlined in the document were followed. This document will include instructions on how to verify the provided in-toto metadata as you update in-toto.

Current behavior:
No publicly available in-toto metadata and no instructions to verify in-toto with in-toto.

Expected behavior:
Generate metadata and provide instructions how to use it to verify in-toto with in-toto.

[lukpueh]

I did prepare such metadata and also a corresponding document for our initial in-toto pre-release 0.1.1.

I'm pasting it below with little redaction. It is the goal of this ticket to do this, taking into account any changes to in-toto's verification routine, and new features such as parameter substitution and gpg support.

---

Overview
This document describes how to use in-toto to verify the software supply chains for the PyPi source and binary distribution of in-toto v0.1.1.

The supply chains are laid out in "in_toto_sdist.layout" and "in_toto_bdist_wheel.layout" respectively, and consist of three steps each:

1. "checkout-tag" -- Records files from git repo at tag 0.1.1 as products
2. "run-tox" -- Runs tests using products from (1) as materials
3. 1. "create-sdist" -- Creates source distribution (.tar.gz) using products from (1) as materials
   2. "create-bdist_wheel" -- Creates binary distribution (.whl) using products from (1) as materials

Both supply chains use the same link files for step (1) and (2) but different link files for (3.a) and (3.b) respectively.

Both supply chains define a "dummy-inspection" whose sole purpose is a match rule, that matches the inspection's material, i.e. the downloaded distribution, with the product of the last step (3).

You can use the following snippet for verification:

Verify in-toto with in-toto (bootstrapping/chicken-egg situation)

# Create a virtualenvironment, you can use these docs:
# https://virtualenvwrapper.readthedocs.io/en/latest/
mkvirtualenv verify-in-toto

# ... and install in-toto
pip install in-toto==0.1.1

# Create an empty directory and change into it
mkdir verifiy_in_toto && cd verifiy_in_toto

# Copy and extract in-toto metadata
wget https://github.com/in-toto/in-toto/files/3072515/in_toto.0.1.1.in_toto.tar.gz
tar xzf in_toto.0.1.1.in_toto.tar.gz

# Download final product target files
wget https://pypi.python.org/packages/2d/50/520c02bdfe6ea46c0cffd0b83684ff3e77785cd7aa1577080913dea3b598/in-toto-0.1.1.tar.gz#md5=43fc9122029cad6a503860d8ff55329b
wget https://pypi.python.org/packages/1a/ad/5a9759213850eb3e544ec83c00ca2f79aa00009430a377ea569b1bf430ff/in_toto-0.1.1-py2-none-any.whl#md5=b078fe50513eb3601387279363270ad5

# Verify source distribution
in-toto-verify -k owner.pub -l in_toto_sdist.layout

# Verify binary distribution
in-toto-verify -k owner.pub -l in_toto_bdist_wheel.layout

Enhancements/Todos

• Use different keys for layout and links (plus thresholds). Right now there is one key for everything, but we should have a separate owner key and probably several functionary keys.
• More steps? E.g. additional testing steps (code coverage, linting).
• More inspections? We could add inspections to verify the git tag's signature or to verify byproducts (standard streams, return value) of the testing steps.

Caveats/Notes/Discussion

• To which steps should we add a threshold and what threshold?

• Dummy inspection vs. terminating step? Currently the only way to actually link (match) the final product's target files with the products of the last step is by using a dummy inspection. Maybe we should reconsider explicitly defined terminating steps?

• I re-discovered a questionable behavior of our artifact rules documented in in-toto@0.1.1/in_toto/verifylib.py#L452-L461. You can reproduce this by running above verification without the final product target files (.tar.gz, .whl) present in the same directory, which still passes, which seems odd. Maybe we need additional rule tokens to distinguish between patterns and names?

• What do you think about having different layouts for different distributions? Btw. GitHub also provides a slightly different tarball and zip archive, for which we'd need two additional layouts.

• Where should we publish the metadata? GitHub's release page might be a good place.

• Where should we publish the project owner key(s)? A different place than where we publish the metadata I'd say. Maybe on our website?

• If we wanted to run any other inspections we'd have to instruct the verifier to download and install additional scripts. Maybe we should re-consider adding some base inspections (e.g. for byproducts) to in-toto's core code? Integrate/require inspection commands #109

• Do we want something like a blog post or tutorial that covers this?

[trishankatdatadog]

I recommend using TUF to bootstrap the whole process

[lukpueh]

To anyone working on this issue these days, I suggest to take a close look at theupdateframework/python-tuf#2000.

It provides a full in-toto integration (layout, attestation generation, verification) for a GitHub Action -based release pipeline of a different Python project (python-tuf), and is based on the ideas sketched out above.

cc @jkjell