signing releases

[anarcat]

the Debian packaging process can check for upstream signatures when packaging new tarballs..

i encourage you to use OpenPGP signatures to certify the source code you ship. it's the only bit missing to ensure a complete trust chain between you as developpers and Debian end-users.

while we do minimal reviews of changes when we package new upstream versions, some things may go unnoticed if a maintainer is in a rush. having OpenPGP signatures ensures that the packages were not attacked while in transit.

thanks!

[warner]

Awesome. The pypi tarballs do include (detached .asc) signatures.. is there an extra step we can do to make these easier for debian to see or check?

[anarcat]

no. i just need to fix my watch file and import your key. :)

i do not have a direct trust path to your key, unfortunately. for now I will just trust that this is the correct key fingerprint? 5810 F125 7F8C F753 7753 895A 8633 33C2 6549 7810

by the way, the tarball from pypi differs from the one here on github, is that normal?

note that I often prefer the github tarballs as they are built reproducibly, with git: given a tag, git creates the exact same tarball every time...

[warner]

5810 F125 7F8C F753 7753 895A 8633 33C2 6549 7810 is indeed correct.

by the way, the tarball from pypi differs from the one here on github, is that normal?

I think that is normal, unfortunately. I make the tarball by running setup.py sdist locally, which I then GPG sign, and upload both to pypi (which is where most folks get it, I believe). I haven't really paid attention to what Github is doing on the release page (it's not part of my flow), but I'm guessing Github makes it's own tarballs on demand from the tag. The file contents should be identical, however the timestamps (and thus the tarball itself) will differ.

I suppose I could change the release workflow to be more like:

• push the (signed) tag to github
• download github's tarball
• compare the contents to what I get from sdist
• sign that tarball
• upload everything to pypi

It'd add a few steps, but then github's tarballs would be the same as pypi (assuming github really does normalize the timestamps reproducibly). I'll think about it.

[meejah]

What I do is use the Web interface of GitHub to upload the "original" tarball and .asc signatures. So, you still have the "github dynamic tarballs" alongside them, but at least you can download the originals. (I also upload wheels and signed wheels). Example here: https://github.com/meejah/txtorcon/releases

I also have a public release checklist (https://github.com/meejah/txtorcon/blob/master/docs/release-checklist.rst) and would be happy to learn of any deficiencies :)

[anarcat]

i prefer to have a single tarball source, which is easily reproducible.

if sdist doesn't create a reproducible tarball, it should be fixed or not used, otherwise it will create confusion... :)

[meejah]

I wonder if there's any way to turn off the github "magic tarball and zip" links....

[warner]

I just finished poking around the API, and I couldn't find any way to turn them off.

[warner]

Some more notes:

• github's tarballs set the timestamps of all files to the time of the tag.. I like it
• github appears to use git-archive to make the tarball, so the Versioneer tags in src/wormhole/_version.py are correctly populated, so the resulting executable can figure out its own version just as well as a setup.py-produced sdist
• the pypi tarball includes the .egg-info directory and PKG-INFO, which are side-effects of the sdist-building process.. I'm not sure how important/relevant they are
• the pypi tarball omits a few files that weren't added to MANIFEST.in (.appveyor.yml, .travis.yml, some docs). I'm on the fence about adding them, or leaving them out.
• the pypi tarball's setup.cfg includes an [egg_info] section, which probably doesn't matter

The 0.8.1 tarball I just grabbed from github (https://github.com/warner/magic-wormhole/archive/0.8.1.tar.gz) is 138638 bytes long and has a SHA256 of 79f1725edaa7643d140ecaf6a54d7a635e488aee9da4d876abcb766de9d54117 . Can other folks confirm that they get the same thing? It'd be nice to know if their generation process is fully deterministic.

I seem to remember that the gzip wrapper has its own timestamp somewhere, and gzip -lv shows me a recent date, but that appears to come from the mtime of the local copy, rather than being part of the contents. The gzip RFC says there's a four-byte MTIME field at offset 4-7, but those appear to be all zeros in the github tarball (the pypi tarball has 67 85 99 57 there).

[anarcat]

On 2016-12-08 01:37:55, Brian Warner wrote: Some more notes: * github's tarballs set the timestamps of all files to the time of the tag.. I like it

that is git-archive for you, iirc.

* github appears to use git-archive to make the tarball, so the Versioneer tags in `src/wormhole/_version.py` are correctly populated, so the resulting executable can figure out its own version just as well as a setup.py-produced sdist

correct.

* the pypi tarball includes the `.egg-info` directory and PKG-INFO, which are side-effects of the sdist-building process.. I'm not sure how important/relevant they are

i think those are just noise, but they are necessary for PyPI to properly populate the release fields. not sure how to deal with this.

* the pypi tarball omits a few files that weren't added to MANIFEST.in (`.appveyor.yml`, `.travis.yml`, some docs). I'm on the fence about adding them, or leaving them out. * the pypi tarball's setup.cfg includes an `[egg_info]` section, which probably doesn't matter

true. but then it does make the tarball different... :/

The 0.8.1 tarball I just grabbed from github (https://github.com/warner/magic-wormhole/archive/0.8.1.tar.gz) is 138638 bytes long and has a SHA256 of 79f1725edaa7643d140ecaf6a54d7a635e488aee9da4d876abcb766de9d54117 . Can other folks confirm that they get the same thing? It'd be nice to know if their generation process is fully deterministic.

same here: 79f1725edaa7643d140ecaf6a54d7a635e488aee9da4d876abcb766de9d54117 magic-wormhole-0.8.1.tar.gz

I seem to remember that the gzip wrapper has its own timestamp somewhere, and `gzip -lv` shows me a recent date, but that appears to come from the mtime of the local copy, rather than being part of the contents. The gzip RFC says there's a four-byte MTIME field at offset 4-7, but those appear to be all zeros in the github tarball (the pypi tarball has `67 85 99 57` there).

In my experience, the tarballs from github are fully reproducible: you can use git-archive locally to create the archive yourself and will end up with the same result. you do need to use the gzip -n flag however, which github also uses, otherwise it indeed embeds a timestamp. a.

…

-- The greatest tragedy in mankind's entire history may be the hijacking of morality by religion. - Arthur C. Clarke

[warner]

Ok, I'm gonna make an executive decision, and declare that the PyPI tarballs are the official ones, and ignore the github-generated ones (at least for now.. I'm happy to revisit this later).

• the main install instructions (pip install magic-wormhole) depends upon the PyPI-hosted tarballs, and doesn't know about the github ones
• so we can't turn off PyPI's tarballs, and they must be sdists (and include PKG-INFO), not from git-archive
• there are no published install instructions that start with the github tarballs

It's a bummer to not have the easy traceability step from git SHA1 to git-archive tarball (and instead having a one-to-many SHA1->tarball conversion function, with a single distinguished/signed application of this function being used to make the official tarballs). But PyPI needs metadata that isn't commited to git.

So yeah, please update the watch file to point at PyPI, and verify the signatures that sit next to the tarballs there. For extra credit, we could automate something to compare the tarball's contents (but not timestamps or EGG-INFO or the missing extra files) against the github tarball, and compare the github tarball against a checkout of the github SHA1.

Maybe some future changes to Python packaging will make this step better :).

[anarcat]

the problem this causes on the debian side is that we can't regenerate the tarball from the git repository anymore. i understand your motivations as an upstream, and they make sense to me, but an alternative could be to sign both tarballs, even though that means more work for you.

obviously, the proper fix would be for python packaging to not require sdists and somehow have out of band metadata.

FWIW, here is the error i get when trying to build the debian package now:

dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building magic-wormhole using existing ./magic-wormhole_0.8.2.orig.tar.gz
dpkg-source: warning: ignoring deletion of file .travis.yml, use --include-removal to override
dpkg-source: warning: ignoring deletion of file docs/events.dot, use --include-removal to override
dpkg-source: warning: file magic-wormhole-0.8.2/src/magic_wormhole.egg-info/SOURCES.txt has no final newline (either original or modified version)
dpkg-source: info: local changes detected, the modified files are:
 magic-wormhole-0.8.2/PKG-INFO
 magic-wormhole-0.8.2/setup.cfg
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/PKG-INFO
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/SOURCES.txt
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/dependency_links.txt
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/entry_points.txt
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/requires.txt
 magic-wormhole-0.8.2/src/magic_wormhole.egg-info/top_level.txt
 magic-wormhole-0.8.2/src/wormhole/_version.py
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/magic-wormhole_0.8.2-1.diff.dDLFh2
dpkg-source: info: you can integrate the local changes with dpkg-source --commit
dpkg-buildpackage: error: dpkg-source -b magic-wormhole-0.8.2 gave error exit status 2

for now i have resorted to importing the tarball in a seperate branch of the git history, but it's a little clunky.

i totally understand, however, your rationale and thank you very much for taking time to sign the release. this was, after all, the request here and the debian side of things shouldn't be of your concern. :)

cheers!

[meejah]

@anarcat have you looked at how the packaging for python-txtorcon works? My release process there definitely does not sign the github tarballs -- I essentially ignore them. "irl" is doing packaging now, originally "Lunar^" did it.

[anarcat]

i have not looked at that process in detail... for me, it is simpler to use PyPI as a binary distribution mechanism (with "wheel" files) and git as the source distribution. therefore, i don't use sdist at all. i just did a release of monkeysign using this release process. notice how i use bdist_wheel instead of sdist. as far as i can tell, pip is happy to use those to install monkeysign directly. see also the monkeysign pypi page and install instructions.

[meejah]

I don't see any signatures at all on the latest release of monkeysign (on PyPI).

[anarcat]

that is correct: binary releases are not signed (and I don't think pip would check that anyways). git release tags are OpenPGP-signed, however, and so are Debian uploads, which include a source tarball.

I should really include documentation on how to verify those signatures, however... But basically, this will verify the monkeysign source code reliably:

dget http://httpredir.debian.org/debian/pool/main/m/monkeysign/monkeysign_2.2.2.dsc

The .tar.gz file extract should match what is generated by git archive | gzip -nc as well.