Clarification about reproducibility of release tarballs

[basilgello]

Dear @vbassmatt

Looking at 8aa87b1 I see an ambiguity that needs to be resolved.

On the page you write:

If you rely on stability of archives for security (for example: to ensure you don't attempt to unzip a maliciously-crafted file), we recommend using releases instead of using source downloads.

At the same time:

Source code archives are generated on request, cached for a while, and then deleted. If the same archive is requested again in the future, it'll be regenerated. It's important to understand what guarantees GitHub makes about source code archives.

So - if a distro maintainer uses release tarball as a base for reproducible build, will this tarball remain bit-by-bit identical over the years?
If the release tarballs are stored intact, the documentation should be rephrased to reflect that.

Otherwise, we will habe to fix Debian Wiki https://wiki.debian.org/Creating%20signed%20GitHub%20releases because Debian assumes the tarball is immutable after release / signing.

Best way to avoid such ambiguities would be to document which archive types are stored and which are regenerated.

Thanks!

[welcome]

Thanks for opening this issue. A GitHub docs team member should be by to give feedback soon. In the meantime, please check out the contributing guidelines.

[bk2204]

Hey,

I'm on the team that's responsible for autogenerated archives. There are two types of data that can be associated with a release.

The first is release assets, which are files that CI or a developer uploads as part of a GitHub release. Those files are stored permanently and immutably. For example, if we take the latest Git LFS release, then all of those entries under Assets, except for "Source code (zip)" and "Source code (tar.gz)" are release assets and immutable. Basically, all of them with the little package icon.

The two exceptions I mentioned above, with the zipper icon, are autogenerated archives. While in the link they're associated with a release, you can actually generate them for any commit, tag, or tree. As the text says, they're generated on demand, cached, and then, when the cache expires, deleted. (These links also appear in the tarball and zipball entries in some parts of the API.)

At the moment, we have no concrete plans to change the output of those values, but we may do so in the future with advance notice. Also, because those archives can sometimes contain an abbreviated object ID, it can sometimes happen that as more objects are added to the archive, the object ID abbreviation becomes longer, and thus the archive is no longer bit-for-bit identical. There's really no way we can avoid that happening, so that's one of the reasons we recommend against using the autogenerated archives for reproducible archives (e.g., if you're using a cryptographic hash or signature).

The example you gave in the Debian wiki is actually mostly correct. The difference is that instead of just a signature, you need to upload a tarball as well. Then you have a tarball as a release asset along with its signature, and they'll both be preserved indefinitely. Based on what you've said and my knowledge about Debian's packaging, that would be the right way to avoid any problems in the future. We do that with Git LFS specifically so Debian and Homebrew, among other distributors, have a stable source archive.

Does that help answer your question?

[basilgello]

@bk2204 thanks for the answer! So if I understand your reply correctly, the only way to get immutable source code tarball is to upload one manually by the developer? I.e, if I (as an upstream author) make a release via Github web interface, but I do not upload tar.gz or zip file as an asset, there is no way for automated systems to get the same tarball if compression settings change?

fyi @mapreri @h01ger @fuzzard

[basilgello]

The question is - 99% of upstream authors using Github make releases either via gh cli or web interface, and very very few do upload release tarballs because tarball is already there. Packagers then use these tarballs as an ultimate source of truth like shown in man 1 uscan and only special search reveals one can trust omly signed Git tags amd commits…

[mapreri]

Yes, exactly.

If I remember correctly from our experience, over the last decade it happened only once that github changed something somewhere in their tar/gzip implementations and as such the resulting file changed.

That Debian wiki page you linked is indeed… dangerously close as those files a liable to change every so often. Since it very rarely/~never happen however it normally does not matter on day-to-day operations.

If we would like to actually rely on those tarballs being immutable, however, once does need to upload them as separate artefacts.

[mapreri]

In a way, this is one of the many outcomes of the dissociating worlds of modern software developers (who only ever work on git, you can hardly find one that even knows how to run tar), and distributors, who do need a tarball to run things.

BTW, I remember talking with a bunch of enterprise people, and all of them are going out of their way to develop in-house solutions to track software tarballs and figure out a way to verify their contents. None of them even remotely pretty, and all of them clearly needed only thanks to developers having "given up" on decent distribution mechanism. So... We are not alone! 🙃

[basilgello]

@mapreri The reason I started digging up this topic is because I've got a wishlist to use tarballs instead of git mode in uscan to simplify release diff inspection by Debian sponsors/ftpmasters. But it looks that with Github, git mode + predictable mk-origtargz / pristine-tar machinery ensures better reproducibility of the tarball. This paradox makes me puzzled :/

[basilgello]

@bk2204 Have your team ever considered storing metadata similar to pristine-tar to reduce storage requirements for the tarball?

[mapreri]

I wonder…

Even us running mk-origtargz is dependent on that particular version of the day of tar and gzip/your-compressor-of-choice. I haven't run the math (nor am I going to), but I wouldn't be surprised if the "level of determinism" between github-generated tarballs and locally-generated-from-git tarballs is much different given a time span of a couple of decades.

[basilgello]

Would be interesting to do the math, though :) Or even better, standardize the pristine-tar under Reproducible Builds project :) I think I can help with that

[bk2204]

@bk2204 thanks for the answer! So if I understand your reply correctly, the only way to get immutable source code tarball is to upload one manually by the developer? I.e, if I (as an upstream author) make a release via Github web interface, but I do not upload tar.gz or zip file as an asset, there is no way for automated systems to get the same tarball if compression settings change?

Yes, that's correct. We do intentionally try not to change compression settings or other aspects of the generation process without advance notice and we do have tests for those cases to avoid accidental changes, but since we essentially use git archive under the hood and git archive can sometimes produce minor variations in some cases like I mentioned, there's no 100% guarantee of immutability.

We strongly recommend uploading a tarball or other source archive as a release asset for that reason. You can do that with git archive yourself; you don't need to the GitHub ones, and in fact Git LFS uses git archive to generate ours.