mk-ca-bundle.pl: make generated timestamps deterministic

[vszakats]

With default invocation, make generated file timestamps deterministic
by looking up (via the GitHub API) the last commit that modified
certdata.txt, along with its commit timestamp.

Also:

• show the URL used to download certdata.txt from.
• make ca-bundle.crt timestamp match certdata.txt's.

---

• fail if the stable timestamp could not be determined, and therefore the output is not reproducible. [another time]
• maybe just omit the timestamp from the comment when using -r? [NO, but also skip adding -r for now]
• make the timestamp of the output the same as the input's?

[bagder]

It struck me that it could be an idea to change the default behavior to first check which ref is the latest in the particular tree to check, and then get that ref.

That would make us always get certs by ref and thus also get a stable date.

Does it make sense?

[vszakats]

It'd be a useful feature to detect upstream updates. Though for a deterministic
build, it's still useful (or rather: necessary) to pick a specific hash that's pinned
to the build.

I'm not sure how to detect the latest commit for a specific file in a specific
branch though. I guess there should be an API for it.

[bagder]

Though for a deterministic build, it's still useful (or rather: necessary) to pick a specific hash that's pinned
to the build.

Absolutely, and I'm not arguing about that. I was just thinking that by switching to always getting a ref, we would get a date for all fetches. The difference would only be which ref we get.

I'll take a look as well.

[bagder]

I'll take a look as well.

Meh, I can't figure it out but the API docs seems to imply it should be possible.

[vszakats]

This seems to be working at first sight:

curl 'https://api.github.com/repos/mozilla-firefox/firefox/commits?path=/security/nss/lib/ckfw/builtins/certdata.txt&sha=release'

[bagder]

So with jq the last commit to the certdata file in the release branch:

curl 'https://api.github.com/repos/mozilla-firefox/firefox/commits?path=/security/nss/lib/ckfw/builtins/certdata.txt&sha=release' | jq '.[0].sha'

[bagder]

I learned jq has a "first" builtin which makes it even simpler looking:

curl 'https://api.github.com/repos/mozilla-firefox/firefox/commits?path=/security/nss/lib/ckfw/builtins/certdata.txt&sha=release' | jq first.sha

[vszakats]

I'm trying to figure out what would be a nice way to integrate it into c4w,
with provisions for curl-container. certdata date/timestamp is the source
of pain. It either needs a GitHub-specific API all to figure it out, or a way
to omit it or set it manually via an option in mk-ca-bundle.pl.

psl is similar, but we can avoid the GH API call by downloading the
commit-specific source tarball, which has the correct timestamp in it.
Works with Codeberg/Gitea/GitLab too. Not viable for certdata, because
the tarball is 1GB.

Version detection can be done with that one curl call, and this already
works in c4w for psl. Also for certdata extended with the path/ref query
options.

[vszakats]

I ended up solving it within c4w, by reusing the existing bump/download
logic and adding the timestamp retrieval for certdata.txt. Then call
mk-ca-bundle.pl -n, which sets the header timestamp based on the
input certdata.txt file:
curl/curl-for-win@d23f024

For curl-container it'd still be useful to implement some of this within
mk-ca-bundle.pl. Details depend on how it would be best used there.

In c4w, the slight downside is departing from the curl website, which is
a widely used reference source for the CA bundle. Upside is that it makes
the dependency chain shorter and may potentially make updates quicker.

The direct way also has the limitation that it lacks support for verifying
the payload hash. This is because the payload hash is re-used as the
commit hash to address the Git content. This should be safe, but
perhaps keeping a hash would be useful here too, for good measure? [→ IMPLEMENTED]

[vszakats]

Actually another benefit of implementing some of this logic within
the script, is that it would make its output more deterministic.
Meaning, everyone running the same revision of this script with
the Mozilla repo being also at the same revision would end up with
a bit-by-bit identical ca-bundle.crt output. (assuming using the
same set of script options (or none) of course.)

[bagder]

everyone running the same revision of this script with
the Mozilla repo being also at the same revision would end up with
a bit-by-bit identical ca-bundle.crt output

I think it is also nicer to get the actual date of the commit/remote file into the generated ca-bundle.crt rather than the time of when the script ran.

[vszakats]

Ended up keeping it simple by just making timestamps
stable by retrieving the last certdata.txt commit and its
timestamp.

[bagder]

@vszakats a downside with this approach has turned up: the date in the mozilla file is potentially a long time from the current date, and we no longer get the download time stamp in there.

The caextract web page now says "This bundle was generated at Feb 11 2026" for the certificate that was new in the bundle on March 19!

I've updated it to say "This bundle was updated by Mozilla at ..." but still. Users on the caextract page did not get this bundle until March 19. I think we should show the latter date, not the date from one month before it was accessible...

[vszakats]

I saw that, yes. We can just revert this (curl-for-win doesn't rely on this), or move it behind an option?

I was surprised by such time difference, given that the script looks for the commit date. Does it mean Mozilla pushes to the public repo not in real-time (or close to), but only once a month / occasionally? (But even a small diff between publish and detection by curl.se may cause a different date in the archived bundle filename and the reproducible date within it)

edit: the only line to keep unconditionally is utime() at the end.