allow fetching obsolete artifacts from snapshot.debian.org

[pulpbot]

Author: evgeni (evgeni)

Redmine Issue: 4866, https://pulp.plan.io/issues/4866

---

Debian removes old versions of packages that have been updated from their mirrors. Because of that using lazy sync against official Debian mirrors might end up with incomplete repos when Pulp syncs metadata on day 1, on day 2 a package gets updated/replaced and on day 3 pulp tries to actually get that package.

Debian offers a snapshot service at https://snapshot.debian.org/, where you can get almost every file ever seen in the Debian archive by accessing https://snapshot.debian.org/file/HASH (where HASH is currently the SHA1 hash of the file). The API docs of snapshot can be found at https://salsa.debian.org/snapshot-team/snapshot/raw/master/API

The idea would be to allow a secondary source in pulp_deb that is tried when the primary mirror replied with a 404.

This is probably also useful for other backends, not just Debian.

Snapshot currently only supports sha1 hashes, but the authors promised to accept patches for sha256 if anyone would write one (the source of the service is in the same repo as the api doc linked above).

[pulpbot]

From: @mdellweg (mdellweg)
Date: 2019-05-22T15:12:17Z

---

As we save all the hashes, SHA1 should work.

[pulpbot]

From: @quba42 (quba42)
Date: 2020-09-02T08:51:03Z

---

Can somebody explain to me how exactly lazy sync works.

In particular, how exactly is the repository associated with the relevant remote? (If I am only actually downloading packages when they are finally needed, then I guess I need to find a path of associations from a finished distribution back to the remote?)

Would the "secondary/fallback" content source be stored in a remote or elsewhere?

[stale]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

[stale]

This issue is no longer marked for closure.

[quba42]

I have tested this as follows:

$ wget ftp.de.debian.org/debian/pool/main/v/vim/vim_9.0.1378-2_amd64.deb
$ sha1sum vim_9.0.1378-2_amd64.deb
6349f5525cb7bfaa2e5c623d35573469385eb56f  vim_9.0.1378-2_amd64.deb
$ wget https://snapshot.debian.org/file/6349f5525cb7bfaa2e5c623d35573469385eb56f

It looks like snapshot files are only available under their SHA-1 hash, so this feature would require sha1 to be enabled on the Pulp instance.

This service could also be used to re-fetch broken artefacts, or else in combination with on_demand streamed when URL's are no longer available, as suggested in this ticket.