Please publish announced caching proxy

[safinaskar]

In this video https://debconf21.debconf.org/talks/22-making-use-of-snapshotdebianorg-for-fun-and-profit/ they said that they wrote caching proxy for snapshot.debian.org , which has special quirks for snapshot.debian.org . Please, publish it. As well as I understand software in this repo ( https://github.com/fepitre/debian-snapshot ) is not that caching proxy. This repo ( https://github.com/fepitre/debian-snapshot ) contains tool, which downloads all packages for given timestamp+arch+suite triple at once. I don't need this. Please, publish caching proxy mentioned in that video.

Now let me describe why I need this.

I'm creating tool for creating Debian images "from past". I cannot use snapshot.debian.org directly because of its download limits. So I use https://debian.notset.fr/snapshot/archive/debian/ . But it doesn't contain suites such as "bullseye-updates". So it seems the only way to proceed for me is to set up my own local caching proxy for snapshot.debian.org with snapshot.debian.org-specific quirks. I cannot use this repo ( https://github.com/fepitre/debian-snapshot ), because (as well as I understand) this tool downloads whole timestamp+arch+suite at once, it is not caching proxy. It seems I also cannot use general purpose caching proxy, such as squid, because it is not aware of snapshot.debian.org quirks. So now I'm currently writing my own caching proxy for snapshot.debian.org in Rust. But this would be very good if you simply publish caching proxy you already have

[safinaskar]

(Or please add secondary suites such as bullseye-updates and bullseye-security to https://debian.notset.fr )

[rclobus]

Have you read my Wiki page? https://wiki.debian.org/ReproducibleInstalls/LiveImages
I'm using aptcacher-ng, and that reduces a huge amount of bandwidth.

[safinaskar]

@rclobus , does apt-cacher-ng work with snapshot.debian.org ? I mean do you observed rejects when you connect to snapshot.debian.org via apt-cacher-ng?

[safinaskar]

P. S. You wrote "An additional line in /etc/apt-cacher-ng/acnf.conf: VfilePatternEx: /project/trace/ftp-master.debian.org$". It seems you meant acng.conf, not acnf.conf

[fepitre]

I don't see which caching proxy you are referring unless aptcache-ng. Please join IRC for such discussions. There is no actionable point with respect to this repository serving as a POC. It only refers to what content is chosen to snapshot for the instance I host.

[safinaskar]

For people that search that caching proxy, too:

Johannes Schauer Marin Rodrigues just told me in private mail, that the caching proxy actually exists. It is embedded in his software:
https://sources.debian.org/src/devscripts/2.22.2/scripts/debbisect/#L121
https://sources.debian.org/src/devscripts/2.22.2/scripts/debootsnap/#L57
https://salsa.debian.org/metasnap-team/metasnap/-/blob/8f138bbe572e645c3345906d5df3c25bf090e626/run.py#L131

[safinaskar]

I did experiment and I see that apt-cacher-ng kinda works. I was able to use it so:

• Use http://snapshot.notset.fr/archive/debian as lot as possible, resort to snapshot.debian.org when snapshot.notset.fr doesn't work, for example, for buster-updates. For example, here is sources.list:

deb http://snapshot.notset.fr/archive/debian/20221030T084935Z buster main
deb http://snapshot.debian.org/archive/debian-security/20221030T084935Z buster/updates main
deb http://snapshot.debian.org/archive/debian/20221030T084935Z buster-updates main

• Connect to snapshot.debian.org via apt-cacher-ng
• If big download fails, restart it after timeout 4^r, where r is number of retries, as said in https://debconf21.debconf.org/talks/22-making-use-of-snapshotdebianorg-for-fun-and-profit/ . For example: I=1; while ! sudo apt-get install -y BIG-PACKAGE; do I=$((I * 4)); echo sleep $I; sleep $I; done. Of course, attempts starting from 2nd will reuse files from apt-cacher-ng cache. In my case 5 attempts was enough, i. e. last sleep was sleep 256.
• Sometimes you will see errors about hash mismatch. If you see them, just try again, errors will disappear. You don't need to manually clean apt-cacher-ng cache. (But I cleaned apt's cache, i. e. /var/cache/apt/archives, I don't know whether this helped.)