package-indexer: add .gz, .xz and .bz2 compression for Packages

[alltilla]

Tested locally:

# apt update
Ign:1 http://127.0.0.1:8000/apt nightly InRelease
Get:2 http://127.0.0.1:8000/apt nightly Release [2497 B]                                      
Ign:3 http://127.0.0.1:8000/apt nightly Release.gpg                                           
Get:4 http://127.0.0.1:8000/apt nightly/ubuntu-jammy amd64 Packages [11.7 kB]                 
Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease                                        
Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Get:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease                              
Fetched 121 kB in 7s (17.4 kB/s)                                                              
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
10 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@alltilla-Precision-5530:/etc/apt/sources.list.d# apt search syslog-ng-core
Sorting... Done
Full Text Search... Done
syslog-ng-core/nightly 4.0.1.116.gfdb88de.dirty-snapshot+20230206T094016 amd64
  Enhanced system logging daemon (core)

$ python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [06/Feb/2023 11:35:23] code 404, message File not found
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/InRelease HTTP/1.1" 404 -
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/Release HTTP/1.1" 200 -
127.0.0.1 - - [06/Feb/2023 11:35:23] code 404, message File not found
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/Release.gpg HTTP/1.1" 404 -
127.0.0.1 - - [06/Feb/2023 11:35:23] "GET /apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.gz HTTP/1.1" 200 -

Note: The 404 comes from the repo not being signed and the InRelease and Release.gpg files are missing, which is due to the local testing.

Fixes #4309

Signed-off-by: Attila Szakacs szakacs.attila96@gmail.com

[github-actions]

No news file has been detected. Please write one, if applicable.

[bazsi]

I am not sure how large those Packages files can become, but this will read everything into
a memory based buffer and then compress it in memory before writing it out.

As long as those files are tiny, I am ok, but this will eventually bite us. :)

With all that said, this is nice and simple and will probably work for years to come, so I am ok with this as implemented.

[bazsi]

The nightly is now green, please run nightly if you merge this to check if this break anything as it gets to master.

[alltilla]

$ find -name "Packages*" | xargs ls -algh
-rw-rw-r-- 1 alltilla 61K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages
-rw-rw-r-- 1 alltilla 11K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.bz2
-rw-rw-r-- 1 alltilla 12K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.gz
-rw-rw-r-- 1 alltilla 11K febr   6 11:35 ./apt/dists/nightly/ubuntu-jammy/binary-amd64/Packages.xz

I think we can get rid of the simple Packages file, if we are afraid of memory/bandwidth usage.
E.g.: http://hu.archive.ubuntu.com/ubuntu/dists/jammy/main/binary-amd64/

Storing the Packages file is not a big deal IMO, and from my quick test with apt, it ignored the Packages file, and downloaded Packages.gz automatically, so our bandwidth is reserved. Cannot comment on other clients, though.

[alltilla]

Oh, I have completely missed your point about storing it in memory while compressing it, sorry.
Yes that can be a problem, but I presume these files do not get that large (maybe a couple of MBs). We will see it in the future.

[bazsi]

agreed