archzfs-kernels repo is moving

[endreszabo]

First of all, I'm sorry for posting this request here, but formerly the SSL cert expiration of my archzfs kernels repo ended up here before. :)

Anyways, I'd like to move my semi-official archzfs-kernels repo off of my end.re domain of my blog as I would like to move the blog to github pages.

People sync my repo 100-160k times a month, so apparently, this move affects a lot of people.

My idea is a subdomain delegation of kernels.archzfs.com to my nameservers so that I can acquire a TLS cert for it using the Let's Encrypt dns-01 challenge validation. (Not that TLS would make it more secure as packages are officially signed.)

FAQ

Why don't you put the packages up on the GitHub pages as well?

While GH Pages has an allowance of 1 TB of storage, the kernel packages account for 745MB of space usage which is not great to host in a git repo anyway. Moreover, since yesterday, I started to answer a HTTP 301 redirection for the package requests to the archive.archlinux.org so that packages will be downloaded from there instead. It might be faster for you this way. GitHub Pages does not support this kind of redirection matching at all:

    location ~ ^/archzfs-kernels/(?<fn>(?<pn>(?<fl>.).+?)-[0-9].+pkg\.tar.+)$ {
            rewrite ^(.*) https://archive.archlinux.org/packages/$fl/$pn/$fn permanent;
    }

Will you share with us the script that generates the repo?

It is right there, executed in place every day at 00:00 UTC.

Will there be any migration period to change pacman.conf?

Sure. After that period I'll answer a 301 (permanent redirect) and later 410 (content gone) for a while on the original URL.

What is needed to be set in pacman.conf for the new location of the repo?

If agreed to use kernels.archzfs.com, then:

[kernels.archzfs.com]
Server = http://$repo/

If not, I'll make use of one of my spare domains.

Maybe it's safe to tag @minextu here as he seemed to be dealing with archzfs related web issues so far.

[minextu]

Yeah I think having kernels.archzfs.com for this would be a great idea.
I can setup a NS record on that domain, just let me know where to point it to.

[endreszabo]

I can setup a NS record on that domain, just let me know where to point it to.

for NS records:

kernels.archzfs.com.		IN	NS	iedub0.kernels.archzfs.com.
kernels.archzfs.com.		IN	NS	hubud0.kernels.archzfs.com.

and since they are in-zone, the glue records:

iedub0.kernels.archzfs.com.	IN	A	54.77.48.44
iedub0.kernels.archzfs.com.	IN	AAAA	2a05:d018:563:7900::25
hubud0.kernels.archzfs.com.	IN	A	88.151.99.71
hubud0.kernels.archzfs.com.	IN	AAAA	2a01:270:9047::1

[minextu]

I had some problems setting up glue records on my registrar. So I used iedub0.archzfs.com instead. Can you work with that?

[endreszabo]

Thanks, works just fine. While we are at it, one of their nameservers called maceio.porkbun.com is not responding. :/

[endreszabo]

Ok, I'm done setting up the web server for that.

PUBLIC ANNOUNCEMENT:

In order to use the repository from the new location simply change your previous archzfs-kernels pacman.conf entry to this one:

[archzfs-kernels]
Server = http://kernels.archzfs.com/$repo/$arch

Or this one if you really want to be a minimalist:

[kernels.archzfs.com]
Server = http://$repo/

[endreszabo]

Fixed a bug that caused kernel from the 4.x series added to the repo instead of the now-current 5.x. It came from the zfs-archiso-linux dependency.

[heinrich5991]

http://kernels.archzfs.com/ redirects to a 404 page.

[ivdimitro]

At least since 02.12 neither the new nor the old repo are working.

[endreszabo]

You're right. My script failed to build the repo as newly introduced packages have conflicting dependencies .. that can't be referenced in a single repo.

So either I'll have to blacklist some packages or split the repo for each one of them.

[endreszabo]

So either I'll have to blacklist some packages or split the repo for each one of them.

For the record before this issue I already had to blacklist processing of the zfs-archiso package as it depends on an older linux package than the zfs-linux itself. And, as we know, a single repository can't hold multiple versions of the same package.

I think the safe and future proof way is to split up the repo to one unique repos, one per archzfs package.

[endreszabo]

http://kernels.archzfs.com/ redirects to a 404 page.

Ah I missed this comment here. :/ This was due to a change to my blog structure which altered the URL of the actual post. Fixed.

[endreszabo]

Ok, I finally got time to modify my script.

The repository has been split up into many repositories each referencing different kernel packages supported by archzfs. From now on please use the kernels.archzfs.com repository as below:

For example, if you use the linux-lts kernel package, you should specify:

[zfs-linux-lts]
Server = http://kernels.archzfs.com/$repo/

or...

[kernels.archzfs.com]
Server = http://$repo/zfs-linux-lts/

...whichever you like

[grawlinson]

Thanks for providing all this support, @endreszabo! 👍

The unofficial user repositories page will need to be updated as well.

[endreszabo]

The unofficial user repositories page will need to be updated as well.

Indeed, thanks.

[vterron]

I'm encountering the following error after switching to kernels.archzfs.com:

# pacman -Suy
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
error: failed retrieving file 'kernels.archzfs.com.db' from kernels.archzfs.com : The requested URL returned error: 404
error: failed to update kernels.archzfs.com (failed to retrieve some files)
error: failed to synchronize all databases

with:

$ tail -n2 /etc/pacman.conf
[kernels.archzfs.com]
Server = http://$repo/zfs-linux

[endreszabo]

[kernels.archzfs.com]
Server = http://$repo/zfs-linux

Thanks for your report @vterron. Upon this today I learned that pacman wants to retrieve the db filename matching the section name and as such my naive proposal will not work (without symlinking, or nginx-rewrites etc) . In your case what surely works is this:

[zfs-linux]
Server = http://kernels.archzfs.com/$repo/

[vterron]

That indeed works. Thanks @endreszabo.

[bernhard-da]

hi @endreszabo could it be that kernels.archzfs.com is down? i have

[zfs-linux]
Server = http://kernels.archzfs.com/$repo/

in my pacman.conf and get

Failed to connect to kernels.archzfs.com port 80

[endreszabo]

hi @endreszabo could it be that kernels.archzfs.com is down? i have
Failed to connect to kernels.archzfs.com port 80

Thanks for noticing. Obviously, I don't have any monitoring for my webservers. 🤔

Home alone with toddlers does not make life easy. :) And when it comes to Archlinux upgrade that silently restarts haproxy after upgrade also leads to uncertainty. Turns out that the now-current version of haproxy does not support the debug clause in its logging configuration and fails to start with it.

haproxy[803996]: [ALERT] 353/032003 (803996) : parsing [/etc/haproxy/haproxy.cfg:7] : unknown keyword 'debug' in 'global' section

Thanks, @bernhard-da, fixed.

[bernhard-da]

@endreszabo thx a lot for the quick fix! i always use (a selfhosted) instance of healthcheck.io for this and run cronjob every minute against the server i want to upgrade. very simple and sometimes life-saving. thx again a lot!

[endreszabo]

Annoyingly lot of people have opened 'issue tickets' on various 'non-official' platforms (which also led the removal of my repository from the archwiki unofficial user repositories page) so I decided to create a github repo for the scripts that generate the pacman repositories on kernels.archzfs.com. So, from now on--except for my blogpost's comment section--I will not answer to issues posted anywhere else than the issue tracker or discussion section of the kernels.archzfs.com scripts repository.

BTW, I fixed the package signature problem. It was due to a bug in the arch-official repo-add script.