releases.turnkeylinux.org down?

[m-klecka]

Hi, i'm having problems running pveam update on proxmox. It says theres a 522 error, but i can ping releases.turnkeylinux.org from my pc and from proxmox console.

Heres code from the log

2023-07-04 18:43:46 starting update 2023-07-04 18:43:46 start download http://download.proxmox.com/images/aplinfo-pve-8.dat.asc 2023-07-04 18:43:46 download finished: 200 OK 2023-07-04 18:43:46 start download http://download.proxmox.com/images/aplinfo-pve-8.dat.gz 2023-07-04 18:43:46 download finished: 200 OK 2023-07-04 18:43:46 signature verification: gpgv: Signature made Fri Jun 30 16:58:28 2023 CEST 2023-07-04 18:43:46 signature verification: gpgv: using RSA key F4E136C67CDCE41AE6DE6FC81140AF8F639E0C39 2023-07-04 18:43:46 signature verification: gpgv: Good signature from "Proxmox Bookworm Release Key <proxmox-release@proxmox.com>" 2023-07-04 18:43:46 update successful 2023-07-04 18:43:46 start download https://releases.turnkeylinux.org/pve/aplinfo.dat.asc 2023-07-04 18:44:18 download failed: 522 Unknown code 2023-07-04 18:44:18 update failed - no signature file '/var/lib/pve-manager/apl-info/pveam-releases.turnkeylinux.org.tmp.4769.asc'

Any tips?
Thanks

[Jameson42]

It looks like archive.turnkeylinux.org is down as well. I'm unable to ping it or do an apt update/install.

[markusdd]

can confirm, also my containers just bombarded me with status mails that they are unable to reach the repository:

[rtpguy82]

Confirmed here as well. I have been unable to connect to archive.turnkeylinux.org all day.

[JedMeister]

Hi all,

Deep apologies on the breakage...

It certainly does appear that the "master mirror" server (that also hosts releases.turnkeylinux.org & archive.turnkeylinux.org) is down. Unfortunately, whilst I have root access to the server, I don't have access to the AWS account that it runs under. The server is so broken that I can't even log in. So we'll need to wait for my colleague Alon to wake up and reboot it for us. He'll probably wake up in a couple of hours or 2 and hopefully he'll see my message and get straight onto it.

I would recommend that you just wait until it's back up. If it's really urgent, then you could manually work around it by using (one of) the (round robin) mirrors accessible via http://mirror.turnkeylinux.org/turnkeylinux/

https://releases.turnkeylinux.org/pve/aplinfo.dat.asc (and related files can be found in http://mirror.turnkeylinux.org/turnkeylinux/metadata/pve/ It's been a while since I manually did it, but IIRC you should be able to download the relevant files and put them in the right place on your PVE server and it should work. Having said that, it's perhaps better/easier to just manually download the desired template directly from the mirror. E.g. to download LAMP (from your PVE server):

cd /var/lib/vz/template/cache
wget http://mirror.turnkeylinux.org/turnkeylinux/images/proxmox/debian-11-turnkey-core_17.1-1_amd64.tar.gz

The hash file is also there (append .hash to the line above to download the hash file) if you want to double check and make sure it's legit (FWIW PVE does that behind the scenes by default).

The apt repo is a little more convoluted, but changing the URL in apt list files from:

http://archive.turnkeylinux.org/debian

To

http://mirror.turnkeylinux.org/turnkeylinux/apt/debian

Will workaround the issue and it should continue to work fine. Whilst it should be just as secure as connecting directly (because all the apt repo files are still signed by our package signing key, which your server has already), I wouldn't recommend leaving it like that.

I say that because occasionally a mirror may fall out of sync and that could cause issues as apt makes multiple connections to the servers it downloads from. As our mirrors are "round robin" you may connect to a different mirror each time and obviously if the mirrors have different files that will cause errors. IMO a hard failure (so you know something is definitely wrong) is better than weird unexpected issues that aren't reproducible.

Regardless, here's a one liner to update your current apt config:

sed -i.bak 's|archive.turnkeylinux.org|mirror.turnkeylinux.org/turnkeylinux/apt|g' /etc/apt/sources.list.d/*.list

To revert to the original config, rename all the .bak files back to what they were (i.e. remove the .bak suffix). E.g. in my case:

root@tkldev ~# ls -l /etc/apt/sources.list.d/
total 28
-rw-r--r-- 1 root root 189 Jun 28 03:28 debian-backports.list.disabled
-rw-r--r-- 1 root root 442 Jul  5 04:12 security.sources.list
-rw-r--r-- 1 root root 426 Jul  5 03:51 security.sources.list.bak
-rw-r--r-- 1 root root 389 Jul  5 04:12 sources.list
-rw-r--r-- 1 root root 373 Jul  5 03:51 sources.list.bak
-rw-r--r-- 1 root root 138 Jul  5 04:12 turnkey-testing.list
-rw-r--r-- 1 root root 122 Jul  5 03:51 turnkey-testing.list.bak

I'd run this:

mv /etc/apt/sources.list.d/security.sources.list.bak /etc/apt/sources.list.d/security.sources.list
mv /etc/apt/sources.list.d/sources.list.bak /etc/apt/sources.list.d/sources.list
mv /etc/apt/sources.list.d/turnkey-testing.list.bak /etc/apt/sources.list.d/turnkey-testing.list

Most of you won't need the third line (the TKL testing repo is disabled by default - but I had it enabled).

[JedMeister]

Oh also, I meant to mention that we have plans for the apt repo (and the pve files too) to make them much better and more reliable. That will allow https support as well as IPv6 access too (which is becoming a hard requirement for some these days).

We devised it last year, but we've been flat out with other higher priority items and haven't had time to circle back and implement our plan.

I'm hoping that once we have completed the v18.0 release (a few months at least) I'll get a chance to implement the new file storage setup.

[JedMeister]

It's back up. FWIW it seems that our instance was shut down by AWS because the underlying hardware was retired. I assume that they sent an email out about that and we must have missed it...

Server.ScheduledStop: Stopped due to scheduled retirement

As I said above, we do have plans to make it better, but we'll get v18.0 out the door first. Regardless it should be fairly reliable from here on in (a least as reliable as it has been - which is generally fairly good, although more outages than I would like...).

[markusdd]

no need to apologize, you guys do an outstanding job with Turnkey.
thanks for the quick resolution!

[JedMeister]

Thanks for the kind words @markusdd 😁