Create CT: Internal Server Error: SSH public key validation error

[AndreasSchwalb]

When I try to create an LXC via Ansible I get the response "Error: 500 Internal Server Error: SSH public key validation error".
I also send this request via curl and get exact the same result. If I remove the public key and just enter a password, everything works fine.
I send the same request on an "normal" amd64 installation, and everything works fine.
On both systems The storage is a directory.

The request via curl (Error):
curl -k --request POST --url https://192.168.1.130:8006/api2/extjs/nodes/RPi4-PVE-1/lxc --header 'CSRFPreventionToken: 61CACEDF:PrFwO+0a05XfXy19Ci9SYOVxDIplQQ9kvtOBwlPW/Mg' --header 'Content-Type: application/x-www-form-urlencoded' --header 'Cookie: PVEAuthCookie=PVE%3Aroot@pam%3A61CACEDF%3A%3AJIDTljH2ReIB3IcvLmK+nLF69FRcjFvn7GUKBstJJzNln3Q8ZO/zNUY/0mnimu287k7ICLhYFibuivHW5MUUTJ1bIYkHO9od6qB+SrsKym42dP84WV/EKiwu0+rWANb8ycLWjjz2mySDiesJsFao3JZl8Y2+Zju+qlVe/XWV6dcz+I79rujee0ikKa59MtMFQ1NJVqQX9Km8KoiED50fUVQ2Kmp01mCcllcMarzohu4koBB/88ESphjA4dIdNFfs0NMoS31vI3T8mLCyfhH2Oy7Ce2gtNiZyes3fRX/njuxaIECObVBilzg3Owg3zxabhaMAQVehNzbuWvY+Bjfaxw%3D%3D' --data hostname=test --data ostemplate=local-data:vztmpl/debian-11-standard_11.0-1_arm64.tar.xz --data rootfs=local:8 --data cores=1 --data memory=512 --data swap=512 --data vmid=200 --data ssh-public-keys=ssh-ed25519%20AAAAC3NzaC1lZDI1NTE5AAAAINTE1E65zBpJWr%2BKhOwwOPRXkjsSe49TXT1EcssXcqOu%20andy%40lappi

Request with password (Working):
curl -k --request POST --url https://192.168.1.130:8006/api2/extjs/nodes/RPi4-PVE-1/lxc --header 'CSRFPreventionToken: 61CACEDF:PrFwO+0a05XfXy19Ci9SYOVxDIplQQ9kvtOBwlPW/Mg' --header 'Content-Type: application/x-www-form-urlencoded' --header 'Cookie: PVEAuthCookie=PVE%3Aroot@pam%3A61CACEDF%3A%3AJIDTljH2ReIB3IcvLmK+nLF69FRcjFvn7GUKBstJJzNln3Q8ZO/zNUY/0mnimu287k7ICLhYFibuivHW5MUUTJ1bIYkHO9od6qB+SrsKym42dP84WV/EKiwu0+rWANb8ycLWjjz2mySDiesJsFao3JZl8Y2+Zju+qlVe/XWV6dcz+I79rujee0ikKa59MtMFQ1NJVqQX9Km8KoiED50fUVQ2Kmp01mCcllcMarzohu4koBB/88ESphjA4dIdNFfs0NMoS31vI3T8mLCyfhH2Oy7Ce2gtNiZyes3fRX/njuxaIECObVBilzg3Owg3zxabhaMAQVehNzbuWvY+Bjfaxw%3D%3D' --data hostname=test --data ostemplate=local-data:vztmpl/debian-11-standard_11.0-1_arm64.tar.xz --data rootfs=local:8 --data cores=1 --data memory=512 --data swap=512 --data vmid=200 --data password=asdfg

Any ideas how to debug this issue?
I would prefer public/private key over a password.

[drewbyp]

I've got the same issue but with the cloudinit config on a VM. I'm struggling to get any detailed info about what might be causing it as well. Looked through various logs and such but I haven't managed to find anything yet.

It's also not only via the API. It's via the UI as well (although I expect that just calls the API anyway).

I'd appreciate some help with debugging this too.

[clayshek]

I'm encountering the same issue, exactly as described above. I have found where I think the error message is coming from in the Proxmox source code: https://github.com/proxmox/pve-common/blob/d9339d016ab5a70a291ae34329f64f0667cd30ae/src/PVE/Tools.pm#L1649

sub validate_ssh_public_keys {
    my ($raw) = @_;
    my @lines = split(/\n/, $raw);

    foreach my $line (@lines) {
	next if $line =~ m/^\s*$/;
	eval {
	    my ($filename, $handle) = tempfile_contents($line);
	    run_command(["ssh-keygen", "-l", "-f", $filename],
			outfunc => sub {}, errfunc => sub {});
	};
	die "SSH public key validation error\n" if $@;
    }
}

I just haven't quite figured out yet why the error is being encountered. Still digging, but hoping the above potentially helps anyone else investigating this.

Editing to add: seem to have confirmed the block above as the offending code. On my Ras Pi, I edited the Tools.pm file ( /usr/share/perl5/PVE/Tools.pm ), and by commenting out the die function on line 1649, I am now able to have a container created with no ssh public key validation error. So the eval seems to be failing for some reason. Possibly something to do with tempfile_contents / O_tmpfile

[peio42]

I had the same issue. It looks like O_TMPFILE flag value is hardcoded in the Tools.pm file, while the value is different on arm64 than amd64.
Here's the patch to get SSH key validation working

--- Tools.pm.orig	2023-04-10 11:18:18.820806645 +0000
+++ Tools.pm	2023-04-11 09:19:13.911804667 +0000
@@ -100,7 +100,7 @@

 use constant {O_PATH    => 0x00200000,
               O_CLOEXEC => 0x00080000,
-              O_TMPFILE => 0x00410000}; # This includes O_DIRECTORY
+              O_TMPFILE => 0x00404000}; # This includes O_DIRECTORY

 use constant {AT_EMPTY_PATH => 0x1000,
               AT_FDCWD => -100};