[Bug]: SSH connection fails after system package update: 'Permission denied (publickey,password)'

[Yurgeman]

Error Message and Logs

After a routine system package update (including openssh-server), the localhost server in Coolify became unreachable with the following error:

Server is not reachable.
Error: root@host.docker.internal: Permission denied (publickey,password).

Manual SSH connection using the generated key also initially failed due to misconfiguration, but after fixing /etc/ssh/sshd_config and ensuring the public key is in ~/.ssh/authorized_keys, manual login works:

ssh -i /data/coolify/ssh/keys/id.root@host.docker.internal root@localhost

However, when trying to assign the same private key to the localhost server in Coolify UI, it shows:

This key is not valid for this server.

Both automatic and manual setup steps from https://coolify.io/docs/knowledge-base/server/openssh were followed precisely.

Steps to Reproduce

Example Repository URL

No response

Coolify Version

v4.0.0-beta.460

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 22

Additional Information

No response

[ShadowArcanist]

Create a new key from the dashboard (Keys & tokens page) then add the new keys public key to /root/.ssh/authorized_keys and try using the new key to validate the server

[Yurgeman]

Create a new key from the dashboard (Keys & tokens page) then add the new keys public key to /root/.ssh/authorized_keys and try using the new key to validate the server

This is the first thing I did but got an error: This key is not valid for this server.

[Yurgeman]

Create a new key from the dashboard (Keys & tokens page) then add the new keys public key to /root/.ssh/authorized_keys and try using the new key to validate the server

This is the first thing I did but got an error: `This key is not valid for this server.`

@andrasbacsai Do you have any suggestions or options to solve the problem?

[simonheisstpeter]

Same happened to me. Any updates on this?

is also mentioned here: #7724

[KillerCodeMonkey]

Same here again and again for a newer hetzner server running metabase via cloud coolify.
After a week the server is not reachable with the same error.

The sad part is, that you can not change/add private keys that easy to hetzer cloud servers.
You need to go in rescue mode and then do things.
In general you would kick the old server and setup a new one. this is really frustrating.

Strange is, that we are running 2 more servers with node apps without any issue.
Each server has its own private key.

[Yurgeman]

Coolify Recovery When Root SSH Key Is Lost

1. Generate a new key on the host

ssh-keygen -t ed25519 -f /root/.ssh/coolify_new -N "" -C "coolify-root"
cat /root/.ssh/coolify_new.pub >> /root/.ssh/authorized_keys

2. Fix permissions (if SSH rejects the key)

chown root:root /root/.ssh/authorized_keys /root/.ssh/id_*
chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys

3. Test SSH connection

ssh -i /root/.ssh/coolify_new -o StrictHostKeyChecking=no root@localhost "echo OK"

4. Copy the key into container and create a new PrivateKey

docker exec -e COOLIFY_TEMP_KEY="$(cat /root/.ssh/coolify_new)" coolify php -r "
require '/var/www/html/vendor/autoload.php';
\$app = require_once '/var/www/html/bootstrap/app.php';
\$app->make(\Illuminate\Contracts\Console\Kernel::class)->bootstrap();
\$k = new \App\Models\PrivateKey();
\$k->name = 'coolify-root';
\$k->private_key = getenv('COOLIFY_TEMP_KEY');
\$k->team_id = 0;
\$k->save();
\$s = \App\Models\Server::find(0);
\$s->private_key_id = \$k->id;
\$s->ip = 'host.docker.internal';
\$s->save();
echo 'New key ID: '.\$k->id;
"

5. Verify the result

docker exec coolify php artisan tinker --execute="echo \App\Models\Server::find(0)->isFunctional() ? 'FUNCTIONAL' : 'NOT FUNCTIONAL';"

Should return FUNCTIONAL.

6. If the container lacks Docker socket — add it to docker-compose.yml

In /data/coolify/source/docker-compose.yml, add to volumes under coolify service:

- /var/run/docker.sock:/var/run/docker.sock:ro

Then apply:

cd /data/coolify/source && docker compose up -d coolify

7. If white screen due to Redis — remove the password

sed -i 's/^REDIS_PASSWORD=.*/REDIS_PASSWORD=null/' /data/coolify/source/.env
docker compose up -d --force-recreate coolify