Unable to add host due to networking or authentification error

[MilanHofmann]

ISSUE TYPE

• Bug Report

COMPONENT NAME

host

CLOUDSTACK VERSION

4.18

CONFIGURATION

advanced networking with KVM

OS / ENVIRONMENT

Ubuntu focal fossa

SUMMARY

I am not able to add a host to a zone.
In the ui a message pops up "Unable to reach the management server or a browser extension may be blocking the network request. in the ui".
the mangement server cloudstack service status says "trying to add new host ip-address Failed to authenticate with ssh key, retrying with password." - Although i selected password as auth method.
When adding the ssh key, and selecting the option for system ssh key I get "Could not add host at [http://ip-address/] with zone [1], pod [1] and cluster [1] due to: [Authentication error with ssh private key]."

BUT:
ssh root@ip-address from the management server works just fine, as well as the usual password auth.

STEPS TO REPRODUCE

add a zone and try to add a host.

EXPECTED RESULTS

Host is added.

ACTUAL RESULTS

network error when using the password auth - "Unable to reach the management server or a browser extension may be blocking the network request. in the ui"
Auth error when selecting system ssh key - "Could not add host at [http://ip-address/] with zone [1], pod [1] and cluster [1] due to: [Authentication error with ssh private key]."

[weizhouapache]

@MilanHofmann
When you add a host with password, you can ignore the warning message like "unable to connect to host via ssh key", which is not the root cause if you are unable to add the host.
You may need to check other errors in the log.

Regarding adding host via ssh key, please refer to cloudstack documentation: https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#adding-a-host

Before adding the host in CloudStack do the following,

Copy the SSH public key from /var/cloudstack/management/.ssh/id_rsa.pub on the management server
Add the copied key to /root/.ssh/authorized_keys file on the host

[MilanHofmann]

@weizhouapache yes I did that.
ssh root@ip-address from the management server works just fine, as well as the usual password auth.

[MilanHofmann]

@weizhouapache
After un- and reinstalling cloudstack-agent I get ->
Could not add host at [http://10.200.1.11] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].

Can you give me a hint, why this is suddenly not possible anymore and how I maybe give the right permsissions for an automatic setup?

[weizhouapache]

@MilanHofmann
Which cloudstack version do you use ?
The issue should have been fixed by #7090 (which fixed #6716)

[MilanHofmann]

@weizhouapache 4.18

[weizhouapache]

@weizhouapache 4.18

@MilanHofmann
you can retry after the following changes on the kvm host

if [ -f "$LIBVIRTD_FILE" ]; then
        echo "Reverting libvirtd to not listen on TLS"
        sed -i "s,^listen_tls=1,listen_tls=0,g" $LIBVIRTD_FILE
        systemctl restart libvirtd
    fi

    echo "Removing cloud.* files in /etc/cloudstack/agent"
    rm -f /etc/cloudstack/agent/cloud.*

[MilanHofmann]

@weizhouapache

I have removed the files and even reinstalled the operating system on the host.
Still the same issue.
Funny thing is, that management server + cs management service is a fresh install as well.

[weizhouapache]

@weizhouapache

I have removed the files and even reinstalled the operating system on the host. Still the same issue. Funny thing is, that management server + cs management service is a fresh install as well.

@MilanHofmann
can you please upload the full log ?

[MilanHofmann]

@weizhouapache
I hope this covers the relevant part.
The management server's setup was yesterday so there would be a lot of text.
Please let me know if you need the setup part as well.

mgmtlog.txt
agentlog.txt

[boie0025]

If anyone stumbles across this unable to find the CloudStack ssh public key, I finally found on Ubuntu 20.04 -- it was in /var/lib/cloudstack/management/.ssh.

[weizhouapache]

If anyone stumbles across this unable to find the CloudStack ssh public key, I finally found on Ubuntu 20.04 -- it was in /var/lib/cloudstack/management/.ssh.

yes, the location is different on OSes.

on centos, the location is

# echo ~cloud/.ssh/
/var/cloudstack/management/.ssh/

we need to update the doc
https://docs.cloudstack.apache.org/en/latest/installguide/configuration.html#adding-a-host

[weizhouapache]

@weizhouapache I hope this covers the relevant part. The management server's setup was yesterday so there would be a lot of text. Please let me know if you need the setup part as well.

mgmtlog.txt agentlog.txt

@MilanHofmann
I noticed you have created other issues. Has this issue been already fixed ?

[MilanHofmann]

@weizhouapache
Actually the issue was gone after another clean install.
Hard to tell what the exact issue was, it did not occur anymore.