Standalone Fresh Installation of 4.20.0.0 UI provides 503

[tatay188]

problem

I was trying to Install the 4.20.0.0 as Standalone and not as an Upgrade from 4.19.x.x and UI every attempt provides 503 and logs stop even with DEBUG Enabled. Running on a minimum Ubuntu 22.04

versions

The versions of ACS, hypervisors, storage, network etc..

ACS 4.20.0.0
HYP: KVM
STOR: CEPH, EMC Unity
NETWORK: Pure L3.

The steps to reproduce the bug

1. There is a long e-mail trail with Wei ZHOU. [(https://lists.apache.org/thread/1fzkjcfvo5vymdhlpgwl5n7m3nxwoxkd)]

Scenario Two Management servers with Separate DB. ALl Ubuntu 22.04 Minimum Install
MGMT-SEVER-1 Uninstalled 4.20.0.0 getting 503 in the UI. Purged and cleaned
1 ) LOaded the repositories to 4.19
2 ) Installed 4.19.1.3 Succesfully
2.1)UI fails providing 503, The UI works if I don't use -m and -k options
3) Reinstalled Successfully without the -m -k options
3.1) UI Works on 4.19.1.3
4) Upgraded to 4.20.0.0
4.1) UI Works on 4.20.0.0

MGMT-SEVER-2 Installed a fresh standalone 4.20.0.0 - No upgrade on the second server

1. All db was successfully installed without the -m -k and without the --deploy-as options. NOTE: every time it installs without errors when I use the options -m -k, but the UI does not work throws a 503, same problem observed on 4.19.x.x
2. UI throws 503 on the second server
3. What I found so far, I did not check everything:

On MGMT-SERVER-1 SERVER
`/etc/cloudstack/management# ls -al

total 100

drwxr-xr-x 2 root root 4096 Feb 15 21:38 .
drwxr-xr-x 5 root root 4096 Feb 14 13:36 ..
-rw-r--r-- 1 root root 19996 Nov 28 05:00 config.json
-rw-r----- 1 root cloud 4824 Feb 15 21:00 db.properties
-rw-r----- 1 root root 4800 Feb 14 00:20 db.properties.OMAR
-rw-r--r-- 1 root root 22806 Nov 28 04:55 ehcache.xml
-rw-r--r-- 1 root root 981 Nov 28 04:55 environment.properties
-rw-r--r-- 1 root root 931 Nov 28 04:55 java.security.ciphers
-rw-r--r-- 1 root root 8 Feb 15 21:00 key
-rw-r--r-- 1 root root 5338 Nov 28 04:55 log4j-cloud.xml
**lrwxrwxrwx 1 root root 42 Feb 13 01:37 log4j.xml -> /etc/cloudstack/management/log4j-cloud.xml** <<not present on second server lrwxrwxrwx 1 root root 42 Feb 14 01:37 log4j2.xml -> /etc/cloudstack/management/log4j-cloud.xml

On MGMT-SERVER-2 SERVER
`/etc/cloudstack/management# ls -al

total 100

drwxr-xr-x 2 root root 4096 Feb 15 21:38 .
drwxr-xr-x 5 root root 4096 Feb 14 13:36 ..
-rw-r--r-- 1 root root 19996 Nov 28 05:00 config.json
-rw-r----- 1 root cloud 4824 Feb 15 21:00 db.properties
-rw-r----- 1 root root 4800 Feb 14 00:20 db.properties.
-rw-r--r-- 1 root root 22806 Nov 28 04:55 ehcache.xml
-rw-r--r-- 1 root root 981 Nov 28 04:55 environment.properties
-rw-r--r-- 1 root root 931 Nov 28 04:55 java.security.ciphers
-rw-r--r-- 1 root root 8 Feb 15 21:00 key
-rw-r--r-- 1 root root 5338 Nov 28 04:55 log4j-cloud.xml
lrwxrwxrwx 1 root root 42 Feb 14 13:36 log4j2.xml -> /etc/cloudstack/management/log4j-cloud.xml

On the 4.20.0.0 the link log4j.xml is missing. Then I manually added the symlink on the MGMT-SERVER-2 /etc/cloudstack/management directory

ln -s /etc/cloudstack/management/log4j-cloud.xml log4j.xml

restarted the Management server and works!!!, even If I never installed the 4.19.x.x on the second MGMT-SERVER-2
...

What to do about it?

Work around for the standalone:
on the /etc/cloudstack/management directory create a symlink.

ln -s /etc/cloudstack/management/log4j-cloud.xml log4j.xml

Expected to work as Standalone. Not as Upgrade.
If 4.20.0.0 is intended as an upgrade only: Please establish/define a path for the upgrade.
If is not intended as an Standalone, please test/validate as a Standalone.

Thank you

Omar K.
Sonic Cloud 11 Inc.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

can you check if key file is changed when you run with -m and -k option

the content of key file should be the value of -k argument

[tatay188]

Hello Wei,

We conducted several tests and even removed the key file. I can confirm that the key is correctly populated with the -k argument value; however, it does not work on either version 4.20 or 4.19—we consistently receive a 503 error and very little amount of logs as I have had share with you on the e-mail trails.

Additionally, In our preprod scenario we have multiple management servers and a separate cluster database.

Omar F Kaltenberg
Sonic Cloud 11 Inc.

[weizhouapache]

Hello Wei,

We conducted several tests and even removed the key file. I can confirm that the key is correctly populated with the -k argument value; however, it does not work on either version 4.20 or 4.19—we consistently receive a 503 error and very little amount of logs as I have had share with you on the e-mail trails.

Additionally, In our preprod scenario we have multiple management servers and a separate cluster database.

Omar F Kaltenberg Sonic Cloud 11 Inc.

@tatay188
have you tried with different keys ? password, password2, password3, etc

[DaanHoogland]

@weizhouapache was this reproduced by you?

[DaanHoogland]

@tatay188 , is this still an issue?