sap_hana_install: Idempotency check is not thorough enough

[marcelmamula]

Issue

Current idempotency check using /usr/sap/hostctrl/exe/saphostctrl -function ListInstances is not catching midway failed installations.

My installation failed because of SElinux in enforcing mode, but command above is still showing HANA regardless. It is not installed and it is not running.

community.sap_install/roles/sap_hana_install/tasks/hana_exists.yml

Line 17 in 8a89687

ansible.builtin.shell: set -o pipefail && /usr/sap/hostctrl/exe/saphostctrl -function ListInstances | cut -d":" -f2-

hdblcm.log and command output

h02hana0:~ # tail -20 /var/tmp/hdblcm.log
09:54:23.090 - INFO: Total execution time                                          | 00:04:05
09:54:23.090 - INFO:
09:54:23.092 - ERR : Installation of SAP HANA Database System failed.
09:54:23.140 - INFO: -----------------------------------------------------------
09:54:23.140 - INFO: Cleaning up configuration changes...
09:54:23.140 - INFO: -----------------------------------------------------------
09:54:23.140 - INFO:   Checking for any auto-failover settings disabled by the SAP HANA Lifecycle Management...
09:54:23.140 - INFO:     Getting nameserver.ini
09:54:23.140 - INFO:       Reading nameserver.ini
09:54:23.140 - INFO:         Layer 'Default' found
09:54:23.144 - INFO:         Layer 'System' found
09:54:23.145 - INFO:   No auto-failover settings disabled by the SAP HANA Lifecycle Management were found.
09:54:23.145 - INFO: -----------------------------------------------------------
09:54:23.145 - INFO: END: Cleaning up configuration changes (start: 09:54:23.140 duration: 00:00:00)
09:54:23.145 - INFO: -----------------------------------------------------------
09:54:23.145 - INFO: Summary of critical errors
09:54:23.092 - ERR :   Installation of SAP HANA Database System failed.
09:53:51.500 - ERR :     Start instance 90 on master host 'h02hana0' (worker) failed.
09:53:51.500 - ERR :       Cannot establish http connection to unix domain socket '/tmp/.sapstream59013' (No such file or directory)
09:53:51.501 - ERR :     Check the log file for SAP HANA Database error messages
h02hana0:~ # /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
 Inst Info : H02 - 90 - h02hana0 - 753, patch 1236, changelist 2222163
h02hana0:~ #

HANA status

h02hana0:~ # su - h02adm
h02adm@h02hana0:/usr/sap/H02/HDB90> HDB start


StartService
FAIL: Service start failed: systemdI_msg_handler: job '/org/freedesktop/systemd1/job/77170' canceled with 'failed'

ERROR: start service command '/usr/sap/H02/SYS/exe/hdb/sapcontrol -nr 90 -function StartService H02' failed, rc = 1
h02adm@h02hana0:/usr/sap/H02/HDB90> HDB info
USER          PID     PPID  %CPU        VSZ        RSS COMMAND
h02adm     138152   138151   0.8       9740       8576 -sh
h02adm     138231   138152  10.0       7460       5768  \_ /bin/sh /usr/sap/H02/HDB90/HDB info
h02adm     138256   138231   0.0       9408       6384      \_ ps fx -U h02adm -o user:8,pid:8,ppid:8,pcpu:5,vsz:10,rss:10,args

@berndfinger FYI

[berndfinger]

@marcelmamula Just for confirmation: The task

community.sap_install/roles/sap_hana_install/tasks/hana_exists.yml

Line 26 in 8a89687

- name: SAP HANA Checks - Desired HANA is installed and running

succeeds and all tasks which follow are skipped in this case?

[marcelmamula]

@berndfinger Yes that is correct, it detects HANA because it exists, but it is not in executable state because of permissions related to SElinux as documented in https://me.sap.com/notes/3390375 - It cannot be started.

Messaging says Desired HANA is installed and running which is misleading and incorrect.

[berndfinger]

I currently see several possibilities of changing the current behavior:

• Change the message to state that saphostctrl -function ListInstances reported the SAP HANA system as already installed (not implying that it is installed successfully). It would then be up to the user to tell if the system is really installed and to decide to clean up and restart the installation (and maybe also to report this as a bug to SAP).
• Optionally (or always if saphostctrl -function ListInstances reports that an HANA system is installed) run an hdblcm installation check (see

  community.sap_install/roles/sap_hana_install/tasks/post_install.yml

  Line 60 in 8a89687

  - name: Perform an hdblcm installation check

  ).

If this behavior is known to only occur related to SELinux, then maybe running this task:

community.sap_install/roles/sap_hana_install/tasks/pre_install.yml

Line 132 in 8a89687

- name: SAP HANA Pre Install - Configure 'sap_hana_install_root_path' SELinux file contexts

, probably in combination with the SELinux related tasks in the preconfigure roles, could avoid the issue at all?

[marcelmamula]

Proposals for improved idempotency.

Post tasks to ensure HANA is started

Proposal:

1. Execute HDB info and validate if necessary processes are available
2. Execute HDB stop/start if validation in step 1 failed
3. Execute HDB info and validate again
4. Fail if second validation failed.

sap_hana_install_new_system handling

Issue:

If existing HANA is detected, the variable sap_hana_install_new_system is set to false resulting in skip of all tasks.
This causes issue if HANA installation failed, but it created files that resulted in positive detection because:

• firewall ports will not open
• other post tasks will not trigger

Proposal

Improve conditionals for post tasks to actually check if they are applied, without fully skipping with sap_hana_install_new_system.