SAPHanaSR_maintenance_examples.7

.\" Version: 0.160.1
.\"
.TH SAPHanaSR_maintenance_examples 7 "27 Mar 2024" "" "SAPHanaSR"
.\"
.SH NAME
SAPHanaSR_maintenance_examples \- maintenance examples for SAPHana and SAPHanaController.
.PP
.\"
.SH DESCRIPTION
.PP
Maintenance examples for SAPHana and SAPHanaController.
Please see ocf_suse_SAPHana(7) respectively ocf_suse_SAPHanaController(7),
SAPHanaSR.py(7), SAPHanaSrMultiTarget.py(7), susTkOver.py(7), susChkSrv.py(7),
SAPHanaSR-manageAttr(8), for more examples and read the REQUIREMENTS section
below.
.RE
.PP
.\"
.SH EXAMPLES
.PP
\fB*\fR Check status of Linux cluster and HANA system replication pair.

This steps should be performed before doing anything with the cluster, and
after something has been done. See also cs_show_saphanasr_status(8) and section
REQUIREMENTS below.
.PP
.RS 4 
# cs_clusterstate -a
.br
# crm_mon -1r
.br
# crm configure show | grep cli-
.br
# SAPHanaSR-showAttr
.br
# cs_clusterstate -i
.RE
.PP
\fB*\fR Watch status of HANA cluster resources and system replication.

This might be convenient when performing administrative actions or cluster tests. It does not replace the afore mentioned checks. See also cs_show_saphanasr_status(8).
.PP
.RS 4
# watch -n9 "crm_mon -1r --include=none,nodes,resources,failures;echo;SAPHanaSR-showAttr;cs_clusterstate -i|grep -v '#'"
.RE
.PP
\fB*\fR Overview on stopping the HANA database at one site.

This procedure does work for scale-up and scale-out. No takeover will be done. This procedure
should be used, when it is necessary to stop the HANA database. Stopping the HANA database
should not be done by just stopping the Linux cluster or shutting down the OS. This particularly
applies to scale-out systems. It might be good to define upfront which HANA site needs to be
stopped. In case both sites need to be stopped, it might be good to define the order. First
stopping the primary should keep system replication in sync.
.br
How long a stop will take, depends on database size, performance of underlying infrastructure,
SAP HANA version and configuration. Please refer to SAP HANA documentation for details on
tuning and stopping an HANA database.
.PP
.RS 4
1. Checking status of Linux cluster and HANA system replication pair.
.br
2. Setting SAPHana or SAPHanaController multi-state resource into maintenance.
.br
3. Stopping HANA database at the given site by using "sapcontrol -nr <nr> -function StopSystem".
.br
4. Checking that HANA is stopped.
.RE
.PP
Note: Do not forget to end the resource maintenance after you have re-started the HANA database.
.PP
\fB*\fR Initiate an administrative takeover of the HANA primary from one node to the other by using the Linux cluster. 

This procedure does not work for scale-out. On scale-up, it will stop the HANA primary.
This might take a while. If you want to avoid waiting for the stopped primary,
use the below procedure which suspends the primary.
If the cluster should also register the former primary as secondary, AUTOMATED_REGISTER="true" is needed. Before the takeover will be initiated, the status of the Linux cluster and the HANA system replication has to be checked. The takeover should be initiated as forced migration of the multi-state SAPHana resource.
.br
Not working: Regular migration, migration of IP address, migration of primitive SAPHana resource, setting primary node standby.
.br
After takeover of the primary has been finished, the migration rule has to be deleted. If AUTOMATED_REGISTER="true" is set, finally the former primary will be registered as secondary, once the migration rule has been deleted.
.PP
.RS 4
# crm_mon -1r
.br
# SAPHanaSR-showAttr
.br
# crm configure show | grep cli-
.br
# cs_clusterstate -i
.br
# crm resource move msl_SAPHana_SLE_HDB10 force
.br
# cs_clusterstate -i
.br
# SAPHanaSR-showAttr
.br
# crm resource clear msl_SAPHana_SLE_HDB10
.br
# SAPHanaSR-showAttr
.br
# cs_clusterstate -i
.RE
.PP
Note: Former versions of the Linux cluster used "migrate" instead of "move" and "unmigrate" instead of "clear".
.PP
\fB*\fR Perform an SAP HANA takeover by using SAP tools. 

The procedure is described here for scale-out. It works for scale-up as well. 
The procedure will stop the HANA primary. This might take a while. If you want
to avoid waiting for the stopped primary, use the below procedure which suspends
the primary.
The status of HANA databases, system replication and Linux cluster has to be
checked.
The SAP HANA resources are set into maintenance, an sr_takeover is performed,
the old primary is registered as new secondary.
Therefor the correct secondary site name has to be used, see later example.
Finally the SAP HANA resources are given back to the Linux cluster.
See also section REQUIREMENTS below and later example on determining the correct site name.
.PP
.RS 2
1. On either node
.RE
.RS 4
# crm_mon -1r
.br
# SAPHanaSR-showAttr
.br
# crm configure show | grep cli-
.br
# cs_clusterstate -i
.br
If everything looks fine, proceed.
.br
# crm resource maintenance msl_SAPHana_SLE_HDB10
.br
# crm_mon -1r
.RE
.RS 2
2. On the HANA primary master nameserver (e.g. node11)
.RE
.RS 4
# su - sleadm
.br
~> sapcontrol -nr 10 -function StopSystem HDB
.br
.\" TODO check the below
~> sapcontrol -nr 10 -function GetSystemInstanceList
.RE
.PP
.RS 2
\fBOnly proceed after you made sure the HANA primary is down!\fR
.RE
.PP
.RS 2
3. On the HANA secondary master nameserver (e.g. node21)
.RE
.RS 4
# su - sleadm
.br
~> hdbnsutil -sr_takeover
.br
~> HDBsettings.sh systemReplicationStatus.py; echo RC:$?
.br
~> HDBsettings.sh landscapeHostConfiguration.py; echo RC:$?
.br
If everything looks fine, proceed.
.RE
.RS 2
4. On the former HANA primary master nameserver, now future secondary master nameserver (e.g. node11)
.RE
.RS 4
~> hdbnsutil -sr_register --remoteHost=node21 --remoteInstance=10 --replicationMode=sync
--name=site2 --operationMode=logreplay
.br
~> sapcontrol -nr 10 -function StartSystem HDB
.br
~> exit
.br
.RE
.RS 2
5. On the new HANA primary master nameserver (e.g. node21)
.RE
.RS 4
.br
~> HDBsettings.sh systemReplicationStatus.py; echo RC:$?
.br
~> HDBsettings.sh landscapeConfigurationStatus.py; echo RC:$?
.br
~> exit
.br
If everything looks fine, proceed.
.RE
.RS 2
6. On either node
.RE
.RS 4
.br
# cs_clusterstate -i
.br
# crm resource refresh msl_SAPHana_SLE_HDB10
.br
# crm resource maintenance msl_SAPHana_SLE_HDB10 off
.br
# SAPHanaSR-showAttr
.br
# crm_mon -1r
.br
# cs_clusterstate -i
.RE
.PP
\fB*\fR Overview on SAP HANA takeover using SAP tools and suspend primary feature.

The procedure works for scale-up and scale-out.
The status of HANA databases, system replication and Linux cluster has to be
checked.
The SAP HANA resources are set into maintenance, an sr_takeover is performed
with suspending the primary, the old primary is registered as new secondary.
Therefor the correct secondary site name has to be used.
Finally the SAP HANA resources are given back to the Linux cluster.
See also section REQUIREMENTS below and later example on determining the correct site name.
.PP
.RS 2
1. Check status of Linux cluster and HANA, show current site names.
.br
2. Set SAPHanaController multi-state resource into maintenance.
.br
3. Perform the takeover, make sure to use the suspend primary feature:
.RE
.RS 4
~> hdbnsutil -sr_takeover --suspendPrimary
.RE
.RS 2
4. Check if the new primary is working.
.br
5. Stop suspended old primary.
.br
6. Register old primary as new secondary, make sure to use the correct site name.
.br
7. Start the new secondary.
.br
8. Check new secondary and its system replication.
.br
9. Refresh SAPHanaController multi-state resource.
.br
10. Set SAPHanaController multi-state resource to managed.
.br
11. Finally check status of Linux cluster and HANA. 
.RE
.PP
\fB*\fR Check the two site names that are known to the Linux cluster. 

This is useful in case AUTOMATED_REGISTER is not yet set. In that case a former  primary needs to be registered manually with the former site name as new secondary. The point is finding the site name that already is in use by the Linux cluster. That exact site name has to be used for registration of the new secondary. See also REQUIREMENTS of SAPHanaSR(7) and SAPHanaSR-ScaleOut(7).
.br
In this example, node is suse11 on the future secondary site to be registered. Remote HANA master nameserver is suse21 on current primary site. Lowercase-SID is ha1.
.PP
.RS 4
# crm configure show suse11 suse21
.br
# crm configure show SAPHanaSR | grep hana_ha1_site_mns
.br
# ssh suse21
.br
# su - ha1adm -c "hdbnsutil -sr_state; echo rc: $?"
.br
# exit
.RE
.PP
\fB*\fR Manually start the HANA primary if only one site is available.

This might be necessary in case the cluster can not detect the status of both sites.
This is an advanced task.
.PP
\fBBefore doing this, make sure HANA is not primary on the other site!\fR
.PP
.RS 2
1. Start the cluster on remaining nodes.
.br
2. Wait and check for cluster is running, and in status idle.
.br
3. Become sidadm, and start HANA manually.
.br
4. Wait and check for HANA is running.
.br
5. In case the cluster does not promote the HANA to primary, instruct the cluster to migrate the IP address to that node.
.br
6. Wait and check for HANA has been promoted to primary by the cluster.
.br
7. Remove the migration rule from the IP address.
.br
8. Check if cluster is in status idle.
.br
9. You are done, for now.
.br
10. Please bring back the other node and register that HANA as soon as possible. If the HANA primary stays alone for too long, the log area will fill up.
.RE
.PP
.\"
\fB*\fR Overview on maintenance procedure for HANA, the Linux cluster remains running, on pacemaker 1.0.

See also section REQUIREMENTS below.
.PP
.RS 2
1. Check if everything looks fine, see above.
.br
2. Set the Linux cluster into maintenance mode.
.RE
.RS 4
# crm configure property maintenance-mode=true
.RE
.RS 2
3. Perform the HANA maintenance, e.g. update to latest SPS.
.br
4. Set the SAPHanaController m/s resource to unmanaged.
.RE
.RS 4
# crm resource unmanage <m/s-resource>
.RE
.RS 2
5. Set the Linux cluster back into ready mode.
.RE
.RS 4
# crm configure property maintenance-mode=false
.RE
.RS 2
6. Clean up the SAPHanaController m/s resource.
.RE
.RS 4
# crm resource cleanup <m/s-resource> node <node>
.RE
.RS 2
7. Set the SAPHanaController m/s resource back to managed.
.RE
.RS 4
# crm resource manage <m/s-resource>
.RE
.RS 2
8. Check if everything looks fine, see above.
.RE
.PP
.RE
Note: The YaST module hana_updater does something similar, combined with an
administrative takeover.
.PP
On pacemaker 1.1.19 and 2.0 respectively do the following. 
.PP
.RS 2
1. Check if everything looks fine, see above.
.br
2. Set the SAPHanaController multi-state resource into maintenance mode.
.RE
.RS 4
# crm resource maintenance msl_SAPHanaCon_SLE_HDB10 on
.RE
.RS 2
3. Perform the HANA maintenance, e.g. update to latest SPS.
.br
4. Tell the cluster to forget about HANA status and to reprobe the resources.
.RE
.RS 4
# crm resource refresh msl_SAPHanaCon_SLE_HDB10
.RE
.RS 2
5. Set the SAPHanaController multi-state resource back to managed.
.RE
.RS 4
# crm resource maintenance msl_SAPHanaCon_SLE_HDB10 off
.RE
.RS 2
6. Remove the meta attribute from CIB, optional.
.RE
.RS 4
# crm resource meta msl_SAPHanaCon_SLE_HDB10 delete maintenance
.RE
.RS 2
7. Check if everything looks fine, see above.
.RE
.PP
The two procedures must not be mixed. If the procedure for pacemaker-1.0 has
been used, left-over maintenance attribute have to be removed from the CIB
before proceeding with the new procedure for pacemaker-2.0.
.PP
\fB*\fR Overview on maintenance procedure for Linux, HANA remains running, on pacemaker-2.0.

It is necessary to wait for each step to complete and to check the result. It
also is necessary to test and document the whole procedure before applying in production.
See also section REQUIREMENTS below and example on checking status of HANA and cluster above.
.PP
.RS 2
1. Check status of Linux cluster and HANA, see above.
.br
2. Set HANA multistate resource into maintenance mode.
.RE
.RS 4
# crm resource maintenance msl_... on
.RE
.RS 2
3. Set the Linux cluster into maintenance mode, on either node.
.RE
.RS 4
# crm maintenance on
.RE
.RS 2
4. Stop Linux Cluster on all nodes. Make sure to do that on all nodes.
.RE
.RS 4
# crm cluster run "crm cluster stop"
.RE
.RS 2
.PP
5. Perform Linux maintenance.
.PP
6. Start Linux cluster on all nodes. Make sure to do that on all nodes.
.RE
.RS 4
# crm cluster run "crm cluster start"
.RE
.RS 2
7. Set cluster ready for operations, on either node.
.RE
.RS 4
# crm maintenance off
.RE
.RS 2
8. Let Linux cluster detect status of HANA multistate resource, on either node.
.RE
.RS 4
# crm resource refresh msl_...
.RE
.RS 2
9. Set HANA multistate resource ready for operations, on either node.
.RE
.RS 4
# crm maintenance msl_... off
.RE
.RS 2
10. Check status of Linux cluster and HANA, see above.
.RE
.PP
\fB*\fR Overview on simple procedure for stopping and temporarily disabling the Linux cluster,
HANA gets fully stopped.

This procedure can be used to update HANA, OS or hardware.
HANA roles and resource status remains unchanged.
It is necessary to wait for each step to complete and to check the result.
It also is necessary to test and document the whole procedure before applying in production.
.PP
.RS 2
1. disabling pacemaker on HANA primary
.br
2. disabling pacemaker on HANA secondary
.br
3. stopping cluster on HANA secondary
.RS 2
- HANA secondary will be stopped
.br
- system replication goes SFAIL
.RE
4. stopping cluster on HANA primary
.RS 2
- HANA primary will be stopped
.RE
5. doing something with OS or hardware
.br
6. enabling pacemaker on HANA primary
.br
7. enabling pacemaker on HANA secondary
.br
8. starting cluster on HANA primary
.RS 2
- HANA stays down
.RE
9. starting cluster on HANA secondary
.RS 2
- HANA primary and secondary will be started
.br
- system replication recovers to SOK
.RE

Note: HANA is not available from step 4 to step 9. 
.RE
.PP
\fB*\fR Overview on update procedure for the SAPHanaSR and SAPHanaSR-ScaleOut package.

This procedure can be used to update RAs, HANA HADR provider hook scripts and related tools while HANA and Linux cluster stay online. See also SAPHanaSR-manageAttr(8) for details on reloading the HANA HADR provider.
.PP
.RS 2
1. Check status of Linux cluster and HANA, see above.
.br
2. Set resources SAPHana (or SAPHanaController) and SAPHanaTopology to maintenance.
.br
3. Update RPM on all cluster nodes.
.br
4. Reload HANA HADR provider hook script on both sites.
.br
5. Refresh resources SAPHana (or SAPHanaController) and SAPHanaTopology.
.br 
6. Set resources SAPHana (or SAPHanaController) and SAPHanaTopology from maintenance to managed.
.br
7. Check status of Linux cluster and HANA, see above.
.RE
.PP
\fB*\fR Remove left-over maintenance attribute from overall Linux cluster.

This could be done to avoid confusion caused by different maintenance procedures.
See above overview on maintenance procedures with running Linux cluster.
Before doing so, check for cluster attribute maintenance-mode="false".
.PP
.RS 4
# SAPHanaSR-showAttr
.br
# crm_attribute --query -t crm_config -n maintenance-mode
.br
# crm_attribute --delete -t crm_config -n maintenance-mode
.br
# SAPHanaSR-showAttr
.RE
.PP
\fB*\fR Remove left-over standby attribute from Linux cluster nodes.

This could be done to avoid confusion caused by different maintenance procedures.
See above overview on maintenance procedures with running Linux cluster.
Before doing so for all nodes, check for node attribute standby="off" on all nodes.
.PP
.RS 4
# SAPHanaSR-showAttr
.br
# crm_attribute --query -t nodes -N node1 -n standby
.br
# crm_attribute --delete -t nodes -N node1 -n standby
.br
# SAPHanaSR-showAttr
.RE
.PP
\fB*\fR Remove left-over maintenance attribute from resource.

This should usually not be needed.
See above overview on maintenance procedures with running Linux cluster.
.PP
.RS 4
# SAPHanaSR-showAttr
.br
# crm_resource --resource cln_SAPHanaTop_HA1_HDB00 --delete-parameter maintenance --meta
.br
# SAPHanaSR-showAttr
.RE
.PP
\fB*\fR Manually update global site attribute.

In rare cases the global site attribute hana_<sid>_glob_prim or
hana_<sid>_glob_sec is not updated automatically after successful takeover,
while all other attributes are updated correctly. The global site attribute
stays outdated even after the cluster has been idle for a while.
In this case, that site attribute could be updated manually.
Make sure everything else is fine and just the global site attribute has not
been updated. Updating hana_<sid>_glob_sec for SID HA1 with site name VOLKACH:
.PP
.RS 4
# crm configure show SAPHanaSR
.br
# crm_attribute --type crm_config --name hana_ha1_glob_sec --update=VOLKACH
.br
# crm configure show SAPHanaSR
.RE
.PP
\fB*\fR Upgrade scale-out srHook attribute from old-style to multi-target.

As final result of this upgrade, the RAs and hook script are upgraded from
old-style to multi-target. Further the Linux cluster's old-style global srHook
attribute hana_${sid}_glob_srHook is replaced by site-aware attributes
hana_${sid}_site_srHook_${SITE}. New auxiliary attributes are introduced.
The complete procedure and related requirements are described in detail in
manual page SAPHanaSR-manageAttr(8).
.br
The procedure at a glance:
.PP
.RS 2
a. Initially check if everything looks fine.
.br
b. Set Linux cluster resources SAPHanaController and SAPHanaTopology into maintenance.
.br
c. Install multi-target aware SAPHanaSR-ScaleOut package on all nodes.
.br
d. Adapt sudoers permission on all nodes.
.br
e. Replace HANA HADR provider configuration on both sites.
.br
f. Reload HANA HADR provider hook script on both sites.
.br
g. Check Linux cluster and HANA HADR provider for matching defined upgrade entry state.
.br
h. Migrate srHook attribute from old-style to multi-target.
.br
i. Check Linux cluster for matching defined upgrade target state.
.br
j. Set Linux cluster resources SAPHanaController and SAPHanaTopology from maintenance to managed. 
.br
k. Optionally connect third HANA site via system replication outside of the Linux cluster.
.br
l. Finally check if everything looks fine.
.RE
.PP
.\"
.SH FILES
.br
.PP
.\"
.SH REQUIREMENTS
.br
\fB*\fR For the current version of the resource agents that come with the software packages SAPHanaSR and SAPHanaSR-ScaleOut, the support is limited to the scenarios and parameters described in the respective manual pages SAPHanaSR(7) and SAPHanaSR-ScaleOut(7).
.PP
\fB*\fR Be patient. For detecting the overall HANA status, the Linux cluster
needs a certain amount of time, depending on the HANA and the configured
intervals and timeouts.
.PP
\fB*\fR Before doing anything, always check for the Linux cluster's idle status,
left-over migration constraints, and resource failures as well as the HANA
landscape status, and the HANA SR status.
.PP
\fB*\fR Maintenance attributes for cluster, nodes and resources must not be mixed.
.PP
\fB*\fR The Linux cluster needs to be up and running to allow HA/DR provider events
being written into CIB attributes. The current HANA SR status might differ from CIB
srHook attribute after Linux cluster maintenance.
.PP
\fB*\fR Manually activating an HANA primary, like start of HANA primary or takeover
outside the cluster creates risk of a duplicate-primary situation. The user is
responsible for data integrity, particularly when activating an HANA primary. See
also susTkOver.py(7).
.PP
\fB*\fR HANA site names are discovered automatically when the RAs are activated the
very first time. That exact site names have to be used later for all manual tasks.
.PP
.\"
.SH BUGS
.\" TODO
In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.
.PP
.\"
.SH SEE ALSO
.br
\fBocf_suse_SAPHanaTopology\fP(7) , \fBocf_suse_SAPHana\fP(7) ,
\fBocf_suse_SAPHanaController\fP(7) ,
\fBSAPHanaSR.py\fP(7) , \fBSAPHanaSrMultiTarget.py\fP(7) ,
\fBsusCostOpt.py\fP(7) , \fBsusTkOver.py\fP(7) , \fBsusChkSrv.py\fP(7) ,
\fBSAPHanaSR-monitor\fP(8) , \fBSAPHanaSR-showAttr\fP(8) , \fBSAPHanaSR\fP(7) ,
\fBSAPHanaSR-ScaleOut\fP(7) , \fBSAPHanaSR-manageAttr\fP(8) ,
\fBSAPHanaSR-manageProvider\fP(8) ,
\fBcs_clusterstate\fP(8) , \fBcs_show_saphanasr_status\fP(8) ,
\fBcs_wait_for_idle\fP(8) ,
\fBcrm\fP(8) , \fBcrm_simulate\fP(8) , \fBcrm_mon\fP(8) , \fBcrm_attribute\fP(8) ,
.br
https://documentation.suse.com/sbp/sap/ ,
.\" TODO https://www.suse.com/media/presentation/TUT90846_towards_zero_downtime%20_how_to_maintain_sap_hana_system_replication_clusters.pdf ,
.br
https://www.suse.com/support/kb/doc/?id=000019253 ,
.br
https://www.suse.com/support/kb/doc/?id=000019207 ,
.br
https://www.suse.com/support/kb/doc/?id=000019142 ,
.br
https://www.suse.com/c/how-to-upgrade-your-suse-sap-hana-cluster-in-an-easy-way/ ,
.br
https://www.suse.com/c/tag/towardszerodowntime/ ,
.br
https://help.sap.com/doc/eb75509ab0fd1014a2c6ba9b6d252832/1.0.12/en-US/SAP_HANA_Administration_Guide_en.pdf
.PP
.\"
.SH AUTHORS
.br
F.Herschel, L.Pinne.
.PP
.\"
.SH COPYRIGHT
(c) 2017-2018 SUSE Linux GmbH, Germany.
.br
(c) 2019-2024 SUSE LLC
.br
This maintenance examples are coming with ABSOLUTELY NO WARRANTY.
.br
For details see the GNU General Public License at
http://www.gnu.org/licenses/gpl.html
.\"