-
Notifications
You must be signed in to change notification settings - Fork 10
/
s4s_tune_wmp.xml
1043 lines (966 loc) · 38.2 KB
/
s4s_tune_wmp.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter
[
<!ENTITY % entities SYSTEM "generic-entities.ent">
%entities;
]>
<chapter xmlns="http://docbook.org/ns/docbook" xml:id="cha-memory-protection"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0">
<title>Tuning &wmp;</title>
<important>
<para>
If you have &systemd;-based &sap; instances, read
<xref linkend="sec-sap-systemd-mix-wmp" /> and
<xref linkend="sec-sap-systemd-wmp" /> before setting up &wmp;.
</para>
</important>
<para>
Keeping &sap; applications in physical memory is essential for their
performance. In older product versions, the Page Cache Limit prevented a swap
out to disk by a growing page cache (in &productname; 11 SP1 onwards and in
&productname; 12). In &productname; 15, the Page Cache Limit has been
replaced by the more advanced &wmp;.
</para>
<para>
&wmp; puts &sap; instances into a dedicated cgroup (v2) and tells the kernel,
by the <varname>memory.low</varname> parameter, the amount of memory to keep
in physical memory. This protects the processes in this cgroup against any
form of memory pressure outside that cgroup, including a growing page cache.
&wmp; cannot protect against memory pressure inside this cgroup. It covers
the memory of <emphasis>all</emphasis> instances together on one host.
</para>
<para>
The value for <varname>memory.low</varname> depends on the kind of &sap;
instance and the workload and needs to be configured manually. If the system
is under extreme pressure, the Linux kernel will ignore the
<varname>memory.low</varname> value and try to stabilize the whole system,
even by swapping or invoking the OOM killer.
</para>
<para>
For more information about cgroups, see
<link xlink:href="https://documentation.suse.com/sles-15/html/SLES-all/cha-tuning-cgroups.html"/>.
</para>
<sect1 xml:id="sec-memory-protection-architecture">
<title>Architecture</title>
<para>
WMP relies on three components:
</para>
<variablelist>
<varlistentry>
<term>cgroup2 memory controller (Linux kernel)</term>
<listitem>
<para>
The cgroup2 memory controller parameter <parameter>memory.low</parameter>
allows defining an amount of memory, which the Linux kernel will keep in
physical memory. This amount of memory will be excluded from the
reclaiming process unless the entire system is in a critical memory
situation.
</para>
<para>
WMP uses <parameter>memory.low</parameter> to prevent memory from SAP
processes from being paged or swapped out to disk. Apart from the memory
controller, cgroup1 controllers are still available, but are not mounted
any more.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>&systemd;</term>
<listitem>
<para>
&systemd; provides the infrastructure to create and maintain the cgroup
hierarchy and allows the configuration of cgroup parameters. WMP ships
&systemd; configuration files to allow easy configuration of
<parameter>memory.low</parameter> via &systemd; methods.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>&sap; start service</term>
<listitem>
<para>
The SAP start service manages the starting and stopping of &sap;
instances. An important feature for WMP is the configurable execution of
programs before the instance itself gets started in the instance profile.
WMP uses this method to call a program to move the
<systemitem>sapstart</systemitem> process into a designated cgroup, so
the &sap; instance will be started inside that cgroup.
</para>
</listitem>
</varlistentry>
</variablelist>
</sect1>
<sect1 xml:id="sec-memory-protection-support">
<title>Support for &wmp;</title>
<para>
WMP is supported for &productname; &productnumber; on &x86-64; and &power;
for one or multiple &sap; systems on one host, such as:
</para>
<itemizedlist>
<listitem>
<para>
App Server (&netweaver;, &s4h;) or
</para>
</listitem>
<listitem>
<para>
&hana; 1.0/2.0
</para>
</listitem>
</itemizedlist>
<para>
&wmp; does not cover databases other than &hana;. Depending on their start
method, the processes might run inside or outside the dedicated cgroup. If
they run inside, the memory consumption needs to be taken into account when
determining <varname>memory.low</varname>.
</para>
<important>
<title>Restrictions of WMP</title>
<para>
Using WMP comes with benefits, but you should be aware of some
restrictions:
</para>
<itemizedlist>
<listitem>
<para>
WMP cannot protect against memory pressure inside the dedicated cgroup.
</para>
</listitem>
<listitem>
<para>
WMP cannot protect &sap; systems or their instances from each other. All
&sap; processes share the same memory limit. If you have multiple &sap;
systems (for example, &netweaver; and &s4h;), WMP cannot shield one &sap;
application from the other.
</para>
</listitem>
<listitem>
<para>
Support for &suse;’s HA cluster solution is not yet available.
</para>
</listitem>
</itemizedlist>
</important>
</sect1>
<sect1 xml:id="sec-memory-protection-setup">
<title>Setting up &wmp;</title>
<remark>toms 2020-09-28: Some introduction para is missing here.</remark>
<sect2 xml:id="sec-memory-protection-setup-preparation">
<title>Preparing for &wmp;</title>
<procedure>
<step>
<para>
Check if your &sap; software (&hana;, &netweaver; etc) is installed. The
group <systemitem class="groupname">sapsys</systemitem> is needed during
the package installation of <package>sapwmp</package> later. If you skip
that part, you will get a warning message (see
<xref linkend="adm-order-of-packages"/>).
</para>
</step>
<step>
<para>
Stop the &sap; system:
</para>
<screen>&prompt.root;<command>systemctl</command> stop sapinit</screen>
<para>
The service can be enabled, but all &sap; processes need to be
terminated.
</para>
</step>
<step>
<para>
Install the package <package>sapwmp</package>:
</para>
<screen>&prompt.sudo;<command>zypper</command> install sapwmp</screen>
<important xml:id="adm-order-of-packages">
<title>Watch out for order of packages</title>
<para>
The following message should only appear if no &sap; software has been
installed on the system:
</para>
<screen>Warning: sapsys group not found warning: group<!--
--> sapsys does not exist - using root</screen>
<para>
Remove the package <package>sapwmp</package> and install the &sap;
software first before installing <package>sapwmp</package> again.
</para>
<para>
As an alternative, you can fix ownership and permission
<emphasis>after</emphasis> installing the &sap; software with:
</para>
<screen>&prompt.sudo;<command>chgrp</command> sapsys /usr/lib/sapwmp/sapwmp-capture && \
chmod +s /usr/lib/sapwmp/sapwmp-capture</screen>
</important>
<para>
The following message can be ignored:
</para>
<screen>Warning: Found memory controller on v1<!--
--> hierarchy. Make sure unified hierarchy only is<!--
--> used.</screen>
<para>
Switching to unified hierarchy is done in the next step.
</para>
</step>
<step>
<para>
Add <varname>systemd.unified_cgroup_hierarchy=true</varname> to the
kernel command line by adding it to
<varname>GRUB_CMDLINE_LINUX_DEFAULT</varname> in
<filename>/etc/default/grub</filename> like:
</para>
<screen>GRUB_CMDLINE_LINUX_DEFAULT="... systemd.unified_cgroup_hierarchy=true swapaccount=1"</screen>
<para>
With this change, only cgroup2 controllers will be mounted on
<filename>/sys/fs/cgroup</filename>. Cgroup1 controllers, except the
memory controller, are still available and can be used though. Tools
using cgroup1 might not work out of the box any more and might need
reconfiguration. Also, the required mount structure for cgroup1 needs to
be provided.
</para>
<para>
The parameter <varname>swapaccount=1</varname> is not needed for WMP to
work, but it aids the analysis in support cases to show the amount of
swapped out memory for each cgroup.
</para>
</step>
<step>
<para>
Rewrite the GRUB2 configuration:
</para>
<screen>&prompt.sudo;<command>grub2-mkconfig</command> -o /boot/grub2/grub.cfg</screen>
<para>
After reboot (will be done later), the cgroup hierarchy is switched to v2
(unified hierarchy) only.
</para>
</step>
<step xml:id="st-conf-memorylow-sap-slice">
<para>
Configure <varname>MemoryLow</varname> for the
<systemitem>SAP.slice</systemitem>:
</para>
<screen>&prompt.sudo;<command>systemctl</command> set-property SAP.slice MemoryLow=...</screen>
<para>
This command creates a drop-in in
<filename>/etc/systemd/system.control/SAP.slice.d/</filename> to set
<varname>MemoryLow</varname>.
</para>
<para>
The <package>sapwmp</package> package includes the &systemd;
configuration <systemitem>SAP.slice</systemitem> which creates the cgroup
of the same name for the SAP instances. <varname>MemoryLow</varname> is
the &systemd; equivalent of the cgroup parameter
<varname>memory.low</varname> mentioned in the introduction. The value
for <varname>MemoryLow</varname> depends on the type of the &sap;
application and the workload.
</para>
<variablelist>
<varlistentry>
<term>For &hana;</term>
<listitem>
<para>
Since &hana; has a Global Allocation Limit, its value can be used
directly.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>&sap; Application Server (&netweaver;, &s4h;)</term>
<listitem>
<para>
For the Application Server, the sizing for the workload should
indicate the value for <varname>MemoryLow</varname>. The
<package>sapwmp</package> package contains a monitoring part which
might be useful to determine <varname>MemoryLow</varname>. See
<xref linkend="sec-memory-protection-operation-monitor-memory-usage"/>.
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
Keep in mind:
</para>
<itemizedlist>
<listitem>
<para>
All SAP instances on one host are inside the
<systemitem>SAP.slice</systemitem>. <varname>MemoryLow</varname> must
cover the amount of memory of <emphasis>all</emphasis> instances
together on that host. You cannot protect &sap; systems or their
instances from each other.
</para>
</listitem>
<listitem>
<para>
If you are using a database other than &hana;, some database processes
might be part of <systemitem>SAP.slice</systemitem>. Their memory
consumption needs to be taken into account when determining the
<varname>MemoryLow</varname> value.
</para>
</listitem>
<listitem>
<para>
Never choose a value for <varname>MemoryLow</varname> very close to or
larger than your physical memory. System services and additional
installed software require memory too. If they are forced to use swap
too extensively, at the expense of the &sap; application, your system
can become unresponsive.
</para>
</listitem>
</itemizedlist>
<note>
<title>Correctly calculate <varname>MemoryLow</varname> value</title>
<para>
<varname>MemoryLow</varname> takes the memory size in bytes. If the
value is suffixed with K, M, G, or T, the specified memory size is
parsed as Kibibytes, Mebibytes, Gibibytes, or Tebibytes (with the base
1024 instead of 1000, see
<link xlink:href="https://en.wikipedia.org/wiki/Binary_prefix"/>),
respectively. Alternatively, a percentage value may be specified, which
is taken relative to the installed physical memory on the system.
</para>
<para>
The underlying cgroup memory controller will round up the value to a
multiple of the page size. To avoid confusion, set the value for
<varname>MemoryLow</varname> to a multiple of the page size.
</para>
</note>
</step>
<step>
<para>
Create a backup of each &sap; instance profile. Errors in a profile can
prevent a &sap; system from starting.
</para>
</step>
<step>
<para>
For each &sap; instance, add the following line to the instance profile
(usually located in
<filename>/usr/sap/<replaceable>SID</replaceable>/SYS/profile/</filename>)
after the last <literal>Execute_</literal> line:
</para>
<screen>Execute_20 = local /usr/lib/sapwmp/sapwmp-capture -a</screen>
<para>
If necessary, increase the number of the Execute statement so that it is
the highest one, which means that that line is executed last.
</para>
<important>
<title>Editing instance profiles</title>
<para>
Edit the instance profiles directly
<emphasis role="strong">only</emphasis> if you do not have the profiles
imported into the database to manage them via the &sap; GUI (transaction
RZ11). If you have imported them, use the &sap; GUI to add the lines.
Profile files located in the file system are overwritten and any manual
changes would be lost!
</para>
</important>
</step>
</procedure>
<para>
Now the system is ready for a reboot.
</para>
</sect2>
<sect2 xml:id="sec-memory-protection-reboot-and-verification">
<title>Reboot and verification</title>
<remark>toms 2020-09-16: introduction of the procedure is missing.</remark>
<procedure>
<step>
<para>
Reboot the system.
</para>
</step>
<step>
<para>
After rebooting, verify that cgroups v2 has indeed been used:
</para>
<screen>&prompt.root;<command>grep</command> cgroup /proc/mounts
cgroup /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0</screen>
</step>
<step>
<para>
Verify that the cgroup was created successfully and the low memory value
has been set:
</para>
<screen>&prompt.user;<command>systemctl</command> show -p MemoryLow SAP.slice
MemoryLow=18487889920 <- Should be your chosen value (always in bytes)!
# cat /sys/fs/cgroup/SAP.slice/memory.low
18487889920 <- Should be your chosen value!</screen>
<para>
The variable <varname>MemoryLow</varname> can be set to any value, but
the content of the variable is always a multiple of the page size. Keep
this in mind if you notice a slight difference between the values.
</para>
</step>
<step>
<para>
Check that all &sap; instance processes are in the correct system
slices/cgroup.
</para>
<para>
If you have not enabled
<systemitem
class="service">sapinit.service</systemitem>
start the service now. If autostart is not enabled in the instance
profiles, start the instances before you check.
</para>
<para>
Example:
</para>
<screen>&prompt.root;<command>systemd-cgls</command> -a /sys/fs/cgroup/SAP.slice
Directory /sys/fs/cgroup/SAP.slice:
|-wmp-rd91fd6b3ca0d4c1183659ef4f9a092fa.scope
| |-3349 sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
| `-3375 er.sapHA0_ERS10 pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er N...
|-wmp-r360ebfe09bcd4df4873ef69898576199.scope
| |-3572 sapstart pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
| |-3624 dw.sapHA0_D01 pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
...</screen>
<para>
The <systemitem>sapstartsrv</systemitem> process of an instance always
remains in the user slice of
<literal><replaceable>SID</replaceable>adm</literal>. Only the
<systemitem class="process">sapstart</systemitem> process and its
children will be moved to the target cgroup.
</para>
<para>
For each instance, a directory
<filename>wmp-r<replaceable>SCOPEID</replaceable>.scope</filename> exists
with all processes of this instance. The
<replaceable>SCOPEID</replaceable> is a random 128-bit value in
hexadecimal.
</para>
<para>
The &sap; HostAgent is not covered by WMP and remains partly in
<systemitem>sapinit.slice</systemitem> and partly in the user slice of
<systemitem>sapadm</systemitem>.
</para>
</step>
<step>
<para>
If the processes are not in the cgroup, check if the
<literal>Execute</literal> lines in the instance profiles are correct.
Also each instance start should now be logged in the system log
<filename>/var/log/messages</filename>:
</para>
<screen>...
2020-06-16T18:41:28.317233+02:00 server-03 sapwmp-capture: Found PIDs:
2020-06-16T18:41:28.317624+02:00 server-03 sapwmp-capture: 17001
2020-06-16T18:41:28.317813+02:00 server-03 sapwmp-capture: 16994
2020-06-16T18:41:28.317959+02:00 server-03 sapwmp-capture: 16551
2020-06-16T18:41:28.319423+02:00 server-03 sapwmp-capture: Successful capture into SAP.slice/wmp-r07a27e12d7f2491f8ccb9aeb0e080aaa.scope
2020-06-16T18:41:28.319672+02:00 server-03 systemd[1]: Started wmp-r07a27e12d7f2491f8ccb9aeb0e080aaa.scope.
...</screen>
</step>
</procedure>
<para>
To verify the correct setup, run <command>wmp-check</command>. The script
checks the setup of &wmp;:
</para>
<itemizedlist>
<listitem>
<para>
Correct setup of cgroup2.
</para>
</listitem>
<listitem>
<para>
Ownership and permission of the capture program.
</para>
</listitem>
<listitem>
<para>
WMP entries of &sap; instance profiles.
</para>
</listitem>
<listitem>
<para>
Correct cgrop of running &sap; instance processes.
</para>
</listitem>
<listitem>
<para>
Correct setup of <systemitem>SAP.slice</systemitem>.
</para>
</listitem>
<listitem>
<para>
Sane configuration of MemoryLow. However, it cannot determine if the
MemoryLow value has been chosen wisely.
</para>
</listitem>
<listitem>
<para>
Setup of the optional memory sampler.
</para>
</listitem>
<listitem>
<para>
Setup of optional swap accounting.
</para>
</listitem>
</itemizedlist>
<para>
It assumes &sap; instances profiles can be found beneath
<filename>/usr/sap/<replaceable>SID</replaceable>/SYS/profile/</filename>.
</para>
</sect2>
</sect1>
<sect1 xml:id="sec-memory-protection-configuration">
<title>Configuring &wmp;</title>
<para>
To configure WMP, edit <filename>/etc/sapwmp.conf</filename>:
</para>
<screen># NOTE: Local changes may be reverted after update of WMP package. Check for
# .rpmsave file to restore & merge changes.
## Description: Slice unit name where workload is put into
## Type: string
## Default: "SAP.slice"
DEFAULT_SLICE="SAP.slice"
## Description: Comma-separated list of command names to which capture is
## applied (matching against /proc/$PID/stat)
## Type: string
## Default: sapstart
PARENT_COMMANDS=sapstart</screen>
<para>
After any change, restart all &sap; instances.
</para>
<warning>
<para>
Altering <filename>/etc/sapwmp.conf</filename> should not be necessary. Do
not do it until you know exactly what you are doing!
</para>
</warning>
</sect1>
<sect1 xml:id="sec-memory-protection-operation-change-the-value-of-memorylow">
<title>Changing the value of MemoryLow</title>
<para>
To change the value of <varname>MemoryLow</varname> run:
</para>
<screen>&prompt.root;<command>systemctl</command> set-property SAP.slice MemoryLow=...</screen>
<para>
The changes will take effect immediately.
</para>
<para>
The underlying cgroup memory controller will round up the value to a
multiple of the page size. To avoid confusion, set the value of
<varname>MemoryLow</varname> to a multiple of the page size.
</para>
<important>
<title>Value of <varname>MemoryLow</varname></title>
<para>
Never set <varname>MemoryLow</varname> to a value lower than the memory
already accounted in <systemitem>SAP.slice</systemitem>. To check, run:
</para>
<screen>&prompt.root;<command>systemctl</command> show -p MemoryCurrent SAP.slice</screen>
</important>
</sect1>
<sect1 xml:id="sec-memory-protection-operation-monitor-memory-usage">
<title>Monitoring memory usage</title>
<para>
Logging the memory usage can be necessary not only to determine the value
for <varname>memory.low</varname>, but also for monitoring the correct
operation of WMP.
</para>
<para>
To enable monitoring, activate the shipped timer unit:
</para>
<screen>&prompt.root;<command>systemctl</command> enable --now wmp-sample-memory.timer</screen>
<para>
Now the timer should be listed by <command>systemctl list-timers</command>:
</para>
<screen>&prompt.root;<command>systemctl</command> list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
...
Tue... 9min left Tue... 4s ago wmp-sample-memory.timer wmp-sample-memory.service
...</screen>
<para>
If you check the current configuration, you can see that memory data is
collected every 10 minutes with a randomized delay of
three minutes:
</para>
<screen>&prompt.root;<command>systemctl</command> cat wmp-sample-memory.timer
# /usr/lib/systemd/system/wmp-sample-memory.timer
[Unit]
Description=WMP periodic log of memory consumption
[Timer]
OnCalendar=*:0/10
RandomizedDelaySec=180
AccuracySec=60
[Install]
WantedBy=timers.target</screen>
<para>
To change this, create a drop-in file and reload &systemd; (for example, by
increasing the interval to 30 minutes):
</para>
<screen>&prompt.root;<command>mkdir</command> /etc/systemd/system/wmp-sample-memory.timer.d
# cat <<EOF >/etc/systemd/system/wmp-sample-memory.timer.d/override.conf
[Timer]
OnCalendar=
OnCalendar=*:0/30
EOF
# systemctl daemon-reload</screen>
<para>
<emphasis>(The first OnCalendar= line is important for deleting previously
defined OnCalendar= settings.)</emphasis>
</para>
<para>
To see the memory consumption, check the system log for lines written by
<literal>wmp_memory_current</literal>:
</para>
<screen>&prompt.root;<command>grep</command> wmp_memory_current /var/log/messages
...
2020-09-14T12:02:40.337266+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294059008 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499219968 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8364032 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1863335936 memory.swap.current=0
2020-09-14T12:03:00.767838+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294022144 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499473920 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8364032 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1862586368 memory.swap.current=0
2020-09-14T12:04:00.337315+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294022144 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499207680 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8355840 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1862746112 memory.swap.current=0
...</screen>
<para>
Here is a reformatted log line to get a better impression:
</para>
<screen>2020-09-14T12:02:40.337266+02:00 server-03 wmp_memory_current:
SAP.slice : memory.low=21474836480 memory.current=2294059008 memory.swap.current=0 ,
user.slice : memory.low=0 memory.current=5499219968 memory.swap.current=0 ,
init.scope : memory.low=0 memory.current=8364032 memory.swap.current=0 ,
system.slice : memory.low=0 memory.current=1863335936 memory.swap.current=0</screen>
<para>
For each cgroup directly below <filename>/sys/fs/cgroup/</filename> one
comma-separated block exists. On a normal system, you should find at least
<systemitem>user.slice</systemitem>, <systemitem>system.slice</systemitem>,
and <systemitem>init.scope</systemitem>. WMP adds
<systemitem>SAP.slice</systemitem>.
</para>
<para>
Each block contains the information about the current value of
<varname>memory.low</varname> and <varname>memory.current</varname>, and the
currently allocated amount of physical memory of processes in this cgroup.
</para>
<para>
If you enabled swap accounting (<varname>swapaccount=1</varname>) during
setup, you also have <varname>memory.swap.current</varname>, the amount of
swapped-out memory of the cgroup.
</para>
<para>
All values are in bytes. See <xref linkend="st-conf-memorylow-sap-slice"/>
in <xref linkend="sec-memory-protection-setup-preparation"/>.
</para>
<tip>
<title>Script for printing</title>
<para>
You can find a script to print the information as table or CSV here:
<link xlink:href="https://github.com/scmschmidt/wmp_log_extract"/>
</para>
</tip>
</sect1>
<sect1 xml:id="sec-memory-protection-verify-correct-operation">
<title>Verifying correct operation</title>
<para>
Besides monitoring memory consumption and swapping (see
<xref linkend="sec-memory-protection-operation-monitor-memory-usage"/>), you
should also regularly check that all &sap; instance processes are in their
scopes below <systemitem>SAP.slice</systemitem>.
</para>
<para>
To do so, run <command>systemd-cgls</command> and check each instance
process.
</para>
<para>
Example:
</para>
<screen>&prompt.root;<command>systemd-cgls</command> -a /sys/fs/cgroup/SAP.slice
Directory /sys/fs/cgroup/SAP.slice:
|-wmp-rd91fd6b3ca0d4c1183659ef4f9a092fa.scope
| |-3349 sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
| `-3375 er.sapHA0_ERS10 pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er N...
|-wmp-r360ebfe09bcd4df4873ef69898576199.scope
| |-3572 sapstart pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
| |-3624 dw.sapHA0_D01 pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
...</screen>
<para>
A simpler test would be to list all processes, including cgroups, for all
<literal>SID</literal>s used on the system.
</para>
<para>
Example:
</para>
<screen>&prompt.user;<command>ps</command> -eo user,pid,cgroup:60,args | <command>grep</command> -e [h]a0adm
ha0adm 2062 0::/user.slice/user-1001.slice/user@1001.service/init.scope /usr/lib/systemd/systemd --user
ha0adm 2065 0::/user.slice/user-1001.slice/user@1001.service/init.scope (sd-pam)
ha0adm 3081 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope sapstart pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm 3133 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope ms.sapHA0_ASCS00 pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm 3134 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope en.sapHA0_ASCS00 pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm 3327 0::/SAP.slice/wmp-ra42489517eb846c282c57681e627a496.scope sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
...</screen>
<para>
All instance processes except <systemitem>sapstartsrv</systemitem> need to
be in a scope below <systemitem>0::/SAP.slice/</systemitem>.
</para>
<para>
To verify the correct setup, use the <command>wmp-check</command> tool. See
<xref linkend="sec-memory-protection-reboot-and-verification"/> for more
details.
</para>
</sect1>
<sect1 xml:id="sec-memory-protection-deinstallation">
<title>Uninstalling &wmp;</title>
<procedure>
<step>
<para>
Stop the &sap; system completely. The
<systemitem class="service">sapinit.service</systemitem> has to be
stopped, but can stay enabled. All &sap; processes have to be terminated.
</para>
</step>
<step>
<para>
Remove any changes made to <systemitem>SAP.slice</systemitem>, such as
setting <varname>MemoryLow</varname>:
</para>
<screen>&prompt.root;<command>systemctl</command> revert SAP.slice</screen>
</step>
<step performance="optional">
<para>
Remove the package <package>sapwmp</package>:
</para>
<screen>&prompt.root;<command>zypper</command> remove sapwmp</screen>
<para>
This step is optional. The package can stay on the system without having
an influence.
</para>
</step>
<step performance="optional">
<para>
Remove <varname>systemd.unified_cgroup_hierarchy=true</varname> from
<varname>GRUB_CMDLINE_LINUX_DEFAULT</varname> in
<filename>/etc/default/grub</filename>.
</para>
<para>
This step is optional. You can keep cgroup2 without using WMP.
</para>
</step>
<step>
<para>
Rewrite the GRUB2 configuration:
</para>
<screen>&prompt.root;<command>grub2-mkconfig</command> -o /boot/grub2/grub.cfg</screen>
<para>
After the next boot, the system is switched back to the hybrid cgroup
hierarchy.
</para>
</step>
<step>
<para>
Remove the line to call sapwmp-capture from each &sap; instance profile
(usually located in
<filename>/usr/sap/<replaceable>SID</replaceable>/SYS/profile/</filename>):
</para>
<screen>Execute_20 = local /usr/lib/sapwmp/sapwmp-capture -a</screen>
<important>
<title>Backup is necessary</title>
<para>
Before editing an instance profile, create a backup! Errors in a profile
can prevent an &sap; system from starting!
</para>
</important>
<important>
<title>About editing profiles directly</title>
<para>
Edit the instance profiles directly
<emphasis role="strong">only</emphasis> if you have not imported the
profiles into the database to manage them via the &sap; GUI (transaction
RZ11). If you have imported them, use the &sap; GUI to add the lines.
Profile files located in the file system are overwritten, and any manual
changes will be lost!
</para>
</important>
</step>
<step>
<para>
Reboot the system and verify that your &sap; system has been started
successfully.
</para>
</step>
</procedure>
</sect1>
<sect1 xml:id="sec-sap-systemd-mix-wmp">
<title>Systems running both &systemd;-based and regular SAP instances</title>
<para>
With SAP kernel version 788 and certain specified patch levels of older
kernel releases (722, 753, 777, 781, 785), the SAP Start Service supports
&systemd;. Such a &systemd;-based instance comes with its own &systemd;
service that the SAP Start Service places into its own cgroup (<systemitem
class="service">SAP<replaceable>SID_NR</replaceable>.service</systemitem>)
under <systemitem class="service">SAP.slice</systemitem>. This affects the
&wmp; setup. For &systemd;-based SAP instances, the following instructions
apply:
</para>
<itemizedlist>
<listitem>
<para>
Skip the step that adds the capture program to the instance profile. If
you are migrating an instance to support &systemd;, remove this entry from
the profile. Keep in mind that &systemd;-based instances are always put
into a cgroup under <systemitem class="service">SAP.slice</systemitem> and
therefore become part of &wmp; protection.
</para>
</listitem>
<listitem>
<para>
Set <option>MemoryLow=infinity</option> for the
<systemitem class="service">SAP<replaceable>SID_NR</replaceable>.service</systemitem>
to make the protection work correctly. Example for
<systemitem class="service">SAPNW1_01.service</systemitem>:
</para>
<screen>sudo systemctl set-property SAPNW1_01.service MemoryLow=infinity</screen>
</listitem>
<listitem>
<para>
Set <option>MemoryLow=infinity</option> for the
<systemitem class="service">saphostagent.service</systemitem> to make the
protection work correctly
</para>
<screen>sudo systemctl set-property saphostagent.service MemoryLow=infinity</screen>
</listitem>
</itemizedlist>
<para>
The following example demonstrates a mixed environment. The SAP Host Agent
and instance 01 are &systemd;-based, instance 00 is not. Both instances are
under <systemitem class="service">SAP.slice</systemitem> either in the
cgroup managed by the SAP Start Service
(<systemitem class="service">SAPNW1_01.service</systemitem>) or &wmp;
(<literal>wmp-rece5b7fa372e4619a9623e120aa23a23.scope</literal>).
<option>MemoryLow=</option> for
<systemitem class="service">SAP.slice</systemitem> has been set as well as
<option>MemoryLow=infinity</option> for all the cgroups below.
</para>
<screen>&prompt.user; systemd-cgls
Control group /:
-.slice
├─SAP.slice
│ ├─SAPNW1_01.service
│ │ ├─ 887 /usr/sap/NW1/ASCS01/exe/sapstartsrv pf=/usr/sap/NW1/SYS/profile/NW1_ASCS01_systemdproj
│ │ ├─2838 sapstart pf=/usr/sap/NW1/SYS/profile/NW1_ASCS01_systemdproj
│ │ ├─2895 ms.sapNW1_ASCS01 pf=/usr/sap/NW1/SYS/profile/NW1_ASCS01_systemdproj
│ │ └─2896 en.sapNW1_ASCS01 pf=/usr/sap/NW1/SYS/profile/NW1_ASCS01_systemdproj
│ ├─saphostagent.service
│ │ ├─ 900 /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile -nodaemon -trace
│ │ ├─ 984 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile
│ │ └─2428 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile
│ └─wmp-rece5b7fa372e4619a9623e120aa23a23.scope
│ ├─5522 sapstart pf=/usr/sap/NW1/SYS/profile/NW1_D00_systemdproj
│ ├─7824 dw.sapNW1_D00 pf=/usr/sap/NW1/SYS/profile/NW1_D00_systemdproj
│ ├─7825 ig.sapNW1_D00 -mode=profile pf=/usr/sap/NW1/SYS/profile/NW1_D00_systemdproj
...
&prompt.user; systemctl show -p MemoryLow SAP.slice
MemoryLow=24584065024
&prompt.user; systemctl show -p MemoryLow wmp-rece5b7fa372e4619a9623e120aa23a23.scope
MemoryLow=infinity
&prompt.user; systemctl show -p MemoryLow SAPNW1_01.service
MemoryLow=infinity
&prompt.user; systemctl show -p MemoryLow saphostagent.service
MemoryLow=infinity</screen>
<para>
Further information about the &systemd; integration is available at
<link xlink:href="https://launchpad.support.sap.com/#/notes/3139184">SAP
Note 139184 - Linux: &systemd; integration for sapstartsrv and SAP Host
Agent</link>.
</para>
</sect1>
<sect1 xml:id="sec-sap-systemd-wmp">
<title>Systems running only &systemd;-based instances</title>
<note>
<para>
SUSE's HA cluster solutions are supported for fully &systemd;-based SAP
systems.
</para>
</note>
<para>
A SAP system consisting of &systemd;-based instances can only rely on
cgroup2 for memory protection.
</para>
<para>
The instances are placed into a dedicated cgroup (default is
<systemitem class="service">SAP.slice</systemitem>) by the SAP Start
Service. Therefore it is only necessary to switch to the unified cgroup2
hierarchy and set <option>MemoryLow=</option> correctly.
</para>
<para>
If &wmp; is already configured, follow the instructions in
<xref linkend="sec-sap-systemd-mix-wmp" />. Afterwards, you can remove the
package <package>sapwmp</package> from the system, unless you want to use
the <systemitem class="service">wmp-sample-memory.timer</systemitem> and
<systemitem class="service">wmp-sample-memory.service</systemitem> to
monitor memory usage.
</para>
<para>
To configure the memory protection for a system consisting of only
&systemd;-based SAP instances, perform the following steps:
</para>
<itemizedlist>
<listitem>
<para>
Stop all SAP instances and the SAP Host Agent.
</para>
</listitem>
<listitem>
<para>
Switch to the unified cgroup hierarchy, rewrite the boot loader (step 4
and 5 in <xref linkend="sec-memory-protection-setup-preparation" />).
</para>
</listitem>
<listitem>
<para>
Set <option>MemoryLow=</option> for
<systemitem class="service">SAP.slice</systemitem> (step 6 in
<xref linkend="sec-memory-protection-setup-preparation" />).
</para>
</listitem>
<listitem>
<para>
Set <option>MemoryLow=infinity</option> for
<systemitem class="service">saphostagent.service</systemitem> and for the
services of all SAP instances.
</para>
</listitem>
<listitem>
<para>
Reboot the system.
</para>
</listitem>
</itemizedlist>