Implement manage_rlimit_as; add PTL test #2332
Conversation
alexis-cousein
force-pushed
the
cgroups_rlimit_as
branch
from
f9a771a
to
50772d3
Compare
|
Test also passed on Test Harness, on CentOS7, CentOS8 and SLES15. 6752 | Custom | Tests from given sources on platforms CENTOS7, CENTOS8, SUSE15 | alexis-cousein/pbspro@cgroups_rlimit_as | 26th Mar 2021 05:35:50.619 PM +01:00 | 00:07:24.458 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 3 When rerunning all cgroup tests I see there's still a race condition in the test to see if stuck nodes are correctly offlined, because the job that should get stuck can still be deleted before the processes are frozen. I'm going to implement a more robust test to confirm the freezer has frozen the processes (instead of "if I sleep 1 second it should be OK"). Stay tuned for another commit. |
|
It will also be necessary to rewrite test_cgroup_enforce_memsw, since the 'success' criterion depends on the bug worked around by manage_rlimit_as! It currently tests on whether Python was stopped because it ran out of address space, rather than because the memsw limit was enforce. |
|
Since the Test Harness has no swap and will not run it, the relevant output for the fixed test_cgroup_enforce_memsw test on my system: |
|
Note a cgroup hook fix was also necessary: for joined stderr/stdout, if the cgroup limit detection happens only in the execjob_epilogue, it is too late to append to stderr (it does not make it to the joined file), so we need to write in stdout if we detect Join_Path set to 'oe'. That was indeed a valid 'test failure', because users who use this feature would otherwise never see the cgroup limit exceeded messages in what is supposed to be a joined stdout/stderr. The dmesg searcher was also extended to support more recent kernels, which no longer have lines matching the old parser; for more recent kernels the lines added to stderr (or the joined stdout/stderr) are in the form: As before, they identify the limit for which failcnt was non-zero (here both limits were exceeded) , the cgroup (which validates it's about the correct job) and the victim selected by the OOM killer. More information can be found by site admins in dmesg output. |
|
The sibling test test_cgroup_enforce_memory is also incorrect: it uses a job that sets only a mem limit, and then uses a configuration file that enables "memory" and disables "memsw", leaving "swappiness" to the default 1. That will not make the job exceed the memory limit in all cases: if the machine has swap, once the job reaches its memory limit, it is allowed to swap out pages to swap (and since memsw is disabled there is no limit to it). On some kernels even when swap is depleted there may not even be a failcount set on the memory cgroup when the OOM killer wakes up because the swap is depleted -- so the job may get killed because of a "general" lack of swap, rather than a cgroup limit violation in the specific cgroup. To avoid this behaviour, you can either enable memsw and set a vmem limit, or set swappiness to 0 in the configuration file. Since
the latter is preferable, and was implemented by setting up a slightly different configuration file. |
|
All tests that run without swap passed on CentOS7, CentOS8 and SLES15 on the test harness: 6763 | cgroups_cousein | Tests from given sources on platforms CENTOS7, SUSE15, CENTOS8 | alexis-cousein/pbspro@cgroups_rlimit_as | 27th Mar 2021 03:09:14.645 AM +01:00 | 00:32:36.812 | 159 | 0 | 0 | 0 | 0 | 0 | 39 | 120 159 run, 39 skipped (for lack of swap or a number of nodes that is too low), 120 successes. All tests also pass on my machine which has swap enabled. |
|
Fixing a few other problems: |
alexis-cousein
force-pushed
the
cgroups_rlimit_as
branch
from
645c5cc
to
747e518
Compare
alexis-cousein
force-pushed
the
cgroups_rlimit_as
branch
from
747e518
to
2ab345d
Compare
|
All tests passed or (correctly) skipped on test harness; now also on Ubunto 20.04: |
alexis-cousein
force-pushed
the
cgroups_rlimit_as
branch
from
680d8dc
to
09ba4ed
Compare
alexis-cousein
force-pushed
the
cgroups_rlimit_as
branch
from
09ba4ed
to
299e714
Compare
|
There were a number of issues with exception handling in the hook and in the tests. The tests weren't correctly skipping some tests when self.swapctl was set to 'false' (because it's set to a string you can't use 'if not self.swapctl'), on nodes that have no kernel support for swap accounting (aka memsw). Corrected that in the tests -- and when detecting memsw kernel support it will now log an info message in all case--. But I also made sure that the hook would not fail with incomprehensible exceptions when memsw was enabled but kernel support for it was missing. I added guards around all the messages encountered, reinstalled some sanity checks for vmem_avail and vmem_default, and fixed one indentation issue when we were looking up memory.memsw.limit_in_bytes that would make the periodic hook fail when hosts have no swap accounting. size_to_int was also fixed to return None instead of generating an incomprehensible exception when one passes something that cannot be a pbs size (e.g. a negative number, which was caused by not being able to read the memory.memsw.limit_in_bytes file). Finally, the cgroup hook was modified to neuter this "impossible" configuration: if memsw is enabled in the config file but kernel support is lacking, memsw will be disabled and every hook event will print three messages in the logs at EVENT_ERROR level starting with "CONFIG ERROR:" telling the site something needs to be fixed. one test was forcibly creating two vnodes on all hosts, named hostname[0] and hostname[1], but was also doing so when there is only one socket. Needless to say, configure_job does not take kindly to a request by the scheduler to assign memory on "hostname[1]", which it takes to be a request to assign it to the non-existing socket 1. The hook did crash with an ungainly exception (a KeyError on 1), so I modified the hook to fail with "Failed to assign resources" and a specific error message in the log. The test, though, needs to be rewritten to let the cgroup hook create vnodes and work with what it gets, instead of messing with the vnodes that are supposed to be managed by the cgroup hook. It now sets vnode_per_numa_node, lets MoM create the extra vnodes, checks for the presence of a vnode for socket [1] (and skips if not present), tallies the total memory available on all vnodes of the first host, and then creates a job that uses 1 CPU and every memory available except the two last 2mb and that forces the host to match the first MoM host; it then attempts to run a job that uses 1024mb more (to ensure that the first job run was not successful because memory requests are ignored), which should fail because there is not sufficient memory on the requested host. The test success criterion was also changed; it was looking for a "Can Never Run" message for insufficient mem, but if there is more than one node the message can also start with "Not Running" instead of that string ('Can Never Run' is printed when the entire complex does not have enough memory, not when the requested host does not), so it now just matches the "insufficient resource" part of the string. Finally, some tests were printing a value without first asserting that the value is valid, resulting in an unreadable exception rather than the failure with the human readable assertion message. That was fixed: these tests now first assert the value is valid and only then print an informational message using them. |
|
Also changed some tests that will be problematic on hosts that are not "real" VMs or hosts but containers. Some tests were using sync(), which if unfiltered is going to hang until the entire host kernel has flushed its buffer cache (with very uncertain timings, and creating a process that is unkillable while the sync is in progress and may mess up test cleanup). I replaced it with a command to echo 3 to /proc/sys/vm/drop_caches, which on hosts and VMs will ask to flush the caches (but in contrast with sync will not wait around and will not create an unkillable process), on privileged containers will ask to flush the host caches, and on unprivileged containers will simply return an error (as /proc is not writeable on these containers). |
|
The drop_caches turns out to be a bad idea -- all it does on recent kernels is mark clean pages free, but if the tunables of the host OS are sane (i.e. vm.min_free_kbytes is not too low) that's pointless -- it's the dirty pages that are hard to get rid of.. |
|
Tests on test harness on all the OSes I could find all either passed or skipped. Description: Rerun All Tests From #6905
|
|
Thanks for posting test results, looks good. |
|
the rlimit_as test was not robust to hosts that add extra lines of spam in job outputs; the test was changed to search for the required output in all lines of the output. New test results on the Test Harness: Description: Tests from given sources on platforms CENTOS7, SUSE15, CENTOS8, SLES12, SLES15, UBUNTU1604, UBUNTU1804, UBUNTU2004
|
|
@nishiya : All the tests are passing now after the latest commit from Alexis. I'll wait until you've seen the latest commit before I merge this in. |
|
Merged #2332 into master. |
Describe Bug or Feature
When enabling the "memsw" functionality in the cgroup hook, the semantics of "vmem" change.
Without it "vmem" is managed by MoM and denotes the sum of the address spaces used by all processes on the mother superior node, and MoM sets a per process limit for RLIMIT_AS (since clearly a single process cannot use less address space than the sum for all processes).
But when the cgroup hook "memsw" functionality is enabled, "vmem" requests instead specify the sum of physical memory plus swap usage for the job, which is often smaller than the address space limit.
Indeed, it is doubtful that in the face of kernel limits on real memory plus swap usage anyone would care about the address space limits, especially on 64-bit systems (which have an 8 Exabyte positive address address space). Address space limits were used in the past when there were no physical memory and swap limits enforced by the kernel, but cgroups make these unnecessary.
But on most current versions of PBSPro, the MoM does not know about this change in semantics for vmem, and it still sets a RLIMIT_AS limit for all processes in the job. Since many applications allocate much more address space than they actually use memory, that can make applications fail even though they are staying well within the memory+swap usage corresponding to their vmem requests.
Even MoM itself can be a victim for small vmem limits: when the job starter child forks to run the execjob_launch hook, the fork and exec of pbs_python can make it use up to 150mb-200mb of address space, so setting a small RLIMIT_AS limit can make either the job starter or the hooks fail.
Furthermore, a per-job 'pvmem' resource request unambiguously requests exactly a RLIMIT_AS, so there is no real need to use vmem to specify this limit if one is desired.
To summarise: when enabling the "memsw" functionality in the cgroup hook, RLIMIT_AS should be unlimited, unless the job also requests "pvmem" (which is an unambiguous request to set RLIMIT_AS).
Note: this really is a MoM bug; ideally it should be possible to enable or disable automatic setting of the RLIMIT_AS limit when 'vmem' is requested in the MoM config file. But until such a mechanism exists, at least the cgroup hook can mitigate the effect for values of 'vmem' large enough to allow the job starter and the hooks to run.
Describe Your Change
A new parameter is introduced to the cgroup configuration file, manage_rlimit_as, which by default is true.
If enabled, then the cgroup hook will reset the RLIMIT_AS process limit for task processes to either unlimited (if pvmem is not specified) or the value of pvmem requested for the job.
This functionality requires a kernel that supports the prlimit system call (i.e. Linux kernel 2.6.36 and above).
If hooks use Python 3, i.e. for PBS versions 2020 and above, that is the only requirement.
For older versions of PBS, the 'prlimit' command should be present. This command is available in util-linux versions 2.21 and above on:
-CentOS7/RedHat from version 7.0
-SLES from version 12 onward,
-Ubuntu from version Ubuntu 16.04 onward
-Debian versions from version 8.0 onward
(i.e. most OS versions with a kernel that supports the prlimit system call and have a version of util-linux >= 2.21).
If the support for changing process limits of other processes is not present, then limits set by MoM are never changed (i.e. the hook behaves as before, and as if manage_rlimit_as were disabled).
When the flag is disabled, limits set by MoM will not be changed. It is unlikely anyone would want this behaviour, except on MoMs where it is possible to disable enforcement of 'vmem' as an RLIMIT_AS limit set on processes.
Link to Design Doc
https://openpbs.atlassian.net/wiki/spaces/PD/pages/2668199965/Fix+incorrect+MoM+RLIMIT+AS+when+vmem+is+used+for+the+group+hook+s+memsw+functionality
Attach Test and Valgrind Logs/Output
A PTL test was added, test_manage_rlimit_as.
A job requests -lselect=1:ncpus=0:mem=300mb:vmem=300mb and the test checks that ulimit -v in a job returns 'unlimited'. A job requests -lselect=1:ncpus=0:mem=300mb:vmem=300mb,pvmem=300mb, and then ulimit -v should return 307200 (since ulimit -v output is in kb units).
Relevant portion of test_manage_rlimit_as output: