gh-117755: Fix mimalloc for huge allocation on s390x

[vstinner]

Fix mimalloc allocator for huge memory allocation (around 8,589,934,592 GiB) on s390x.

• Abort allocation early in mimalloc if the number of slices doesn't fit into uint32_t, to prevent a integer overflow (cast 64-bit size_t to uint32_t).
• Add test_large_alloc() to test_bigaddrspace.
• Reenable test_maxcontext_exact_arith() of test_decimal on s390x.

• Issue: Python 3.13.0a6 freethreading on s390x: test.test_io.CBufferedReaderTest.test_constructor crash with Floating point exception #117755

[vstinner]

cc @brandtbucher @colesbury @corona10

See #117755 (comment) for the rationale for this fix.

The current https://github.com/microsoft/mimalloc upstream doesn't seem to have the concept of "slices" in mi_page_t anymore. I don't know if it's still affected by issue gh-117755 bug, but this change includes a regression test in test_bigaddrspace so we will catch such bug quicker. I cannot report the issue to mimalloc upstream since the code changes: there are no "slices" anymore (upstream is not affected by this exact bug anymore).

[colesbury]

The current https://github.com/microsoft/mimalloc upstream doesn't seem to have the concept of "slices" in mi_page_t anymore.

They are still there. The branches are a bit confusing. dev-slice is the development branch for mimalloc 2.x.

[colesbury]

We should report and fix the uint32_t overflow, but I don't think we should be adding more tests that try to allocate huge amounts of memory. Especially with Linux's OOM killer, we are only setting ourselves up for more problems

[colesbury]

I filed an issue upstream: microsoft/mimalloc#876

[vstinner]

For now, I skipped the 3 test_io tests on s390x impacted by the bug: #117801 (merged). I rebased this PR on top of it to reenable these 3 test_io tests.

[vstinner]

I filed an issue upstream: microsoft/mimalloc#876

I proposed a fix upstream: microsoft/mimalloc#877

[vstinner]

@colesbury: Would it be acceptable to have this downstream change for now?

I suppose the mimalloc upstream cares less about Linux s390x than Python (which has Linux s390x CIs: buildbots), and it may take a few weeks to get the fix merged upstream. At least, I already proposed the fix upstream.

[colesbury]

The mimalloc changes look good to me.

I'm not thrilled with the test_large_alloc test. If mimalloc fixes the bug by making the slice count 64-bits, then we'll run into problems. It strikes me as another problematic test that makes too many assumptions and won't provide enough value in terms of catching actual bugs.

I feel the same way about the test_constructor tests. We are skipping these tests on sanitizers, 32-bit builds, and previously on s390x. That's a sign that they are unreliable tests. We should get rid of them! Unreliable tests are worse than no test.

[vstinner]

I'm not thrilled with the test_large_alloc test. If mimalloc fixes the bug by making the slice count 64-bits, then we'll run into problems.

Let me try with uint64_t... :-) In short, the Linux s390x kernel doesn't kill the process allocating an insane amount of memory. I didn't check if the process is killed later with Out Of Memory (OOM) since I tested on a shared machine.

The kernel is fine with a process allocating 822368422110644 kB of data. The strange part is "VmPTE: 26876 kB": I expected way more. I don't know which page size is used for the large allocation.

Command:

gdb -args ./python -m test test_bigaddrspace -u all -v

gdb:

(gdb) py-bt
Traceback (most recent call first):
  File "/root/vstinner/cpython/Lib/test/test_bigaddrspace.py", line 99, in allocate
    return b'x' * length
  File "/root/vstinner/cpython/Lib/test/test_bigaddrspace.py", line 113, in test_large_alloc
    allocate(size)
(...)

(gdb) where
#0  0x000003fffdaa38f0 in __memset_mvcle () from /lib64/libc.so.6
#1  0x00000000010ce708 in _PyBytes_Repeat (
    dest=0x5bf70010030 'x' <repeats 200 times>..., 
    len_dest=842105263157894695, src=0x5bf31174330 "x", len_src=1)
    at Objects/bytesobject.c:3640
#2  0x00000000010c9e3a in bytes_repeat (a=0x5bf31174300, n=842105263157894695)
    at Objects/bytesobject.c:1484
(...)

(gdb) frame 1
(gdb) p len_dest
$1 = 842105263157894695
(gdb) p/x len_dest
$2 = 0xbafc24672035e27

Process:

# grep ^Vm /proc/2070842/status
VmPeak:	822368422334080 kB
VmSize:	822368422334080 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	 7319628 kB
VmRSS:	 7086188 kB
VmData:	822368422110644 kB
VmStk:	     388 kB
VmExe:	    6576 kB
VmLib:	    3624 kB
VmPTE:	   26876 kB
VmSwap:	 6529488 kB

# free -m
              total        used        free      shared  buff/cache   available
Mem:           7782        7355         102         183         324         129
Swap:          8167        6723        1444

The quantity of Swap is growing slowly... I stopped the process before most Swap was used...

[vstinner]

I'm fine with removing these tests: I created PR gh-117938 for that.

[vstinner]

I modified this PR to restrict it to the Objects/mimalloc/segment.c change and leave tests unchanged.

[colesbury]

Great! I've also added the PR to my tracking issue in #113141