aarch64 manylinux wheel and wrong pagesize #16677
Comments
|
Thanks for the report. I don't know what we can do about it unless there is a compiler flag we should using. Opening an issue in the manylinux project would be a good thing to do in any case. |
|
Can you provide more specific information about the processor/OS you have? |
This issue was noticed on RHEL8/AArch64 which uses 64 KiB page size. Fedora/AArch64 and Ubuntu/AArch64 use 4 KiB page size. To debug the issue, you can get your page size using: or: See also the getpagesize() function. The page size is part of the toolchain ABI. On my x86-64, the value is provided by the dynamic linker (ld.so). It cannot by changed at runtime.
It's more a linker issue rather than a C compiler issue. Note: If LOAD sections are aligned to 64 KiB, numpy will remain compatible with 4 KiB pages. See also a similar issue in Julia which chose to use the |
Thanks for the detailed info. Is this ever a problem on ppc64 or x86? If it can be fixed with |
|
It seems like only AArch64 arch is affected. On ppc64, the page size is always 64 KiB. On x86 and x86-64, I think that 4 KiB is always fine. |
|
Thank you @mattip for reporting this to auditwheel! |
|
@mattip reported pypa/auditwheel#251 to auditwheel. |
|
@frenzymadness could you confirm that running |
|
Unfortunately, it seems that It seems that the file stays untouched. |
Which command did you try? |
|
I'm talking in generalities here, and as a toolchain developer I wanted to comment on the general concepts that impact this issue. You cannot change the alignment of a PT_LOAD segment to be larger than required by the segment. If the kernel is using larger page sizes then the kernel will only have protections on larger page sized boundaries, while the object may need such protection on smaller page boundaries e.g. two pages may be too close together with different protections (and must stay that way to allow constant offsets in generated code to work). Therefore the dynamic loader explicitly checks that the PT_LOAD segment alignment requirement and fails the to load the segment if the requirements are not met. There is also the matter of the address congruence relationship required for the mappings. The offset must be congruent to the address modulo the alignment. If you increase the alignment the offset and address are no longer congruent. For example, take the last PT_LOAD which is offset 0x3fb000 virtual address 0x3e1000 and alignment 0x1000. The "offset - virtual address" is divisible by the alignment. If you raise the alignment to 0x10000, then 0x1a000 (the difference) is no longer divisible by the alignment. This congruence is critical to offsets, alignments, and the algorithms employed the static and dynamic loader. In summary:
|
|
My takeaway from that is the problem must be solved at compile time. The docker image used to build the manylinux2014 wheels is based on "arm64v8/centos:7". I wonder why they have 4k pages when redhat8 has 64k pages. |
|
We should try adding flags to the nightly wheels. |
|
Page size on aarch64 can be 4/16/64k... |
If the linker is configured for 64k page size, the binary will work on any other smaller page size. |
|
If memory serves there was a bug with gold and alignment https://bugzilla.redhat.com/show_bug.cgi?id=1225156 Maybe manylinux2014 has this issue? |
|
One solution would be to use CentOS 8 as OS on CI instead of Ubuntu (which is on 4K page size). |
|
@hrw The problem is not CI it is in building wheels for relase. When building manylinux wheels, the environment is very tightly controlled. The manylinux2014 standard, laid out in PEP 599 must use a base image based on CentOS 7. Adopting PEP 600, which would allow us to use a more advanced base OS, has stalled, here is the tracking issue. |
The page size will come from the underlying system, not the container. So if you build on top of CentOS 7/8 as the base platform you should get things 64k aligned by default and those wheels would be compatible. I feel like we should run a few ideas around and see if we can find a default fix for the container; because explaining to everyone who creates a arm64 wheel using the standard container on travis that they have built something by default broken on the very reference platform it's building for is a hard sell :) I've raised pypa/manylinux#735 which is probably a good place to talk about it |
|
So the issue is coming from Manylinux2014_aarch64 and its version of patchelf being slightly too old. NixOS/patchelf@0470d69 |
|
Thanks. Let's continue the discussion on the manylinux issue, since I think we have localized the problem to patchelf breaking binaries that are properly compiled with 64k pages, and patchelf is part of manylinux. |
|
manylinux2014 now has patchelf 0.12 - pypa/manylinux#741 |
|
@mattip Do I need to link in different OpenBLAS libraries for the 1.19.2 release to get this fix? |
|
No. The fix is an adjustment to the |
It would be great to have a possibility to test new wheels before you upload them to PyPI to confirm that they are fixed. Because if the problem will still be there, we would have to wait for the next release because the wheels cannot be reuploaded. |
|
That's what https://anaconda.org/multibuild-wheels-staging/numpy is for. |
|
You can always re-upload the wheels with a new build tag : https://github.com/MacPython/wiki/wiki/Build-Tags |
|
I've tested the latest numpy wheels for Python 3.6 and 3.8 on aarch64 machine and everything seems to work well. LOAD sections also seem to be aligned correctly. Thank you. |
Hello.
It seems that numpy-1.19.0-cp38-cp38-manylinux2014_aarch64.whl is built with 4kB pagesize which means that when I install it and import numpy on a system with 64kB, I get:
According to readelf, the _multiarray_umath.cpython-38-aarch64-linux-gnu.so extension has 4 LOAD sections. The first two are aligned on 64 KiB (0x10000), the two last are aligned on 4 KiB (0x1000).
Is there anything you can do about it or should I take a look on recent changes in manylinux project?
Thank you.
The text was updated successfully, but these errors were encountered: