VADER: Replacing VADER_MAX_ADDRESS with XPMEM_MAXADDR_SIZE #1701
Conversation
XPMEM already defines the max address value XPMEM_MAXADDR_SIZE, which has special handling on the kernel level. Therefore we should avoid usage of VADER_MAX_ADDRESS, which actually causes failures on some systems. Signed-off-by: Pavel Shamis (Pasha) <pasharesearch@gmail.com>
|
FYI @hjelmn |
|
@shamisp The problem is XPMEM_MAXADDR_SIZE in the Cray version of xpmem is wrong. If xpmem attempts to attach to an address over that value hard-coded in vader an error is returned. I plan to update my version of xpmem to expose the real maximum not just -1. This was affecting the bound not the attached region. |
|
On the other hand VADER_MAX_ADDRESS does not work for all systems. Do you have any other idea how to fix this ? |
|
Not sure. We may have to define VADER_MAX_ADDRESS by the architecture I fear. Otherwise we may see attach failures. I wonder if SGI ever solved this problem... I will look at their xpmem source code. |
|
@hjelmn how did you come up with the number that you use now ? |
|
I got originally got the value from trial and error with the api. Confirmed it from the kernel source later. |
|
For reference the largest address that can be attached to is TASK_SIZE which is (1 << 47) - 4096 on linux x86_64. See http://lxr.free-electrons.com/source/arch/x86/include/asm/processor.h#L747. This line in xpmem enforces the max attach address: https://github.com/hjelmn/xpmem/blob/master/kernel/xpmem_attach.c#L372. |
|
I wonder if it will be fine to just use the VADER_MAX_ADDRESS on x86. The problem is that linux x86 reserves the last 4096 bytes of the user virtual address space. The causes an attach in the top 2MB of the stack to fail because vader by default tries to attach to 2MB aligned/sized regions. This by definition includes the reserved page. If the linux kernel doesn't reserve the last page on other platforms we could simply use (size_t) -1 on those platforms. There may still be problems if the user attempts to further increase the attach size but that might be ok. |
|
Doing a spot check around different archs. ARM, PPC, MIPS, etc all use a power of two for TASK_SIZE. |
|
@hjelmn - not a bad idea. In UCX we only register the memory that we need, so we don't have this issue. |
|
Yeah, this comes from trying to keep the rcache small so lookups remain fast. |
|
I can make a new commit to define VADER_MAX_ADDRESS as XPMEM_MAXADDR_SIZE on non-x86 systems if you want or you can submit a new patch. If other platforms run into issues we can always special-case those platforms. |
|
No problem. I will update the patch. |
|
Based on the discussion I'm closing this PR, and will be submitting a new one. |
2 participants
XPMEM already defines the max address value XPMEM_MAXADDR_SIZE,
which has special handling on the kernel level. Therefore we should
avoid usage of VADER_MAX_ADDRESS, which actually causes failures
on some systems.
Signed-off-by: Pavel Shamis (Pasha) pasharesearch@gmail.com