Make CoreCLR work properly under PaX's RANDMMAP (#11382)

Make CoreCLR work properly under PaX's RANDMMAP

Issues:
- The ExecutableMemoryAllocator is used to attempt to map native images into a memory range near libcoreclr so that its helper table can use short relative addresses for jumps to libcoreclr
  - RANDMMAP typically prevents mmap calls with a specific address from reserving memory at the requested address, so the executable memory allocator fails to reserve any memory. When Server GC is enabled, the large GC heap can exacerbate the issue by taking address space near libcoreclr.
- Native images are loaded far from libcoreclr, and now jump stub space needs to be allocated near the native image, but RANDMMAP typically prevents this too
- NGenReserveForJumpStubs is intended to reserve some memory near mapped native images for jump stubs, but that reservation is done with a separate mmap call in the same way as above and RANDMMAP typically prevents this too
- The JIT needs to allocate memory for code that may need to jump/call to a native image or libcoreclr, which may require jump stubs near the code that cannot be allocated. CodeHeapReserveForJumpStubs reserves space in code heap blocks without using a separate call to mmap, so this works, but without this environment variable by default there is still a good chance of failing.
- See https://github.com/dotnet/coreclr/blob/56d550d4f8aec2dd40b72a182205d0a2463a1bc9/Documentation/design-docs/jump-stubs.md for more details

Fixes #8480
- It would be ideal to fix all of the above properly, such that there would never be a need to attempt reserving memory within a certain range. Since we're running out of time for 2.0, I figured the following simpler, temporary solution that should cover most of the practical cases, may be appropriate for 2.0.
- Extended ExecutableMemoryAllocator to reserve address space even when it cannot do so near libcoreclr
- Had ClrVirtualAllocWithinRange use the executable memory allocator to reserve memory for jump stubs when the requested range is satisfied
- This covers a maximum of ~2 GB of executable code and should cover most of the practical cases. Once this space is exhausted, under RANDMMAP, native images loaded later will fail, and for jitted code the environment variable above can be used.

Author: Koundinya Veluri

src/dlls/mscordac/mscordac_unixexports.src

@@ -39,6 +39,7 @@ PAL_fprintf
 PAL__wcstoui64
 PAL_wcstoul
 PAL_iswprint
+PAL_VirtualReserveFromExecutableMemoryAllocatorWithinRange
 PAL_wcslen
 PAL_wcsncmp
 PAL_wcsrchr

src/pal/inc/pal.h

@@ -2850,6 +2850,14 @@ PALAPI
 LPCVOID
 PAL_GetSymbolModuleBase(void *symbol);
 
+PALIMPORT
+LPVOID
+PALAPI
+PAL_VirtualReserveFromExecutableMemoryAllocatorWithinRange(
+    IN LPCVOID lpBeginAddress,
+    IN LPCVOID lpEndAddress,
+    IN SIZE_T dwSize);
+
 PALIMPORT
 LPVOID
 PALAPI

src/pal/src/include/pal/virtual.h

@@ -60,7 +60,7 @@ enum VIRTUAL_CONSTANTS
 
     VIRTUAL_PAGE_SIZE       = 0x1000,
     VIRTUAL_PAGE_MASK       = VIRTUAL_PAGE_SIZE - 1,
-    BOUNDARY_64K    = 0xffff
+    VIRTUAL_64KB            = 0x10000
 };
 
 /*++
@@ -130,11 +130,22 @@ class ExecutableMemoryAllocator
         AllocateMemory
 
         This function attempts to allocate the requested amount of memory from its reserved virtual
-        address space. The function will return NULL if the allocation request cannot
+        address space. The function will return null if the allocation request cannot
         be satisfied by the memory that is currently available in the allocator.
     --*/
     void* AllocateMemory(SIZE_T allocationSize);
 
+    /*++
+    Function:
+        AllocateMemory
+
+        This function attempts to allocate the requested amount of memory from its reserved virtual
+        address space, if memory is available within the specified range. The function will return
+        null if the allocation request cannot satisfied by the memory that is currently available in
+        the allocator.
+    --*/
+    void *AllocateMemoryWithinRange(const void *beginAddress, const void *endAddress, SIZE_T allocationSize);
+
 private:
     /*++
     Function:
@@ -160,12 +171,13 @@ class ExecutableMemoryAllocator
     // that can be used to calculate an approximate location of the memory that
     // is in 2GB range from the coreclr library. In addition, having precise size of libcoreclr
     // is not necessary for the calculations.
-    const int32_t CoreClrLibrarySize = 100 * 1024 * 1024;
+    static const int32_t CoreClrLibrarySize = 100 * 1024 * 1024;
 
     // This constant represent the max size of the virtual memory that this allocator
     // will try to reserve during initialization. We want all JIT-ed code and the
     // entire libcoreclr to be located in a 2GB range.
-    const int32_t MaxExecutableMemorySize = 0x7FFF0000 - CoreClrLibrarySize;
+    static const int32_t MaxExecutableMemorySize = 0x7FFF0000;
+    static const int32_t MaxExecutableMemorySizeNearCoreClr = MaxExecutableMemorySize - CoreClrLibrarySize;
 
     // Start address of the reserved virtual address space
     void* m_startAddress;

src/pal/src/map/map.cpp

@@ -2445,8 +2445,10 @@ void * MAPMapPEFile(HANDLE hFile)
 #else // defined(_AMD64_)    
     // First try to reserve virtual memory using ExecutableAllcator. This allows all PE images to be
     // near each other and close to the coreclr library which also allows the runtime to generate
-    // more efficient code (by avoiding usage of jump stubs).
-    loadedBase = ReserveMemoryFromExecutableAllocator(pThread, ALIGN_UP(virtualSize, GetVirtualPageSize()));
+    // more efficient code (by avoiding usage of jump stubs). Alignment to a 64 KB granularity should
+    // not be necessary (alignment to page size should be sufficient), but see
+    // ExecutableMemoryAllocator::AllocateMemory() for the reason why it is done.
+    loadedBase = ReserveMemoryFromExecutableAllocator(pThread, ALIGN_UP(virtualSize, VIRTUAL_64KB));
     if (loadedBase == NULL)
     {
         // MAC64 requires we pass MAP_SHARED (or MAP_PRIVATE) flags - otherwise, the call is failed.