Skip to content

Commit

Permalink
[*] try to move over to having worker processes negotiate an overlapp…
Browse files Browse the repository at this point in the history
…ing mmap range before securing one

I noticed that if one starts up a series of child processes and has them reserve (but not allocate) an immense range (like a terabyte), the odds of detecting an overlap of suitable size goes up. The variance between processes appears to be ~2GiB, which correlates with ASLR only going so far. No matter what, it still has to be written in a way to allow for large contiguous virtual address reservation and that means we can find common overlaps.

So try to make an effective ProcessPool that you can request the same mmap virtual addresses across parent and children.

The intention is that:
	- one creates a posix shared memory file descriptor
	- tries to reserve an appropriate overlapping range for it's eventual size
	- ftruncate() the ``fd`` to the desired ``size``
	- passes the fd to the children
	- mmap(..., fd, ...) on all
	- voila! shared memory block with consistent virtual address mappings that can store/dereference shared on-heap
	  pointers to other shared on-heap data!

This allows for construction of a shared slab allocator.

A shared slab allocator is fundamental to allowing a thin Python proxy object that pulls values from the shared heap.

This thin proxy object could be pickled and sent between parent/children processes and threads. You can even allocate part of the heap for pickle buffers.

This is towards the goal plan of:

	- I want this rich object to have it's data in one place
	- I want to be able to lock my rich object between threads/processes
	- I want to be able to share my numpy array between processes by moving it into the shared memory and then handing out references to that shared memory backed numpy
	- I don't want to manage the lifecycle of shared memory
		+ When processes are done with it, they just close the fildes and the OS frees the entire heap when no one else uses it.
  • Loading branch information
autumnjolitz committed Jul 26, 2023
1 parent 252ce3f commit d4953b3
Show file tree
Hide file tree
Showing 9 changed files with 1,467 additions and 564 deletions.
14 changes: 2 additions & 12 deletions shmutils/__init__.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,7 @@
"Shared memory utils in python"
__version__ = "0.0.2"

from .posix import PosixSharedMemory, shm_open, shm_unlink
from .mmap import MappedMemory, Flags as MapFlags, Protections as MapProtections
from .utils import RelativeView
from .mmap import Flags as MapFlags, Protections as MapProtections, mmap


__all__ = [
"PosixSharedMemory",
"shm_open",
"shm_unlink",
"MappedMemory",
"MapProtections",
"MapFlags",
"RelativeView",
]
__all__ = ["MapProtections", "MapFlags", "mmap"]
Loading

0 comments on commit d4953b3

Please sign in to comment.