{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":496968290,"defaultBranch":"main","name":"shmutils","ownerLogin":"autumnjolitz","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2022-05-27T11:18:18.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/270476?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1690396724.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"d4953b3692e64038d6d4a3e3de5a4cd39cad37b0","ref":"refs/heads/rewrite-for-probing","pushedAt":"2023-07-26T18:38:44.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"autumnjolitz","name":"Autumn Jolitz","path":"/autumnjolitz","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/270476?s=80&v=4"},"commit":{"message":"[*] try to move over to having worker processes negotiate an overlapping mmap range before securing one\n\nI noticed that if one starts up a series of child processes and has them reserve (but not allocate) an immense range (like a terabyte), the odds of detecting an overlap of suitable size goes up. The variance between processes appears to be ~2GiB, which correlates with ASLR only going so far. No matter what, it still has to be written in a way to allow for large contiguous virtual address reservation and that means we can find common overlaps.\n\nSo try to make an effective ProcessPool that you can request the same mmap virtual addresses across parent and children.\n\nThe intention is that:\n\t- one creates a posix shared memory file descriptor\n\t- tries to reserve an appropriate overlapping range for it's eventual size\n\t- ftruncate() the ``fd`` to the desired ``size``\n\t- passes the fd to the children\n\t- mmap(..., fd, ...) on all\n\t- voila! shared memory block with consistent virtual address mappings that can store/dereference shared on-heap\n\t  pointers to other shared on-heap data!\n\nThis allows for construction of a shared slab allocator.\n\nA shared slab allocator is fundamental to allowing a thin Python proxy object that pulls values from the shared heap.\n\nThis thin proxy object could be pickled and sent between parent/children processes and threads. You can even allocate part of the heap for pickle buffers.\n\nThis is towards the goal plan of:\n\n\t- I want this rich object to have it's data in one place\n\t- I want to be able to lock my rich object between threads/processes\n\t- I want to be able to share my numpy array between processes by moving it into the shared memory and then handing out references to that shared memory backed numpy\n\t- I don't want to manage the lifecycle of shared memory\n\t\t+ When processes are done with it, they just close the fildes and the OS frees the entire heap when no one else uses it.","shortMessageHtmlLink":"[*] try to move over to having worker processes negotiate an overlapp…"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADXdHjRwA","startCursor":null,"endCursor":null}},"title":"Activity · autumnjolitz/shmutils"}