Use proper shared memory on Linux, and cache things

[bill-myers]

Currently the code creates and maps a file in /tmp for each structure created, which is obviously completely unacceptable, since /tmp is not supposed to be used for this purpose and with this frequency and the syscall overhead will result in unusably slow performance in many cases.

This means the crate is currently only usable for long-lived deques if one is OK with files being randomly created in /tmp as part of normal operation of a propgram.

Instead, it should first of all use shared memory (with memfd_create, /dev/shm or SysV IPC) instead of /tmp, and second it should probably share one file descriptor among allocations where possible and finally it should probably at least cache some of the smaller buffers so that no syscalls are required if e.g. a function that uses a few SliceDeques internally is called in a loop.

[gnzlbg]

@bill-myers thank you a lot for the feedback. Everything you mention was already in the back of my head. I have split this issue into:

• using memfd_create on Linux: Use memfd_create on Linux once it becomes available in libc #12
• turning the mirrored module into an Allocator: Turn the mirrored allocation module (except for the buffer) into an Allocator #21
• formulate more clearly the lack of an Allocator drawback in the docs: Formulate more clearly in the readme the system-calls issue #24
• benchmarking sysv vs linux-specific algorithm: Benchmark system V IPC shared memory vs Linux-specific/Mach-specific ones #22
• add feature flags to allow selecting different algorithms on linux for benchmarking: Add a feature flag to allow selecting SysV IPC implementation on Linux #23

So that we can discuss and implement these independently without interleaving both discussions.

---

There is one thing from your comment that I don't understand. Maybe you could elaborate.

Currently the library uses mkstemp. I wanted to move it to use memfd_create (and started doing so two weeks ago but life got in the way), but I did not see this as a performance optimization, more like "a nicer" thing to do.

The files in /tmp should not be real hard-disk files, but just an id for some memory region (just like /dev/null and /dev/urandom are not real hard-disk files). What memfd_create does is exactly the same thing as mkstemp, but instead of giving the memory region a name that other processes can use, it makes it anonymous, so one won't see it in /tmp or somewhere else (*) , but the overhead should be "similar" to mkstemp, a little bit faster, but a file-descriptor is still created (which is what induces most of the overhead).

So if consuming file-descriptors is what you mean with this being very slow, memfd_create won't solve this for you.

---

Another tangential topic is using /dev/shm and SysV IPC instead of Linux-specific stuff. There is a SysV IPC implementation in the sysv.rs module that should be POSIX-compliant and is used when the system isn't Linux. The problem is that this implementation is racy (it might need to try multiple times to allocate memory) which might result in a lot of system calls. I have not benchmarked this against the Linux specific one yet but is something that I wanted to do as soon as the Linux algorithm starts using memfd_create (although one can probably already do this now).

• Do you know a way of making the SysV IPC implementation non-racy ?

I also explored using /dev/shm on this issue: #10 but I did not find a way of making it non-racy either and the SysV IPC implementation should be more portable than that one.

• Do you know a way of making the /dev/shm implementation non-racy ?
• In which systems do you think that /dev/shm should be preferred over SysV IPC and why?

Systems programming is a bit new to me so I'd like learning about all of this in more detail. Feel free to submit PRs or discuss my thoughts on the issues in more detail. Any information that you post there will allow anybody to implement things faster. My thoughts on the allocator part are a bit rough still because my mind was prioritizing other parts of the library so if you want to tackle that it would be better if you could explain exactly what and how you want to do on the allocator issue so that we are on the same page before sending a PR, but otherwise, happy hacking!

(*): using /tmp/ was suggested by Linus [back then]() when memfd_create` did not exist.

[gnzlbg]

@bill-myers

Instead, it should first of all use shared memory (with memfd_create, /dev/shm or SysV IPC) instead of /tmp,

This has been fixed.

it should first of all use shared memory (with memfd_create, /dev/shm or SysV IPC)

This has been fixed as well. On Linux it uses a non-racy memfd_create-based algorithm by default, but you can switch to a racy SysV IPC-based algorithm if you prefer by enabling the unix_sysv feature.

second it should probably share one file descriptor among allocations where possible and finally it should probably at least cache some of the smaller buffers so that no syscalls are required if e.g. a function that uses a few SliceDeques internally is called in a loop.

These two points are the same issue: Adding an allocator that recycles memor, and tracked on issue #21 .