Is there a way to provide my own shared memory and notification channel?

[DUOLabs333]

I'm currently in the process of optimizing the communication library I use for many of my projects --- right now, I'm using TCP as a generic channel, but since the communication most of the time is between a VM and the host, I thought that it would be a good idea to implement a special shared memory-based channel for that scenario. While I do have something working (I'm basically using QEMU's ivshmem feature to open a memory segment that's shared between guest and host), it's a lot of code, and optimizing it is taking a while (I'm currently trying to figure out why writing to the same memory address is faster than writing to two different addresses, switching between addresses on each write) --- pushing that work off to a library would be very helpful (though I'm not sure about performance...)

While researching different options, I came across this project, which looked interesting and the most promising --- however, I noticed that iceoryx, like many other projects, seem to handle shared memory and cross-process notification itself. I do remember on a Reddit thread that (ostensibly) of the team members said that the architecture of the project makes it easy to support new platforms as long as APIs for notification and shared memory are available --- is this available as an external API that users can utilize (eg, when creating a publisher/subscriber, the user can supply in a path to shared memory of length N, and a notification callback)? This would be useful for my usecase, as since one side is in a VM, normal IPC mechanisms won't work.

[elBoberido]

@DUOLabs333 sorry for the late response. This totally slipped my attention.

We already have hypervisor communication between host and qemu guest. It is one of the components which are not open-source. It might become open-source at some point, but currently it is one of the pillars we create in order to get have a sustainable long-term funding for the OSS project.

If you are using Rust, it is quite trivial to create custom services. This is how the IPC Service is defined https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/service/ipc.rs. So, you basically have to implement a trait by just defining some types. With this trait, every aspect of the implementation can be adjusted and the API for the user stays the same. This is also what we use for our hypervisor support.

[DUOLabs333]

Thanks --- are there any public examples of implementations available, so I can figure out what needs to be implemented (this would be my first foray into Rust, so examples would be very helpful)?

[elBoberido]

The example is essentially the link I sent you. This is the implementation for a local variant which does not share data between the processes https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/service/local.rs

[elBoberido]

Oh, there is even a real example -> https://github.com/eclipse-iceoryx/iceoryx2/tree/main/examples/rust/service_variant_customization

[DUOLabs333]

My biggest concern is with the shared memory allocator (I'll mostly be using the request-response pattern) --- if the pool is n bytes long, and can't be expanded, is there a hard limit on how many concurrent channels can be active (though not necessarily in use); that is, are blocks of memory be given back to the pool automatically if one channel needs one and another channel doesn't need theirs anymore?

[elBoberido]

The memory is always calculated for the worst case assumption, so one can never run out of memory. There is a new option to override this calculation but then one needs to know its system quite well and potentially has to deal with failing allocations. Due to the worst case assumptions, request-response is a bit memory hungry when used with streaming.