feat: provide redis shared memory plugin #345
Conversation
Signed-off-by: hang lv <xlv20@fudan.edu.cn>
docs/source/examples/ipc.md
Outdated
| @@ -4,7 +4,7 @@ This is an example demonstrating how you can enable the plasma shared memory sto | |||
|
|
|||
| Mosec's multi-stage pipeline requires the output data from the previous stage to be transferred to the next stage across python processes. This is coordinated via Unix domain socket between every Python worker process from all stages and the Rust controller process. | |||
|
|
|||
| By default, we serialize the data and directly transfer the bytes over the socket. However, users may find wrapping this IPC useful or more efficient for specific use cases. Therefore, we provide the `mosec.plugins.IPCWrapper` interface and an example implementation `PlasmaShmWrapper` based on [`pyarrow.plasma`](https://arrow.apache.org/docs/python/plasma.html). | |||
| By default, we serialize the data and directly transfer the bytes over the socket. However, users may find wrapping this IPC useful or more efficient for specific use cases. Therefore, we provide the `mosec.plugins.IPCWrapper` interface and example implementation `PlasmaShmWrapper` based on [`pyarrow.plasma`](https://arrow.apache.org/docs/python/plasma.html) and `RedisShmWrapper` based on [`redis`](https://pypi.org/project/redis). We recommand using `RedisShmWrapper` for better performance. | |||
There was a problem hiding this comment.
Is "better performance" verified experimentally?
There was a problem hiding this comment.
Yes, based on the benchmark experiment at https://github.com/mosecorg/numbin/blob/master/benchmark/bench.py, the results show that when comparing the performance ,Redis connected via a Unix socket > Plasma > Redis connected via a TCP socket.
There was a problem hiding this comment.
Cool!
But the benchmark file mentioned above only tests the serialization / de-serialization performance, so you modify it to test the shared memory performance?
There was a problem hiding this comment.
Since it's a repo of serde library, I reused the cases while did not update it.
There was a problem hiding this comment.
I think the code and benchmark data can be uploaded to mosecorg/benchmark. We can add the link here.
Signed-off-by: hang lv <xlv20@fudan.edu.cn>
| return str(number).encode() | ||
|
|
||
| def bytes_id_gen() -> bytes: | ||
| int_id_gen = partial(self.client.incr, DEFAULT_KEY) |
There was a problem hiding this comment.
If I run multiple mosec instances, will they share the same key to the same redis?
There was a problem hiding this comment.
@n063h Could you please add something (e.g., timestamp) to differentiate?
|
The implementation can refer to the new plasma mixin https://github.com/mosecorg/mosec/blob/main/mosec/mixin/plasma_worker.py |
|
Since redis shm is now implemented as a mixin worker, refer to #367, instead of plugin, this pr is closed. |
provide RedisShmWrapper class and an example.