Best Practices for Concurrent Usage Doubts

[xiaguan]

We've got a bunch of write back tasks in our file system for pushing dirty data back to cloud storage. I'm trying to figure out if it's best to share a single Arc<Operator> among all these tasks or if we should be giving each task its own Operator instance. Is there even a difference between these two approaches when it comes to performance or how things work?

[xiaguan]

And another thing.
I've been feeling like the Memory Service is kinda slow, especially with lotfs of data? Though, I've only noticed this during testing.

[Xuanwo]

I'm trying to figure out if it's best to share a single Arc<Operator> among all these tasks

Operator itself is an Arc. We don't need to have Arc<Operator>.

if we should be giving each task its own Operator instance.

It depends. For services like s3, they have seperate http clients; For services like memory, their data are not shared; For services like fs, they are the same.

In general, we don't need to giving each task its own Operator instance if they are backed by the same storage.

I've been feeling like the Memory Service is kinda slow, especially with lotfs of data?

Yes, the memory service is a basic Arc<Mutex<HashMap<String, Vec<u8>>>>, which isn't suitable for handling large amounts of data, especially when used concurrently. Maybe you could try dashmap or other storage services.

[xiaguan]

I got it , thanks