Queue seems deadlocked in certain scenarios

[jeromegn]

Sorry for the vagueness of this issue, I am looking for help in troubleshooting my queue.

I've started using this crate to pass logs from multiple processes into a single sink (with a single consumer). After an unknown sequence of events which involve crashes, a laptop going to sleep, etc. it seems like the queue becomes stuck.

I am assuming this is because something acquired a write lock and then never released it. I've been reading that using pthread_mutexattr_setrobust might help in detecting "dead owners"? This only works on linux, so something else would be required for macos and windows.

Either way, can I manually, unsafely, unlock it? Possibly by forcing the unlock steps from another process?

[dicej]

Hi @jeromegn. Yeah, I could imagine that if a writer (or reader) crashes while holding the lock, any further attempts to acquire the lock would hang indefinitely, which is consistent with pthread_mutexattr_setrobust defaulting to PTHREAD_MUTEX_STALLED. The alternative -- using PTHREAD_MUTEX_ROBUST and then calling pthread_mutex_consistent after receiving EOWNERDEAD from pthread_mutex_lock -- has its own hazard: depending on what the crashing process was doing when it crashed, it may have left the ring buffer in an inconsistent state, so the new lock owner would be responsible for fixing that before calling pthread_mutex_consistent. Probably the only reliable way to fix it would be to reset it entirely, meaning any unread data it contained (including items written by writers which are still healthy) would be lost.

In any case, I think the first thing to look at is why these processes are crashing in the first place. Have you looked into that at all?

[jeromegn]

Processes crash and exit uncleanly all the time for many reasons that change over time :) I can't assume my programs will be bug-free and crash-free and exit cleanly every time. Sometimes the kernel might SIGKILL a process because of an OOM condition.

My use case is probably an edge case (or even ill-suited) for ipmpsc: hundreds (thousands, eventually) of processes sending log lines into the same ipmpsc and a single process to consume them. Splitting it up per process would make more sense, but opening n receivers and spinning on each of them in a blocking fashion feels wrong. Maybe if this was async-compatible, it wouldn't be as resource expensive (I am not judging or suggesting you should've done that, just mentioning it). I doubt there's a way to do this?

To answer your question, in this particular instance, I think my laptop went to sleep and a thread panicked due to another crate's bug.

Otherwise, ipmpsc has been working pretty well for me on a much smaller scale (2-5 processes writing logs concurrently). I suspect I'll have trouble going to 100+ processes while keeping it reliable.

[dicej]

You're definitely pushing it harder than I ever did. My original motivation for this library was just to support efficient IPC with one sender and one receiver. I realized later that supporting more than once concurrent sender was easy to do, so I did, but I never made use of it outside of tests.

Regarding async-compatibility: yes, it's something I considered, but it wasn't critical for my use case so I never pursued it. Also, it's not obvious to me how to use POSIX mutexes and condition variables or Windows semaphores in an async-native way (i.e. without just throwing a bunch of threads at it, which mostly defeats the purpose).

Anyway, I agree that having some form of crash recovery is important for a use case like yours, and I also agree that PTHREAD_MUTEX_ROBUST would be a good place to start. The main question is how to reset the ring buffer to a known-good state and how to communicate that reset to the application, with the implication that some messages may have been lost.