fix: eliminate Mutex's reliance on setInterval #40
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently the Mutex is relying on
setInterval(keepAlive, 5000)
to refresh it's lease on the mutex. However, this is fatally flawed: there is no guaranteecb
will run every 5 seconds. It may not even ever run at all!In production, we're encountering a scenario where synchronous Markdown parser is running for ~90 seconds. Since JavaScript is single threaded, that means the Mutex is not renewed in time, and other threads decide our thread has crashed, so they go ahead and take the Mutex. Then 85 seconds later, our code continues on, thinking it still has the mutex, then it releases it, only to discover either a) it has lost ownership or b) it is double-freed (because it lost ownership but then the other thread finished its stuff and released the mutex, so nobody has it now).
The only reliable way to fix this is to always double check that our mutex lock hasn't expired before we do things that require the lock. If it has expired, we try to renew our lock. If that fails we throw an
ETIMEOUT
.