Is your feature request related to a problem? Please describe.
In our forum we experimented with adding FOR UPDATE SKIP LOCK to the select statement for the job executor. This does not work for all supported databases, but sped up execution for many nodes running in parallel.
Describe the solution you'd like
We might consider adding an opt-in configuration flag to enable this behavior, so users running on a database which supports this could benefit from higher concurrency.
Describe alternatives you've considered
An alternative provided in the thread was to provide read offsets for different nodes so they don't all try to read on the most recent jobs. This works too, but might be dangerous if a node crashes and the range that was reserved for it is not read anymore.
Is your feature request related to a problem? Please describe.
In our forum we experimented with adding
FOR UPDATE SKIP LOCKto the select statement for the job executor. This does not work for all supported databases, but sped up execution for many nodes running in parallel.Describe the solution you'd like
We might consider adding an opt-in configuration flag to enable this behavior, so users running on a database which supports this could benefit from higher concurrency.
Describe alternatives you've considered
An alternative provided in the thread was to provide read offsets for different nodes so they don't all try to read on the most recent jobs. This works too, but might be dangerous if a node crashes and the range that was reserved for it is not read anymore.