[Messages] Improve throughput of jobs #278
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several improvements to the pending TX/message jobs.
The jobs used to work by batch, meaning they started a series
of N tasks and then waited for these tasks to finish before
starting the next batch. This has the obvious disadvantage
that the job ends up waiting on the slower tasks, while other
tasks could have been started earlier. The jobs now have a
max concurrency parameter that limits the total number of tasks.
The job will automatically collect any finished task and spawn
new ones up until the limit.
Refactored the message processing job to get rid of the i/j
logic that modified the batch size depending on whether the job
was processing STORE messages. The job now uses a system of
semaphores per message type that can limit the number of tasks
running in parallel for a specific message type. This is still
suboptimal as using semaphores per item type would make more
(to limit the number of tasks fetching data from the network
or ipfs), but this feature will be implemented later.
Several job metrics were removed, as they lost meaning:
These metrics were replaced by:
The max concurrency parameter of each job and the limits per
message type can now be configured from the config file.