Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change EntriesProcessor logic from processing batches of entries to queue #186

Open
Tiendil opened this issue May 3, 2024 · 0 comments
Open

Comments

@Tiendil
Copy link
Owner

Tiendil commented May 3, 2024

Currently, EntriesProcessor starts processing N entries at once in parallel and then waits for all of them to finish before starting the next batch. This could slow down the processing of entries if the processing of one entry gets stuck for some reason.

Instead, we should still process them in parallel but not wait for all of them to finish. As soon as one entry is processed, we should start processing the next one.

It could be achieved by refactoring EntriesProcessor into a queue dispatcher, which controls N background consumers.

caused by gh-170

@Tiendil Tiendil added this to the 2.* Stabilisation milestone May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant