Add jitter on the processor fetch backoff sleep #868
+2
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds some jitter to the backoff sleep of the processor fetch loop to avoid workers synchronization, and thus improve worker efficiency and job latency.
Context
We run dozens of workers, with an execution rate of up to 5K jobs/sec.
We started to suspect that the job workers were significantly inefficient.
Changing the TaskCheckInterval from 1s (default) to 100ms has reduced the latency from ~500ms to 40ms on a high traffic worker pool, and 10ms to a low traffic worker pool.
Then, adding the jitter has further reduced the latency to 2 ms on both high and low traffic worker pool.
Synchronization on a distributed system is well known, and adding jitter helped avoid it.
Impact of adding a jitter: Queue size:
![Screenshot 2024-04-15 at 19 05 32](https://private-user-images.githubusercontent.com/60219/322566838-5b2fc86e-5e13-42cf-9e2a-0e9dd8efe05c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzMzNDgsIm5iZiI6MTcyNDI3MzA0OCwicGF0aCI6Ii82MDIxOS8zMjI1NjY4MzgtNWIyZmM4NmUtNWUxMy00MmNmLTllMmEtMGU5ZGQ4ZWZlMDVjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDIwNDQwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWIxYmM2MzllNTBmNmZlMGM3MWU2NGM2N2Y4OWUyMzVjMzU4ZmU1N2I5NDdmNTExYjA2NDY1ZWQ0M2E1MjVlZWUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.3p455tmCdi9xtgSwhPW14gv6fbJfODGOFY60hWzHzgc)
Latency:
![Screenshot 2024-04-15 at 19 05 51](https://private-user-images.githubusercontent.com/60219/322566898-a8e634f6-6224-455d-8a08-6743a2d81b20.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzMzNDgsIm5iZiI6MTcyNDI3MzA0OCwicGF0aCI6Ii82MDIxOS8zMjI1NjY4OTgtYThlNjM0ZjYtNjIyNC00NTVkLThhMDgtNjc0M2EyZDgxYjIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDIwNDQwOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWUwZmI4N2U1OWMxOGQyZTkzYjRiYzRkNGE0OTk2OWM3ZWRlYzI5YzQ3NjNiNzQ0MDc0MTEyMWI5MzhkYzBhYzcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.zXvye5hHCc2bsGRJsCXi5VdH6I8apZC1At-TKsX8QUY)