Use Redis pipelining for projections #226
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Only covers the game projection for now because this is the most network intense as it acts on each single event. The rest is just interested in some of them.
This implementation (without queue deduplication) yields the following results:
game-projection
game-projection
game-projection
Benchmark details:
/deploy/single-server/docker-compose.yml
on the same machine (cheap VPS with 3 vCPUs, which is why it's not comparable to the results of Parallelize event store follower #99).Another benchmark was done with the docker stack described at #170. The number of workers were reduced from 8 to 3, while the game projection was still immediately available. Did go a bit further and added another FPM instance to reach 30k req/s that yielded roughly 31k game updates per second in Redis without a lag. The test was done with a max pipeline size of 32.
With those numbers, the other projections (running games, games by player and open games) don't need the added code complexity (as described in the first paragraph).