Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Redis pipelining for projections #226

Merged
merged 1 commit into from
Apr 13, 2024

Conversation

marein
Copy link
Owner

@marein marein commented Apr 12, 2024

Only covers the game projection for now because this is the most network intense as it acts on each single event. The rest is just interested in some of them.

This implementation (without queue deduplication) yields the following results:

Used subscriber Pipeline size 1 worker 3 workers
game-projection 1 - behaviour before this PR 2900 sets/s 6050 sets/s
game-projection 16 5080 sets/s 9900 sets/s
game-projection 50 5790 sets/s 11200 sets/s

Benchmark details:

  • Before execution, the database was filled with 6739 games that generated 175424 events.
  • All containers were deployed using /deploy/single-server/docker-compose.yml on the same machine (cheap VPS with 3 vCPUs, which is why it's not comparable to the results of Parallelize event store follower #99).
  • The data is not aggregated. There was only 1 run for each test.

Another benchmark was done with the docker stack described at #170. The number of workers were reduced from 8 to 3, while the game projection was still immediately available. Did go a bit further and added another FPM instance to reach 30k req/s that yielded roughly 31k game updates per second in Redis without a lag. The test was done with a max pipeline size of 32.

With those numbers, the other projections (running games, games by player and open games) don't need the added code complexity (as described in the first paragraph).

@marein marein self-assigned this Apr 12, 2024
@marein marein linked an issue Apr 12, 2024 that may be closed by this pull request
@marein marein force-pushed the 225-use-redis-pipelining-for-projections branch from bf26d83 to 62aa3bd Compare April 12, 2024 20:26
Only covers the game projection for now.
@marein marein force-pushed the 225-use-redis-pipelining-for-projections branch from 62aa3bd to 92a877e Compare April 13, 2024 12:20
@marein marein merged commit 3d522ea into master Apr 13, 2024
1 check passed
@marein marein deleted the 225-use-redis-pipelining-for-projections branch April 13, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use Redis pipelining for projections
1 participant