New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make global `$all` stream building a separate task #139

Open
slashdotdash opened this Issue Oct 31, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@slashdotdash
Copy link
Member

slashdotdash commented Oct 31, 2018

All events appended to a stream are linked to the globally order all events stream (named $all). Because the events must be assigned a unique, monotomically incrementing, and gapless event number the linking is blocking, resulting in single writer performance.

To improve append event performance the all stream building could be moved to a separate async process which won't block event inserts into a stream. It should also be possible to disable all stream building via a config option.

Running the existing append events benchmark with/without all stream building resulted in a significant speed up.

AppendEventsBench

  • append events, 10 concurrent writers -79.53%
  • append events, 50 concurrent writers -71.69%
  • append events, 20 concurrent writers -59.88%
  • append events, single writer -38.53%

Above comparison shows the difference in time between all stream building on and then completely removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment