Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make global $all stream building a separate task #139

Open
slashdotdash opened this issue Oct 31, 2018 · 1 comment
Open

Make global $all stream building a separate task #139

slashdotdash opened this issue Oct 31, 2018 · 1 comment

Comments

@slashdotdash
Copy link
Member

@slashdotdash slashdotdash commented Oct 31, 2018

All events appended to a stream are linked to the globally ordered all events stream (named $all). Because the events must be assigned a unique, monotonically incrementing, and gapless event number the linking is blocking, resulting in single writer performance.

To improve append event performance the all stream building could be moved to a separate async process which won't block event inserts into a stream. It should also be possible to disable all stream building via a config option.

Running the existing append events benchmark with/without all stream building resulted in a significant speed up.

AppendEventsBench

  • append events, 10 concurrent writers -79.53%
  • append events, 50 concurrent writers -71.69%
  • append events, 20 concurrent writers -59.88%
  • append events, single writer -38.53%

Above comparison shows the difference in time between all stream building enabled and disabled (commented out).

@slashdotdash
Copy link
Member Author

@slashdotdash slashdotdash commented Jun 28, 2019

This feature could be combined with the changes needed for #169.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant