Skip to content

Make global $all stream building a separate task #139

@slashdotdash

Description

@slashdotdash

All events appended to a stream are linked to the globally ordered all events stream (named $all). Because the events must be assigned a unique, monotonically incrementing, and gapless event number the linking is blocking, resulting in single writer performance.

To improve append event performance the all stream building could be moved to a separate async process which won't block event inserts into a stream. It should also be possible to disable all stream building via a config option.

Running the existing append events benchmark with/without all stream building resulted in a significant speed up.

AppendEventsBench

  • append events, 10 concurrent writers -79.53%
  • append events, 50 concurrent writers -71.69%
  • append events, 20 concurrent writers -59.88%
  • append events, single writer -38.53%

Above comparison shows the difference in time between all stream building enabled and disabled (commented out).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions