Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rangefeed: add scheduler based rangefeed processor #107553

Merged
merged 3 commits into from Sep 8, 2023

Commits on Sep 4, 2023

  1. testutil: make RunValues generic

    Previously run values was using interface{} and casting to provide
    execution options for tests. Making function generic reduces boilerplate
    code needed to setup and use test parameters.
    
    Epic: none
    
    Release note: None
    aliher1911 committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    14f1183 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. rangefeed: add scheduler based rangefeed processor

    Previously each rangefeed processor run a dedicated goroutine to
    process range events. With high replica density that produced
    10K's of goroutines waking up periodically and overloading go
    scheduler.
    This commit adds a version of processor that uses a scheduler
    with a fixed pool of workers to process all ranges on the node.
    Scheduler based processor is off by default and can be enabled
    by using `kv.rangefeed.scheduler.enabled` cluster setting.
    Number of workers in scheduler pool is determined by environment
    variable `COCKROACH_RANGEFEED_SCHEDULER_WORKERS`.
    
    Epic: none
    
    Release note (performance improvement): Added scheduler based
    rangefeed processor which improves rangefeed and changefeed
    performance for very large tables. New processor is disabled by
    default, but can be enabled via `kv.rangefeed.scheduler.enabled`
    cluster setting.
    aliher1911 committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    5c2f80c View commit details
    Browse the repository at this point in the history
  2. rangefeed: add transaction pusher to scheduler rangefeed

    This commit adds slow transaction pusher to scheduler based rangefeed.
    Store will trigger bulk enqueueing of relevant push events to
    eligible processors.
    
    Epic: none
    
    Release note: None
    aliher1911 committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    f3a8a95 View commit details
    Browse the repository at this point in the history