Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Shard.list and Shard.get_by_key #127

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Commits on Aug 1, 2019

  1. Optimize Shard.list and Shard.get_by_key

    Previous list and get_by_key had to go through GenServer to acquire
    values ets table and replicas information. In case GenServer was
    processing an update (e.g. heartbeat, track, untrack) then list and
    get_by_key functions were blocked until it was completed. We saw this
    behaviour in our cluster where simple list/get_by_key calls were
    sometimes taking over few hundred milliseconds.
    
    Storing down replicas information in an ets table allows us to avoid
    going through genserver and allows us to process list/get_by_key
    immediately.
    
    I removed dirty_list function which was not public / exposed and which
    was trying to resolve the same issue. dirty_list was called dirty
    because it didn't check for down_replicas. This solution checks
    down_replicas and doesn't change the api interface.
    
    This should also resolve phoenixframework#124
    indrekj committed Aug 1, 2019
    Configuration menu
    Copy the full SHA
    4ebec7c View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2020

  1. Enable read_concurrency for down_replicas

    https://erlang.org/doc/man/ets.html
    
    > Performance tuning. Defaults to false. When set to true, the table is
    optimized for concurrent read operations. When this option is enabled on
    a runtime system with SMP support, read operations become much cheaper;
    especially on systems with multiple physical processors. However,
    switching between read and write operations becomes more expensive.
    
    We have a lot of read operations. Each time get_by_key or get_by_topic
    is called then this table is read. We want it to be as fast as possible.
    
    > You typically want to enable this option when concurrent read
    operations are much more frequent than write operations, or when
    concurrent reads and writes comes in large read and write bursts (that
    is, many reads not interrupted by writes, and many writes not
    interrupted by reads).
    
    This table is rarely updated. It is only updated when a replica goes
    down or when that replica comes back up.
    indrekj committed Jan 30, 2020
    Configuration menu
    Copy the full SHA
    59938a7 View commit details
    Browse the repository at this point in the history