Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSO Proxy: improve tso proxy reliability #6585

Merged
merged 16 commits into from
Jun 12, 2023
Merged

TSO Proxy: improve tso proxy reliability #6585

merged 16 commits into from
Jun 12, 2023

Commits on Jun 9, 2023

  1. Disable super flaky tests TestSplitKeyspaceGroup and TestScheduler. 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    2501b87 View commit details
    Browse the repository at this point in the history

  2. Disable the real flaky TestSplitKeyspaceGroup 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    2da4b0e View commit details
    Browse the repository at this point in the history

  3. Increase mcs ci-test-job timeout from 10 mins to 15 mins 

    It grows to quick. Increase timeout value for now, and we can split the test jobs into different chunks later.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    5d1ae54 View commit details
    Browse the repository at this point in the history

  4. Enable stress test 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    b3ce3a3 View commit details
    Browse the repository at this point in the history

  5. Force the stress test to run within limited time to avoid ci-test-job… 

    … to timeout.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>

Fix a

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    7557e16 View commit details
    Browse the repository at this point in the history

  6. Optimize forceLoad lock with RW lock 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    f3ca31c View commit details
    Browse the repository at this point in the history

  7. Add protection mechanisms to TSO Proxy. 

    1. Throttle the concurrency of TSO Proxy streamings. Default 5000.
2. If TSO Proxy didn't receive the TSO request from the the grpc.ServerStream (client) for 1 1hour, close the stream.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    75b9b49 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2023

  1. Add deadline for API leader forwarding request to TSO service. 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 10, 2023
    Configuration menu
    Copy the full SHA
    623ec6d View commit details
    Browse the repository at this point in the history

  2. bug fix 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 10, 2023
    Configuration menu
    Copy the full SHA
    70e5085 View commit details
    Browse the repository at this point in the history

  3. Refine test cases. 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 10, 2023
    Configuration menu
    Copy the full SHA
    0bfd840 View commit details
    Browse the repository at this point in the history

  4. Make tso response channel more safely. 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 10, 2023
    Configuration menu
    Copy the full SHA
    9a04e32 View commit details
    Browse the repository at this point in the history

  5. Move tso proxy stress test away from the test suite as it has impact … 

    …on other test cases.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 10, 2023
    Configuration menu
    Copy the full SHA
    891c9d1 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. Handle feedback - fix grpc client connection pool (server side) resou… 

    …rce leak problem.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    5bc03e0 View commit details
    Browse the repository at this point in the history

  2. Make MaxConcurrentTSOProxyStreamings (5000 as default) and TSOProxyCl… 

    …ientRecvTimeout (1 hour as default) configurable.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    b06b1eb View commit details
    Browse the repository at this point in the history

  3. Add metrics tsoProxyHandleDuration, tsoProxyBatchSize and tsoProxyFor… 

    …wardTimeoutCounter.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    deafbbb View commit details
    Browse the repository at this point in the history

  4. Handle feedback. 

    Signed-off-by: Bin Shi <binshi.bing@gmail.com>
    @binshi-bing
    binshi-bing committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    eb1d16b View commit details
    Browse the repository at this point in the history