Skip to content

Fix stream error handling #705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Sep 19, 2025
Merged

Fix stream error handling #705

merged 3 commits into from
Sep 19, 2025

Conversation

vgvoleg
Copy link
Collaborator

@vgvoleg vgvoleg commented Sep 17, 2025 •
edited
Loading

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

What is the current behavior?

Fixes: #696
Fixes: #572
Fixes: #564
Fixes: #550
Fixes: #538
Fixes: #335

What is the new behavior?

Other information

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 17, 2025 •
edited
Loading

🌋 Here are results of SLO test for sync-query:

Operation Success Rate

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Success Rate, %" 89 --> 111
    line [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,99.989,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "	Time, m" 0 --> 10
    y-axis "Success Rate, %" 89 --> 111
    line [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,99.961,99.962,99.962,99.962,99.962,99.962,99.962,99.961,99.961,99.961,99.961,99.961,99.961,99.961,99.961,99.96,99.961,99.96,99.96,99.96,99.96,99.96,99.959,99.96,99.96,99.958,99.96,99.96,99.958,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100]
Loading

Operations Per Second

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Operations" 280 --> 760
    line [546.171,567.961,590.026,611.989,634.497,664.862,663.034,663.931,664.793,664.552,662.862,662.586,661.069,661.862,660.931,661.276,659.897,660.31,659.138,658.621,658.483,660,659.552,662.517,661.69,661.448,647,643,642.931,642.448,642.69,641.897,641.931,644.828,644.138,645.793,644.31,641.379,643.586,645.172,645.552,646.621,644.793,647.931,648.138,649.241,647.862,649.103,650.517,650.414,650,653,649.448,649.517,651.897,666.31,667.862,667.655,668.621,669.862,671.655,671.069,667.69,667.862,668.552,670.034,673.448,671.138,669.31,668.759,670.931,669.793,666.759,666.828,666.793,668.483,667.483,666.552,666.724,666.207,666.448,668,667.931,666.103,665.655,666.586,664.483,641.828,630.793,628.931,629.621,629.276,628.655,628.345,627.276,625.897,628.069,628.448,629.448,627.034,629.138,629.966,629.241,629.172,630,630.31,630.724,630.862,631.69,628.517,627.276,627.138,625.69,625.483,624.966,627.552,651.759,660.517,663.586,663.448,663.931,665.069,664.483,665.034,665.034,665.276,665.448,663.828,663.414,666.586,664.345,666.414,665.069,665.172,665.483,664.759,668.034,667.862,669.138,669.069,669.586,670.759,670.207,670.517,672.448,670.379,671.172,647.31,644.517,645.172,645.034,646.207,645.483,647.31,646.517,645.103,650.828,650.31,646.241,647.69,646.552,647.828,646.621,647.552,650.448,647.276,646.069,651.31,652.069,651.897,652.897,654.379,654.69,654.862,656.069,656.759,678.517,681.828,681.276,683.931,683.828,684.103,681.034,681,681.138,680.69,680.793,680.759,680.483,680.034,681.931,683.379,684.69,682.207,681.207,682.034,675.759,674.724,675.621,677.552,677.724,677.897,674.448,673.69,673.034,672.552,674.483,674.172,671.793,671.69,671.793,673.241,671.724,672.828,668.931,672.759,672.759,672.931,673.69,672.759,670.862,668.621,667.655,670.966,670.966,674.207,674.276,673.31,670.793,671.586,671.138,672.517,670.862,671.138,670.897,667.931,667.931,669.517,668.448,669.655,669.483,670,669.379,668.862,665.448,664.69,667.655,667.552,667.138,667.103,666.724,667.621,665.31,664.828,660.414,660.483,660.483,658.897,657.69,657.448,657.103,658.69,659.586,659.345,659.138,659.31,657,656.621,654.586,657.483,655.862,655.207,653.621,652.69,653.034,650.276,649.621,648.414,648.034,650,649.172,648.793,648.448,649.207,650.034,651.828,652.69,655.138,656.034,656.172,656.862,656.138,656.793,657.345,657.034,660.345,663.724,662.241,661.552,661.621,662.379,664.517,664.207,666.552,667.483,671.034,674.103,674.207,672.897,674.414,674.483,676.897,678.69,680,679.103,677.621,676.897,677.414,679.862,679.552,679.862,680.552,681.448,681.793,678.69,677.103,679.069,677.069,678.241,679.69,680.414,682.517,681.276,680.862,678.138,676.414,678,677.862,677.034,680.655,679.552,678.828,677.138,676.862,677.517,676.759,676.724,674.138,674.138,678.034,678.138,677.724,677.172,677.172,679.897,681.517,681.483,681,680.724,679.414,679.931,681,681.414,680.759,682.724,685.241,685.828,685.828,682.276,684.034,685.483,685.414,684.897,685.897,688.724,688.448,690.172,690.31,689,688.069,687.276,686.655,686.966,682.862,681.414,681.483,682.172,684.172,684.103,682.138,680.621,680.069,680.276,680.483,676.345,676.586,677.862,677.138,675.586,677.655,677.586,676.931,677.483,677.138,677.103,675.69,675.207,672.724,673.448,673.931,675.724,675.138,677.897,679.655,680.103,679,677.828,676.621,678.586,678.345,681.172,682.793,680.828,681.897,681.552,680.586,682.966,684.586,681.31,681.724,682.241,685,681.207,679.966,679.793,679.207,678.069,676.724,676.138,675.655,675.517,676.552,674.207,673.552,674.448,672.586,674.103,672.414,675.897,673.724,673.517,673.241,673.172,671.172,670.931,668.862,667.448,665.793,665.207,665.276,661.897,661.483,661.276,662.586,662.931,664.276,665.207,665.793,665.483,666.552,663.69,663.483,662.448,663.931,662.897,662.69,661.759,658.31,656.897,655.241,655.034,655.379,656.655,657.448,657.241,655.897,658.517,657.724,657.931,658.448,658.517,661.586,661.379,664.034,663.138,662.966,662.207,660.931,661.483,661.483,662.138,664.138,662.207,663.655,663.517,668.31,668.483,668.103,667.655,667.069,666.138,665.448,664.448,663.897,663.862,662.897,664.897,663.586,662.103,662.414,661.069,661.621,658.172,657.034,655.931,655.69,656.448,654.621,653.069,653.448,652.345,652,651.138,649.931,646.31,646.034,647.069,649.207,650.276,649.966,649.897,650.414,649.448,649.897,649.241,648.172,649.172,649.931,649.966,648.966,647.448,648.103,648.724,651.414,652.414,652.586,653.207,654.828,654.483,654.517,655.483,655.621,656,654.69,655.31,656.172,653.931,651.897,652.414,652.345,652.069,653.828,653.103,655.276,654.828,654.345,640.175,618.55,596.716,574.671,552.907,531.425,507.928,485.947,464.285,442.089,419.781,397.9,376.271,353.988,332.466,311.571]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "Time, m" 0 --> 10
    y-axis "Operations" 38 --> 104
    line [72.811,76.257,79.463,82.395,85.668,90.069,90,90.034,89.69,89.207,91.241,91.207,91.276,90,90.069,90.103,90.172,90.241,90.172,90.517,90.448,90.483,90.69,87.138,87.586,89.862,88.552,91.586,91.172,91.414,91.31,91.448,91.862,88.552,88.586,88.655,88.621,88.966,89.448,87.345,87.552,86.793,88.379,85.828,85.828,85.586,85.207,85.345,85.103,85.241,85.31,81.897,85.31,85.241,81.862,83.241,83.138,83.138,83.138,81.034,79.759,79.793,83.172,83.103,82.621,80.586,79.966,79.793,81.966,81.931,81.621,80.793,83.517,83.241,83.552,82.897,82.759,83,83.034,83.034,83.897,83.034,81.793,85.103,85.034,84.241,83.138,79.931,82.138,83.414,83.379,83.379,83.414,83.897,85.931,86.552,84.517,83.414,82.655,83.483,84.034,83.931,84.138,83.724,84.793,84.793,83.586,83.586,83.483,85.931,86.862,88.172,88.241,88.31,89.103,90.207,90.069,89.897,86.517,86.517,86.517,86.517,86.517,86.517,86.517,85.483,86.655,87.414,87.931,84.862,84.931,85,85.31,85.241,85.241,86.138,83.172,83.31,83.379,83.31,83.31,83.31,83.31,83.31,81.103,84.483,84,84.483,84.586,84.655,84.69,84.69,84.69,83.069,84.552,84.517,81.138,81.172,84.552,84.552,84.552,84.552,84.552,82.414,81.483,84.483,84.483,81.069,81.069,81.069,81.069,81.034,80.517,82.655,82.448,82.172,84.517,83.207,83.103,79.69,79.69,79.69,81.31,83.069,83.103,83.138,83.138,83.138,83.138,83.138,80.862,79.759,79.069,80.31,79.931,79.724,83.138,83.103,83.103,79.724,79.759,80.276,80.345,80.517,81.379,81.931,79.793,79.828,83.207,83.207,83.207,83.207,83.207,83.207,86.552,83.207,83.207,83.207,83.207,85.483,86.586,89.414,89.414,87.517,86.655,84.345,84.207,84.103,86.655,86.655,86.655,86.655,86.655,86.621,86.621,89.414,88.897,86.586,86.621,86.621,86.621,86.621,86.621,86.621,89.966,89.966,86.621,86.621,86.621,86.621,86.621,86.621,86.621,87.517,89.517,89.793,89.552,90.172,90.069,90.138,90.138,90.069,87.759,87.793,88.034,88.31,90.207,90.138,90.138,86.724,86.759,86.759,86.759,86.759,86.759,90.103,90.069,89.897,89.069,87.517,86.724,89,88.828,88.897,87.103,86.31,86.517,83.69,83.621,83.448,83.31,85.655,85.621,85.862,85.931,83.172,79.793,79.793,83.207,83.172,83.172,83.172,83.172,83.172,83.172,79.828,77.931,78.552,79.897,80.69,80.69,80.069,76.897,78.586,79.724,79.724,82.655,82.655,80.655,80.724,80.793,80.621,80.345,80.138,83.241,86.655,86.621,86.69,86.69,86.69,85.552,85.483,85.034,84.621,87.931,89.724,89.103,89.276,89.241,85.862,86.793,90.207,90.207,90.207,90.207,90.207,90.207,92.379,92.517,89.034,89.241,89.586,89.966,89.862,87.276,86.69,86.621,86.241,85.517,86.241,86.724,86.103,85.483,85.379,85.655,83.069,81.552,81.69,84.483,84.172,83.172,83.172,83.103,83.103,79.69,79.138,78.69,78.793,79.69,79.655,79.655,79.655,79.828,82.414,83.034,83.034,82.897,80.759,81.207,80.793,81.862,82.897,83.069,79.69,83.138,84.69,84.586,85.138,84.621,83.069,83.103,83.172,83.172,83.172,83.724,84.172,84.069,86.552,86.552,86.552,86.552,86.552,85.241,83.31,83.31,83.828,86.69,86.655,86.379,85.379,84.517,83.379,86.793,86.759,86.379,86,84.897,84.207,86.828,86.793,86.793,83.414,86.828,86.828,86.828,86.828,86.828,86.828,86.828,86.828,86.828,84.759,86.724,86.724,86.724,86.724,86.724,86.31,84.724,85.552,84.379,84.31,84,84.414,84.379,85.483,86.931,86.724,86.724,86.69,89.621,89.448,88.69,89.414,89.345,88.793,88.586,88.586,88.414,87.828,91.034,90.69,90.379,90.138,90.138,89.69,89.724,92.586,92.379,94.414,94.517,93.655,93.69,93.138,93.172,93.241,93.379,93.379,93.414,93.862,94.034,92.207,90.69,87.621,88,88.241,88.138,88.276,88.966,89.034,89.345,89.69,89.931,89.897,90.345,87.655,87.379,87.621,87.897,87.828,89,88.931,89.897,89.862,89.862,89.862,88.034,87.724,87.621,87.448,89.069,86.448,89.586,89.759,89.724,89.828,89.862,89.759,89.862,89.862,89.828,89.828,89.862,89.862,93.207,92.966,92.276,89.931,88.172,88.069,87.276,86.897,86.586,86.655,86.655,88.034,87.966,87.931,87.138,87.828,90.724,90.207,90.172,88.655,88.345,88.414,87.862,87.414,86.966,86.931,86.931,86.345,86.31,86,85.345,84.621,86.931,88.69,88.793,89.586,89.966,90.31,90.276,87.931,88.172,88.586,86.417,84.044,80.972,78.145,75.32,72,70.201,67.165,63.739,60.952,58.059,55.167,51.842,48.478,45.73,42.397]
Loading

95th Percentile Latency

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Latency, ms" 35 --> 46
    line [40.35,40.339,40.302,40.351,40.377,40.481,40.44,40.464,40.367,40.333,40.391,40.388,40.457,40.472,40.44,40.375,40.35,40.351,40.381,40.319,40.267,40.216,40.22,40.105,40.146,40.246,40.358,40.465,40.311,40.314,40.342,40.435,40.436,40.218,40.206,40.199,40.21,40.326,40.295,40.207,40.231,40.229,40.336,40.201,40.198,40.261,40.318,40.365,40.334,40.398,40.444,40.341,40.463,40.479,40.285,40.159,40.257,40.39,40.364,40.193,40.124,40.136,40.309,40.307,40.236,40.128,40.084,40.121,40.276,40.273,40.238,40.205,40.359,40.415,40.32,40.279,40.253,40.324,40.241,40.216,40.198,40.248,40.239,40.313,40.352,40.285,40.31,40.326,40.374,40.349,40.339,40.377,40.51,40.56,40.697,40.757,40.708,40.719,40.679,40.778,40.753,40.66,40.64,40.668,40.729,40.712,40.684,40.718,40.739,40.943,40.991,40.997,41.114,41.18,41.121,41.136,41.064,41.198,41.13,41.12,41.141,41.02,41.069,41.062,41.008,40.985,40.993,41.057,41.006,40.867,40.947,40.931,41.037,40.984,41.086,41.112,41.032,41.017,40.967,40.859,40.839,40.81,40.763,40.81,40.71,40.775,40.748,40.942,40.736,40.621,40.681,40.592,40.59,40.517,40.55,40.578,40.367,40.367,40.577,40.596,40.638,40.505,40.489,40.276,40.167,40.237,40.224,40.015,40.006,40.033,39.988,39.919,39.946,39.939,39.942,39.867,39.812,39.993,40.057,39.9,39.92,39.852,39.928,39.969,39.886,39.863,39.884,39.891,39.873,39.81,39.772,39.801,39.788,39.879,39.955,39.938,40.136,40.251,40.198,40.006,40.018,39.963,40.064,40.017,40.094,40.123,40.055,40.09,40.22,40.17,40.24,40.277,40.279,40.251,40.396,40.268,40.214,40.172,40.153,40.218,40.272,40.394,40.402,40.259,40.253,40.127,39.992,40.015,40.168,40.118,40.179,40.16,40.279,40.281,40.286,40.378,40.35,40.311,40.374,40.305,40.258,40.225,40.318,40.341,40.49,40.502,40.372,40.392,40.364,40.333,40.371,40.278,40.308,40.33,40.499,40.555,40.467,40.57,40.598,40.563,40.618,40.556,40.472,40.493,40.528,40.569,40.602,40.645,40.704,40.577,40.63,40.595,40.712,40.666,40.681,40.817,40.83,40.817,40.818,40.754,40.799,40.957,41.025,40.996,40.919,40.929,40.876,40.805,40.826,40.784,40.823,40.794,40.711,40.651,40.612,40.527,40.276,40.356,40.438,40.42,40.475,40.371,40.398,40.363,40.39,40.285,40.163,40.135,40.138,40.19,40.122,39.987,39.815,39.869,39.88,39.96,40.02,40.001,39.857,39.77,39.845,39.865,39.859,39.864,39.984,40.115,39.989,40.082,40.046,39.932,39.825,39.75,39.742,39.691,39.76,39.839,39.861,39.961,39.927,39.777,39.827,39.909,39.881,39.896,39.826,39.905,39.846,39.972,39.921,39.708,39.706,39.728,39.742,39.76,39.678,39.643,39.604,39.651,39.614,39.67,39.671,39.622,39.584,39.577,39.63,39.42,39.281,39.28,39.311,39.268,39.297,39.413,39.447,39.424,39.263,39.289,39.311,39.387,39.512,39.546,39.612,39.643,39.609,39.73,39.807,39.803,39.783,39.74,39.75,39.828,39.938,40.021,40.036,39.907,40.106,40.159,40.11,40.232,40.274,40.138,40.117,40.127,40.136,40.117,40.106,40.096,40.096,40.176,40.141,40.13,40.12,40.105,40.04,39.948,39.933,39.998,40.116,40.23,40.161,40.083,40.028,39.953,40.14,40.113,40.073,40.124,40.018,39.928,40.077,40.059,40.084,39.972,40.146,40.229,40.239,40.255,40.274,40.349,40.337,40.3,40.377,40.306,40.379,40.409,40.409,40.372,40.173,40.203,40.073,40.067,40.07,40.013,40.017,40.154,40.165,40.263,40.378,40.381,40.327,40.307,40.447,40.499,40.429,40.401,40.39,40.364,40.295,40.359,40.368,40.217,40.323,40.324,40.349,40.309,40.453,40.471,40.475,40.708,40.756,40.82,40.823,40.85,40.838,40.813,40.773,40.711,40.709,40.8,40.805,40.727,40.677,40.57,40.512,40.414,40.438,40.509,40.475,40.444,40.523,40.516,40.526,40.47,40.457,40.348,40.407,40.241,40.224,40.284,40.334,40.415,40.42,40.378,40.38,40.43,40.544,40.535,40.422,40.493,40.613,40.612,40.743,40.659,40.71,40.734,40.732,40.711,40.723,40.719,40.836,40.806,40.854,40.897,40.967,41.083,41.273,41.288,41.2,41.043,40.968,40.981,40.898,40.921,40.888,40.865,40.925,40.99,40.901,40.886,40.839,40.822,40.951,40.955,40.911,40.751,40.691,40.714,40.708,40.655,40.674,40.675,40.639,40.533,40.56,40.571,40.476,40.467,40.624,40.632,40.637,40.692,40.712,40.739,40.71,40.597,40.657,40.736,40.701,40.746,40.764,40.78,40.818,40.862,41.017,41.09,41.102,41.15,41.137,41.158,41.134,41.149,41.263,41.045]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "Time, m" 0 --> 10
    y-axis "Latency, ms" 35 --> 47
    line [40.587,40.603,40.568,40.928,41.038,40.978,41.241,41.351,41.264,41.189,41.147,41.192,40.996,41.086,41.142,41.118,41.103,41.124,41.152,41.342,41.379,41.386,41.327,41.503,41.498,41.591,41.707,41.736,41.664,41.523,41.612,41.636,41.307,41.212,41.208,41,40.818,40.731,40.89,40.938,40.907,41.174,41.2,41.2,41.039,40.876,40.889,40.7,40.627,40.585,40.779,40.858,41.02,41.074,40.98,40.873,40.81,40.973,41.019,40.929,40.932,41.109,41.03,41.104,41.222,41.354,41.464,41.244,41.249,41.076,40.996,40.973,40.895,41.059,40.959,40.801,41.022,41.019,41.037,41.037,40.967,40.738,40.758,40.722,40.753,40.793,40.89,40.945,40.841,40.644,40.47,40.653,40.724,40.643,40.71,40.766,40.878,40.786,41.212,41.233,41.076,41.088,41.12,41.215,41.115,41.092,41.041,41.065,40.88,40.863,41.089,41.125,41.294,41.308,41.287,41.17,41.184,41.313,41.499,41.599,41.588,41.427,41.323,41.229,41.229,41.167,41.278,41.068,40.884,40.966,40.981,40.967,40.978,41.18,41.284,41.275,41.291,41.191,41.291,41.06,41.06,41.045,40.838,41.061,41.032,41.045,41.076,41.231,41.075,40.851,41.016,41.118,41.095,40.964,41.079,40.996,40.928,41.075,41.016,41.111,41.226,41.18,41.016,40.85,40.756,40.728,40.886,40.867,41.021,40.841,40.621,40.759,40.348,40.416,40.303,40.338,40.211,40.442,40.559,40.47,40.652,40.73,40.765,40.609,40.6,40.596,40.731,40.829,40.996,40.529,40.514,40.753,40.811,40.947,41.113,40.934,41.019,41.047,41.316,41.404,41.522,41.774,41.629,41.796,41.704,41.693,41.614,41.589,41.538,41.517,41.496,41.475,41.517,41.538,41.413,41.328,41.264,41.065,41.176,41.257,41.076,41.039,41.032,40.993,41.061,41.045,40.993,40.726,40.703,40.604,40.529,40.646,40.6,40.58,40.603,40.443,40.475,40.677,40.603,40.603,40.627,40.461,40.636,40.73,40.825,40.89,40.912,40.844,40.776,41.007,41.188,41.197,41.154,41.209,41.232,41.204,41.31,41.387,41.573,41.43,41.469,41.566,41.565,41.52,41.676,41.834,41.738,41.688,41.668,41.725,41.981,41.819,42.005,41.891,41.948,41.811,42.018,42.035,42.049,41.914,41.906,41.951,41.95,41.888,41.918,42.128,42.034,41.839,41.944,41.937,41.714,41.708,41.731,41.821,41.687,41.676,41.689,41.731,41.57,41.405,41.489,41.468,41.426,41.51,41.64,41.565,41.56,41.577,41.599,41.364,41.439,41.527,41.534,41.41,41.405,41.448,41.344,41.44,41.454,41.51,41.502,41.436,41.202,41.347,41.247,41.163,41.059,41.348,41.389,41.224,41.011,41.085,40.939,40.893,40.829,40.682,40.61,40.833,41.122,40.908,40.92,40.957,40.978,40.727,40.541,40.635,40.345,40.35,40.263,40.227,40.229,40.24,40.221,40.307,40.373,40.541,40.349,40.185,40.296,40.418,40.337,40.355,40.246,40.232,40.406,40.32,40.007,39.874,39.902,39.797,39.774,39.744,39.841,39.929,39.951,39.993,40.021,40.272,40.243,40.415,40.604,40.248,39.993,39.898,39.732,39.732,40.04,40.049,40.022,40.103,39.968,40.151,40.323,40.328,40.524,40.521,40.402,40.692,40.654,40.528,40.724,40.439,40.491,40.465,40.677,40.705,40.385,40.479,40.38,40.437,40.62,40.669,40.622,40.621,40.569,40.32,40.19,40.354,40.54,40.478,40.477,40.345,40.309,40.155,39.979,40.137,39.788,39.634,39.838,39.665,40.041,40.091,40.146,40.036,40.008,40.145,40.064,40.036,40.169,40.27,40.42,40.555,40.672,40.788,40.833,40.967,40.901,40.717,40.798,40.612,40.622,40.678,40.66,40.803,40.713,40.922,40.945,40.982,41.004,40.853,40.855,40.719,40.754,40.7,40.774,40.765,40.764,40.674,40.46,40.33,40.32,40.285,40.246,40.297,40.393,40.534,40.577,40.527,40.681,40.734,40.745,40.682,40.635,40.712,40.486,40.589,40.374,40.259,40.208,40.041,40.143,40.096,40.235,40.177,40.309,40.356,40.119,40.278,40.373,40.366,40.235,39.996,40.02,39.999,39.658,39.61,39.864,39.89,39.938,40.077,39.97,40.206,40.224,40.327,40.278,40.327,40.381,40.535,40.844,40.928,40.961,40.885,41.035,40.933,40.806,41.052,40.986,40.954,40.901,41.111,41.373,41.46,41.241,41.408,41.315,41.062,41.13,41.014,40.78,40.904,40.832,40.873,40.974,41.055,41.033,41.047,41.011,40.903,40.876,40.908,40.78,40.878,40.748,40.842,40.598,40.708,40.722,40.747,40.703,40.688,40.587,40.677,40.604,40.614,40.688,40.525,40.583,40.696,40.708,40.743,40.633,40.502,40.692,40.806,40.865,40.894,40.866,40.838,40.764,40.915,40.641,40.841,40.846,40.965,40.899,40.961,41.081,40.956,40.799,40.813,40.931,40.849]
Loading

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 17, 2025 •
edited
Loading

🌋 Here are results of SLO test for sync-table:

Operation Success Rate

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Success Rate, %" 89 --> 111
    line [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,99.982,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "	Time, m" 0 --> 10
    y-axis "Success Rate, %" 89 --> 111
    line [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100]
Loading

Operations Per Second

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Operations" 821 --> 1094
    line [914.461,947.603,980.813,985.483,984.483,986.207,984.759,985.931,985.897,985.828,982.862,984.793,986,986.862,985.379,987.862,988,984.207,983.828,982.69,982.414,982.483,981.966,981.828,981.069,967,968.034,968,968.379,966.379,965.828,965.621,965.586,964.31,962.276,963.655,963,963.379,962.897,965.207,964.621,964.586,964.034,965.034,964.517,964.621,968.31,968.897,969.621,967.207,964.759,965.034,965.828,966.862,983.931,983.759,984.207,983.483,984.517,984.759,985.241,985.655,987.241,989.207,988.552,987.31,985.862,986.138,987.138,987.345,987.31,987.138,987.207,986.069,982.379,981.69,981.586,981.828,983.931,985.862,985.862,985.793,986.069,986.172,981.897,968.966,941.966,942.759,942.897,942.276,942.552,942.207,942.103,942.793,944.034,944.034,943.552,942.828,943.483,943.448,942.517,943.034,943.862,947.483,948.31,948.172,948.276,947.172,946.724,947.655,947.414,947.103,946.655,947.483,959.724,987.862,988,988.414,988.621,988.448,989.448,986.172,985.828,986.379,987.241,987.759,987.897,987.655,987.586,989.069,989.034,986.793,986.862,986.172,986.414,986.069,987.586,988.552,988.448,987.379,988,987.621,991,988.828,989.828,987.276,985.862,985.931,985.759,985.69,989.207,989.172,988.207,988.793,988.793,987.483,987.379,987.379,987.172,986.966,988.862,988.621,989.069,989.103,989.241,987.724,987.69,987,988.241,985.345,984.207,983.724,986.483,985.483,987.379,988.759,989.345,988.828,988.793,988.759,988.69,989.172,989.069,988.241,990.241,990.655,990.552,990.828,988.414,989.103,989.448,989.345,988.862,989.276,990.724,990.345,990.414,988.655,991.448,993.034,993.448,993.69,992.931,993.241,993.31,992.655,992.448,992.448,992.31,992.724,992.69,992.759,992.897,991.552,991.897,991.828,992.207,993.241,991.172,990.966,990.517,990.552,990.552,990.138,990.621,990.103,991.069,989.759,989.724,990.103,989.138,989.724,990.138,989.621,990.069,990.759,990.862,990.655,990.931,989.759,989.207,989.207,990.172,989.345,989.345,988.966,989.138,988.379,988.138,989.241,989.483,988.448,988.897,987.31,987.276,987.966,989.31,989.655,989.379,989.069,989.207,988.759,988.862,988.69,988.621,988.138,988.966,988.379,986.586,987.345,987.759,987.793,987.931,988.345,987.759,988.862,992,992.138,990.724,990.793,991.621,991.759,993.276,993.966,993.897,993.931,992.103,991.448,992.207,992.034,992.103,990.172,989.31,989.379,989.345,989.276,989.276,992.448,990.931,991.034,990.759,990.793,991,991.483,991.69,990.897,990.207,990.483,990.586,990.31,990.483,990.414,989.759,989.621,989.414,990.345,991.276,991.103,991.069,991.172,993.552,994.414,991.586,992.207,991.897,991.828,991.276,992.586,992.448,992.655,992.931,992.69,992.414,992.724,992.586,992.414,992.966,992.414,992.759,992.103,992.31,992.138,992.276,992.552,993.138,992.828,993.586,993.345,992.517,991.552,991.414,994.483,993.931,993.655,994.103,993.724,992.552,992.414,992.966,992.69,992.069,992.517,990.897,991.138,992.172,991.931,991.862,990.793,992.414,992.69,992.586,992.31,992.379,992.379,992.31,992.172,992.241,990.586,991.448,991.759,990.759,990.69,991.138,989.897,990.586,991.69,991.931,991.379,991.552,992.207,989.931,990.828,991.586,991.31,991.621,991.862,992.897,991.966,991.552,990.345,990.345,990.483,990.69,990.724,990.345,989.414,992.31,992.138,992.207,992.793,993.345,993.034,990.414,990.483,990.483,989.379,989.448,989.103,988.483,990.586,991.034,990.655,990.483,990.517,989.793,989.414,989,988.931,990.931,990.897,990.207,989.897,989.828,989.759,991.241,991.276,991.103,990.966,987.034,986.552,986.414,989.69,989.172,987.621,988.69,988.655,989,989.621,989.759,989.138,989.069,988.138,988.345,988.862,989.172,988.724,989.276,989.034,989.276,989.655,989.517,989.414,989.276,988.828,988.379,984.793,984,987.103,984.241,984.069,984.172,983.586,985.69,985.517,985.759,985.621,985.31,984.897,985.379,984.793,984.034,984.069,984.345,983.828,984.586,984.069,983.862,983.966,982.103,982.034,982.621,982.655,982.414,981.759,986,986.448,987.414,987.655,987.897,988.31,989.345,988.793,988.759,987.448,987.103,983.724,983.414,982.276,982.276,984.31,983.655,983.966,984.483,984.414,983.31,983.552,983.379,985,985.828,985.69,985.069,985.172,986,985,985.414,985.069,987.448,987.414,987.69,987.483,987.517,987.793,988.828,989.31,992.345,992.655,994.103,994.517,992.241,991.897,991.69,990.379,990.379,983.207,983.276,983.379,983.379,983.207,982.483,983.828,983.276,980.897,981.621,981.828,982.276,978.379,982.483,982.069,982.207,981.793,981.862,981.655,981.828,981.655,981.379,981.241,981.069,983.207,984.207,983.69,984.828,984.241,991.034,989.414,945.853,912.715]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "Time, m" 0 --> 10
    y-axis "Operations" 70 --> 103
    line [86.906,89.628,92.934,90.034,90.034,90.069,89.966,90,90,89.931,88.759,88.448,88.379,88.276,87.552,87.621,89.69,89.655,90.034,90,87.31,86.621,86.655,86.655,86.655,85.276,84.414,84.655,81.552,82.517,83.138,83.172,86.552,86.552,85.897,83.172,83.207,83.207,83.207,84.379,84.69,81.414,81.448,78.793,79.724,79.793,79.724,79.724,79.724,82.414,83.103,79.69,79.724,79.724,81.103,83.069,83.138,86.517,86.517,84.172,83.172,83.172,83.172,83.828,86.552,86.517,84.138,83.138,79.724,79.724,83.069,83.138,86.517,86.517,86.517,83.172,83.172,83.172,83.172,83.172,86.552,86.517,86.517,86.483,83.069,83.069,83.069,83.069,85.414,86.414,83.069,83.069,83.069,83.069,83.069,84.69,83.069,84.69,84.655,84.69,83.931,84.034,83.448,83.241,86.483,86.517,86.448,86.483,83.069,83.069,83.069,79.931,79.793,83.207,83.103,83.172,83.172,83.138,83.103,86.552,86.517,86.448,83.103,83.103,83.862,86.483,88.276,88.31,88.276,89.034,88.931,89.517,86.379,86.483,86.448,86.517,86.483,89.897,89.897,89.897,93.034,93.207,93.207,93.31,93.241,93.241,93.276,90.069,89.897,86.552,86.621,89.966,89.966,89.966,89.931,89.138,89,88.966,88.69,87.448,87.207,89.931,89.897,86.517,86.517,86.517,86.517,86.517,86.517,85.241,85.138,84.897,83.103,83.103,83.103,83.103,86.345,86.414,88.621,86.552,86.586,86.621,86.621,86.759,87.552,87.69,87.724,88,87.31,86.103,83.448,83.345,86.724,86.724,86.724,86.724,86.724,85.621,86.793,86.552,85.69,86.897,86.828,84.793,84.793,84.483,84.31,85.517,87.655,87.414,87.414,87.414,86.586,86.586,86.621,86.655,86.655,88.586,89.276,92.483,92.448,92.345,90.897,90.138,90.103,90.138,91.138,90.828,91.103,92.172,92.034,90.103,92.069,88.69,89,89.172,89.138,89.069,89.276,89.241,85.897,86.621,86.621,86.586,86.552,86.552,86.552,87.31,84.793,84.345,81.034,82.483,83.241,83.276,83.241,83.345,83.759,83.828,83.862,84,83.207,81.897,85.034,85.034,84.966,83.897,83.31,83.31,83.069,86.517,86.414,86.379,86.345,86.207,85.931,85.862,85.862,86.103,86.724,90.138,90.138,90,90,89.966,89.931,86.552,86.552,86.552,87.138,89.931,91.31,91.552,89.517,89.655,90.31,90.034,90,90.241,87.621,87.621,87.517,87.517,87.207,87.448,87.517,84.172,86.483,86.483,86.483,86.483,86.621,86.621,86.655,86.69,90.069,90.069,86.828,86.828,86.828,86.828,86.828,88.862,87.759,85.862,86.724,86.759,86.759,89.276,89.379,89.517,89.552,86.621,86.655,86.655,90,86.621,86.621,86.621,86.621,86.621,86.621,86.621,86.621,86.621,86.621,89.862,89.862,89.862,89.862,89.862,87.483,87.724,90.069,90.034,86.621,86.655,86.069,85.966,83.241,83.241,86.655,83.241,83.276,83.276,86.69,86.517,86.31,86.069,83.276,79.862,79.862,79.862,79.862,79.862,79.862,79.862,79.862,79.379,79.379,79,79.793,79.759,79.793,83.207,83.172,83.759,83.862,83.379,83.379,83.345,86.31,86.207,86.345,83.345,83.517,83.724,83.966,86.759,90.172,89.931,86.793,86.828,86.828,86.828,86.207,86,84.276,84.034,84.414,84.31,83.655,83.448,83.414,83.414,80,78.621,81.793,81.138,80.448,80.759,80.69,80.31,83.345,83.31,80.724,80.552,80.241,80,80.207,83.345,80.069,80,80.034,80.552,80.793,83,83.31,85.621,85.69,86.345,83.276,83.31,83.31,86.724,88.103,88.138,85.448,86.138,86.276,86.414,86.655,85.655,83.276,82.655,82.724,82.931,83.31,83.241,83.241,86.483,86.517,83.138,83.241,83.207,83.241,83.172,83.241,83.276,83.276,86.552,86.552,84.31,83.345,83.31,83.31,83.241,83.241,83.241,83.241,83.241,84.207,86.621,89.828,89.931,86.655,86.552,86.655,86.655,86.655,86.69,86.897,86.655,86.69,86.172,86.207,86.034,83.724,83.655,83.379,83.345,85.621,86.552,83.207,83.207,86.621,86.621,86.621,86.621,86.586,83.517,83.241,80.345,80.586,83.759,83.241,83.241,83.172,83.172,83.138,86.31,86.552,86.483,85.483,85.379,83.862,86.138,82.828,83.103,83.138,83.138,83.172,86.552,86.552,83.172,81.793,81.621,81.655,81.517,84.034,83.276,86.207,85.966,86.172,86.759,83.345,83.414,83.414,83.448,83.414,83.414,80.103,81.586,81.655,83.345,83.379,86.759,86.759,86.759,86.724,86.724,86.724,86.724,90.103,91.483,91.655,90.517,90.517,90.793,90.345,90.31,90.31,90.31,89.103,90.345,86.058,82.716]
Loading

95th Percentile Latency

---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=read"
    x-axis "Time, m" 0 --> 10
    y-axis "Latency, ms" 13 --> 20
    line [16.525,16.524,16.532,16.393,16.424,16.367,16.326,16.414,16.389,16.341,16.365,16.389,16.381,16.359,16.334,16.294,16.346,16.373,16.38,16.36,16.273,16.282,16.285,16.286,16.252,16.346,16.341,16.361,16.267,16.283,16.301,16.306,16.373,16.411,16.462,16.431,16.391,16.426,16.436,16.425,16.394,16.28,16.283,16.161,16.182,16.166,16.124,16.15,16.086,16.159,16.151,16.032,16.026,16,15.808,15.807,15.731,15.807,15.794,15.671,15.64,15.638,15.618,15.61,15.667,15.59,15.469,15.565,15.482,15.387,15.581,15.57,15.709,15.678,15.667,15.536,15.432,15.512,15.45,15.395,15.432,15.38,15.354,15.301,15.277,15.396,15.414,15.357,15.529,15.617,15.549,15.633,15.667,15.768,15.907,16.008,15.899,15.972,16.14,16.174,16.319,16.428,16.54,16.637,16.757,16.871,16.962,17.083,17.044,17.072,17.117,17.124,17.24,17.356,17.41,17.436,17.509,17.551,17.559,17.656,17.654,17.683,17.613,17.629,17.643,17.694,17.716,17.743,17.686,17.587,17.557,17.549,17.467,17.421,17.369,17.282,17.24,17.342,17.356,17.323,17.385,17.356,17.304,17.239,17.215,17.238,17.244,17.183,17.161,17.045,17.049,17.159,17.111,17.094,17.048,17.022,16.959,17.028,17.075,17.053,17.066,17.176,17.213,17.138,17.177,17.187,17.208,17.259,17.33,17.303,17.349,17.399,17.4,17.398,17.329,17.24,17.234,17.202,17.272,17.183,17.184,17.179,17.193,17.228,17.263,17.225,17.213,17.269,17.248,17.186,17.025,17.063,17.181,17.213,17.197,17.163,17.164,17.128,17.153,17.13,17.088,17.078,17.073,17.036,17.058,17.139,17.107,17.125,17.161,17.091,17.074,17.1,17.072,17.053,17.089,17.117,17.112,17.199,17.218,17.337,17.323,17.325,17.287,17.305,17.308,17.297,17.33,17.376,17.321,17.337,17.334,17.315,17.376,17.293,17.252,17.293,17.285,17.363,17.412,17.443,17.348,17.353,17.356,17.37,17.394,17.378,17.392,17.456,17.367,17.366,17.274,17.306,17.284,17.294,17.307,17.338,17.319,17.397,17.38,17.385,17.322,17.298,17.374,17.405,17.419,17.409,17.356,17.4,17.401,17.472,17.481,17.479,17.436,17.347,17.309,17.213,17.146,17.156,17.153,17.254,17.229,17.239,17.214,17.18,17.104,16.969,16.951,16.953,16.985,17.079,17.092,17.183,17.14,17.165,17.216,17.165,17.141,17.167,17.103,17.117,17.121,17.183,17.205,17.21,17.27,17.181,17.225,17.244,17.28,17.265,17.276,17.295,17.305,17.338,17.448,17.416,17.339,17.343,17.339,17.318,17.221,17.199,17.114,17.048,17.094,17.043,16.973,17.038,17.017,17,16.928,16.854,16.84,16.765,16.834,16.763,16.784,16.749,16.863,16.822,16.759,16.677,16.618,16.51,16.423,16.423,16.251,16.12,16.181,16.248,16.223,16.292,16.372,16.346,16.176,16.151,16.129,16.138,16.125,16.216,16.308,16.278,16.302,16.288,16.302,16.221,16.184,16.069,16.051,15.995,16.205,16.328,16.457,16.575,16.679,16.801,16.899,16.869,16.905,16.895,16.935,16.932,16.975,17.12,17.158,17.172,17.182,17.121,17.172,17.226,17.248,17.268,17.272,17.261,17.26,17.25,17.221,17.211,17.305,17.198,17.062,17.092,17.111,17.111,17.093,17.143,17.135,17.12,17.12,17.071,17.032,17.066,17.084,17.176,17.11,17.085,17.157,17.044,16.961,16.929,16.869,16.847,17.017,17.107,17.12,17.137,17.153,17.146,17.153,17.254,17.135,17.075,17.038,16.97,16.869,16.822,16.688,16.658,16.75,16.793,16.655,16.624,16.498,16.542,16.511,16.63,16.57,16.647,16.731,16.76,16.821,16.704,16.557,16.454,16.558,16.609,16.619,16.622,16.591,16.633,16.689,16.605,16.7,16.77,16.818,16.961,17.046,17.024,17.084,17.155,17.18,17.157,17.21,17.27,17.209,17.211,17.205,17.193,17.221,17.238,17.289,17.372,17.465,17.351,17.263,17.247,17.253,17.236,17.338,17.304,17.313,17.327,17.287,17.255,17.179,17.132,17.009,16.986,17.048,17.034,17.114,17.107,16.98,17.024,17.16,17.206,17.236,17.27,17.286,17.209,17.15,17.024,17.062,17.161,17.178,17.263,17.281,17.23,17.252,17.339,17.308,17.324,17.37,17.453,17.493,17.525,17.426,17.351,17.32,17.247,17.168,17.193,17.088,16.959,16.901,16.899,16.908,16.88,16.992,17.026,17.208,17.31,17.328,17.246,17.158,17.184,17.189,17.251,17.273,17.263,17.178,17.227,17.252,17.282,17.347,17.429,17.443,17.445,17.518,17.615,17.656,17.685,17.766,17.743,17.713,17.637,17.638,17.618,17.619,17.544,17.439,17.447,17.506,17.51,17.487,17.463]
Loading 
---
config:
    xyChart:
        width: 1200
        height: 400
    themeVariables:
        xyChart:
            titleColor: "#222"
            backgroundColor: "#fff"
            xAxisLineColor: "#222"
            yAxisLineColor: "#222"
            plotColorPalette: "#FF7F0E,#1F77B4,#D62728,#2CA02C,#9467BD,#8C564B,#E377C2,#7F7F7F,#BCBD22,#17BECF"
---
xychart-beta
    title "operation_type=write"
    x-axis "Time, m" 0 --> 10
    y-axis "Latency, ms" 16 --> 39
    line [19.582,19.588,19.582,19.542,19.545,19.544,19.5,19.507,19.484,19.47,19.489,19.51,19.486,19.475,19.46,19.385,19.365,19.373,19.378,19.377,19.369,19.383,19.378,19.384,19.38,19.415,19.435,19.422,19.414,19.41,19.432,19.419,19.419,19.437,19.452,19.473,19.468,19.476,19.474,19.495,19.473,19.473,19.47,19.47,19.448,19.454,19.461,19.45,19.421,19.427,19.442,19.432,19.421,19.402,19.345,19.296,19.271,19.232,19.22,19.164,19.163,19.174,19.196,19.217,19.215,19.177,19.162,19.269,19.249,19.202,19.213,19.205,19.185,19.195,19.199,19.164,19.13,19.148,19.107,19.05,18.992,18.973,18.941,18.882,18.904,18.977,19.034,19.009,18.983,19.001,18.954,18.992,19.039,19.07,19.219,19.266,19.142,19.174,19.295,19.295,19.365,19.479,19.538,19.609,19.667,19.811,19.902,19.996,20.887,21.128,21.365,22.707,25.36,27.127,28.019,28.205,28.335,29.095,29.104,29.538,29.981,30.312,30.668,30.873,31.482,32.338,32.056,32.297,32.056,31.731,31.401,31.457,31.49,31,29.997,28.975,28.128,28.631,28.631,28.395,28.124,27.723,26.009,25.548,24.807,25.268,25.848,26.398,26.443,25.71,25.846,26.137,25.991,25.231,23.568,24.15,23.3,23.851,24.107,24.64,25.043,26.146,27.396,27.862,28.616,28.371,27.328,28.616,29.434,29.848,29.539,30.251,30.459,30.974,31.366,30.771,29.914,29.459,29.385,29.199,29.641,29.182,28.95,28.798,28.723,28.57,29.148,29.529,30.115,29.42,28.647,28.922,28.925,28.565,29.272,29.718,29.385,29.976,29.592,30.391,30.173,30.312,30.924,30.889,31.181,32.583,32.782,32.617,32.265,31.719,31.451,31.541,31.446,31.354,31.254,31.058,31.246,30.923,31.239,31.63,30.941,30.688,31.081,31.592,31.938,32.177,31.814,33.115,32.229,31.857,31.885,31.685,31.712,31.802,31.022,30.892,31.358,32.145,32.348,32.823,33.093,33.253,33.401,33.695,34.247,33.91,34.312,34.306,34.032,34.519,34.871,34.924,34.459,34.25,33.978,34.168,33.287,34.007,34.208,34.385,34.263,34.443,34.192,34.458,34.339,33.993,33.452,33.821,33.94,33.778,33.797,33.52,33.598,32.798,32.853,32.129,31.957,31.993,31.507,31.592,31.763,31.875,31.791,31.627,31.01,30.791,30.791,30.888,31.142,30.729,30.433,30.573,30.234,30.5,30.648,30.22,29.82,29.23,29.285,29.397,29.963,30.172,30.345,30.492,31.06,31.126,31.095,31.559,31.649,31.376,30.677,30.776,30.962,31.332,30.888,30.112,30.016,29.473,29.473,29.247,29.247,28.877,28.909,29.365,29.934,29.818,30.032,30.386,29.855,29.613,29.279,29.522,29.514,28.823,28.25,28.22,27.437,26.741,27.965,28.712,27.965,27.437,26.741,27.024,26.741,26.309,25.102,24.617,24.283,24.113,24.46,23.863,25.045,24.894,24.541,24.182,24,25.398,25.367,25.534,25.201,24.854,24.309,23.939,22.871,22.728,22.989,20.52,20.307,21.05,23.07,24.642,25.362,25.875,26.527,27.873,28.817,29.446,30.155,30.25,30.609,30.508,31.238,30.849,30.755,30.198,30.174,30.071,30.81,31.408,31.596,32.143,32.366,33.058,32.944,32.981,33.09,32.923,33.165,32.917,32.76,33.213,33.507,33.72,33.624,33.664,33.486,33.459,33.233,33.48,33.074,32.206,32.884,33.279,33.585,33.558,33.137,32.613,31.961,31.214,30.169,29.922,30.612,31.51,31.518,31.654,31.434,30.773,29.948,29.632,28.632,27.692,26.79,25.324,24.158,23.256,20.774,19.998,21.974,22.382,22.763,20.774,19.936,19.886,19.791,19.952,19.922,21.397,22.404,23.528,24.87,24.125,23.75,23.755,24.836,25.625,25.84,25.698,25.698,25.412,26.911,27.397,27.785,28.583,29.069,29.9,30.213,28.437,28.308,28.362,28.848,29.396,30.296,30.723,29.64,29.657,29.657,29.771,30.321,29.883,29.762,29.297,29.648,28.387,28.084,27.853,28.211,28.084,29.174,27.686,27.5,28.336,27.947,27.95,27.679,27.59,27.927,28.21,28.151,28.29,28.219,28.486,28.583,29.551,30.063,29.85,30.168,29.85,30.28,30.038,30.942,30.583,30.948,30.924,31.619,32.591,32.522,31.91,32.444,32.856,32.65,32.9,33.25,33.42,33.714,33.272,33.005,32.786,32.097,31.735,30.958,30.184,28.848,28.205,27.762,27.947,28.341,28.378,29.111,28.437,29.396,30.421,30.678,29.377,28.547,29.147,29.729,30.055,29.384,29.729,29.623,30.392,31.403,31.689,32.729,32.447,32.767,32.767,33.454,33.878,34.082,34.281,34.132,33.691,33.387,33.173,33.245,32.903,32.838,32.143,30.93,31.556,32.22,32.136,32.17,31.94]
Loading

@vgvoleg vgvoleg force-pushed the fix_stream_error_handling branch from b5078cb to 5c25731 Compare September 18, 2025 06:34
@vgvoleg vgvoleg requested a review from Copilot September 18, 2025 09:53
Copilot
Copilot AI reviewed Sep 18, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances error handling for YDB topic streams by introducing better error logging and fixing connection lifecycle management. The changes focus on improving observability of topic reader errors and ensuring proper cleanup of disconnected connections.

  • Improves error logging in topic stream connections with debug-level messages for better troubleshooting
  • Fixes connection cleanup by properly exposing and calling the disconnection handler when streams encounter errors
  • Adds async support to SLO test infrastructure for topic operations

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
ydb/pool.py Adds parameter to expose disconnection handler for sync pool operations
ydb/aio/pool.py Adds parameter to expose disconnection handler for async pool operations
ydb/_grpc/grpcwrapper/common_utils.py Implements proper disconnection handler calling in stream error scenarios
ydb/_topic_reader/topic_reader_asyncio.py Enhances error logging for topic reader connection loops
ydb/connection.py Reduces channel close logging from info to debug level
ydb/aio/connection.py Reduces channel close logging from info to debug level
tests/slo/src/jobs/async_topic_jobs.py New async topic job manager implementation
tests/slo/src/runners/topic_runner.py Adds async mode support to topic runner
tests/slo/src/runners/base.py Adds async interface to base runner
tests/slo/src/root_runner.py Implements async command execution infrastructure
tests/slo/src/options.py Adds async mode command line option
tests/slo/requirements.txt Adds aiolimiter dependency
tests/slo/slo_runner.sh Updates test scripts with async examples

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@vgvoleg vgvoleg force-pushed the fix_stream_error_handling branch 2 times, most recently from 75c0b18 to 03dba46 Compare September 19, 2025 09:03
@vgvoleg vgvoleg requested a review from Copilot September 19, 2025 09:25
Copilot
Copilot AI reviewed Sep 19, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

ydb/_topic_writer/topic_writer_asyncio.py
logger.debug(
"writer reconnector %s retry in %s seconds",
self._id,
err_info.sleep_timeout_seconds,
)
await asyncio.sleep(err_info.sleep_timeout_seconds)

Copy link
Preview

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving the logging statement before the sleep operation could help with debugging timing issues, but the current placement after sleep makes more sense for clarity - the log message indicates what action will be taken next. However, this creates a gap between the log message about retrying 'in X seconds' and the actual sleep, which could be confusing. Consider updating the log message to reflect that the retry is happening now.

Suggested change
logger.debug(
"writer reconnector %s retrying now",
self._id,
)

Copilot uses AI. Check for mistakes.

@vgvoleg vgvoleg force-pushed the fix_stream_error_handling branch from 03dba46 to e90fad6 Compare September 19, 2025 10:10
@vgvoleg vgvoleg merged commit 4fcf281 into main Sep 19, 2025
24 checks passed
@vgvoleg vgvoleg deleted the fix_stream_error_handling branch September 19, 2025 10:59
@vgvoleg vgvoleg changed the title Fix topic stream error handling Fix stream error handling Sep 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@vgvoleg