Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve observability of resolved-ts and safe_ts #15082

Closed
22 of 24 tasks
ekexium opened this issue Jul 6, 2023 · 0 comments
Closed
22 of 24 tasks

Improve observability of resolved-ts and safe_ts #15082

ekexium opened this issue Jul 6, 2023 · 0 comments
Assignees
@ekexium
Labels
affects-6.5 affects-7.1 severity/moderate type/bug Type: Issue - Confirmed a bug type/enhancement Type: Issue - Enhancement

Comments

@ekexium
Copy link
Contributor

ekexium commented Jul 6, 2023 •
edited
Loading

Development Task

Sometimes we suffer from resolved-ts not making progress. It's not always easy to find out the reason.
More metrics and logs can be enhanced to diagnose this.
@ekexium ekexium added the type/enhancement Type: Issue - Enhancement label Jul 6, 2023
This was referenced Jul 10, 2023
Merged
Merged
Merged
ti-chi-bot bot added a commit that referenced this issue Jul 24, 2023
@ekexium @ti-chi-bot
ctl: add get_region_read_progress (#15099)
d9087e6 
ref #15082

feat(tikv-ctl): add get_region_read_progress

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
This was referenced Jul 25, 2023
Closed
Merged
ti-chi-bot bot added a commit that referenced this issue Jul 26, 2023
@ekexium @ti-chi-bot
metrics: add min_safe_ts, min_safe_ts_region and min_safe_ts_gap (#15118
6dd40ad 
)

ref #15082

metrics: add min_safe_ts, min_safe_ts_region and min_safe_ts_gap

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ekexium added a commit to ekexium/tikv that referenced this issue Jul 26, 2023
@ekexium @ti-chi-bot
metrics: add min_safe_ts, min_safe_ts_region and min_safe_ts_gap (tik…
4b8edce 
…v#15118)

ref tikv#15082

metrics: add min_safe_ts, min_safe_ts_region and min_safe_ts_gap

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
This was referenced Jul 26, 2023
Merged
Merged
@ekexium ekexium self-assigned this Jul 27, 2023
ti-chi-bot bot added a commit that referenced this issue Jul 27, 2023
@ekexium @ti-chi-bot
metric: fix resolved-ts row (#15218)
0ad33d4 
ref #15082

Fix the unintentional changes in #15118: let resolved-ts row be collapsed by default, and put panels in it.

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Jul 28, 2023
@ekexium @ti-chi-bot
metrics: Cherry pick #15118 to release-6.5: add min_safe_ts, min_safe…
af2117e 
…_ts_region and min_safe_ts_gap (#15212)

ref #15082

metrics: add min_safe_ts, min_safe_ts_region and min_safe_ts_gap

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Jul 31, 2023
@ekexium @ti-chi-bot
ctl: add get_region_read_progress (release-6.5) (#15204)
9bb7bd6 
ref #15082

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ekexium added a commit to ekexium/tikv that referenced this issue Aug 4, 2023
@ekexium @ti-chi-bot
ctl: add get_region_read_progress (tikv#15099)
2a04cb8 
ref tikv#15082

feat(tikv-ctl): add get_region_read_progress

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ekexium ekexium mentioned this issue Aug 4, 2023
Merged
This was referenced Aug 11, 2023
Merged
Merged
ti-chi-bot bot added a commit that referenced this issue Aug 21, 2023
@ekexium @ti-chi-bot
ctl: print durations in get_region_read_progress (#15327)
2c5e7eb 
ref #15082

Record and print `duration to last update_safe_ts` and `duration to last consume leader`.

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Aug 21, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Aug 21, 2023
@ekexium @ti-chi-bot
This is an automated cherry-pick of tikv#15327
ba945c4 
ref tikv#15082

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Aug 21, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Aug 21, 2023
@ekexium @ti-chi-bot
This is an automated cherry-pick of tikv#15327
459c706 
ref tikv#15082

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ekexium added a commit to ekexium/tikv that referenced this issue Aug 21, 2023
@ekexium @ti-chi-bot
ctl: print durations in get_region_read_progress (tikv#15327)
4f1403d 
ref tikv#15082

Record and print `duration to last update_safe_ts` and `duration to last consume leader`.

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@cfzjywxk cfzjywxk mentioned this issue Aug 22, 2023
Merged
17 tasks
@ekexium ekexium mentioned this issue Aug 23, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Aug 31, 2023
@ekexium @ti-chi-bot
metrics: (release-7.1) add min_safe_ts, min_safe_ts_region and min_sa…
c7da859 
…fe_ts_gap (#15315)

ref #15082

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Aug 31, 2023
@ekexium
ctl: add get_region_read_progress (release-7.1) (#15281)
debc328 
ref #15082

Cherry pick #15099 and #15327: add get_region_read_progress command in tikv-ctl

Signed-off-by: ekexium <eke@fastmail.com>
ti-chi-bot bot added a commit that referenced this issue Aug 31, 2023
@ti-chi-bot @ekexium @ti-chi-bot
ctl: print durations in get_region_read_progress (#15327) (#15368)
e833ae8 
ref #15082

Record and print `duration to last update_safe_ts` and `duration to last consume leader`.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ekexium <eke@fastmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Sep 5, 2023
@ekexium @ti-chi-bot
metrics: more logs and metrics for resolved-ts (#15416)
9bf96f9 
ref #15082

Add more logs and metrics for resolved-ts.

Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Sep 5, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Sep 5, 2023
@ekexium @ti-chi-bot
This is an automated cherry-pick of tikv#15416
fd207d3 
ref tikv#15082

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Sep 5, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Sep 5, 2023
@ekexium @ti-chi-bot
This is an automated cherry-pick of tikv#15416
feebc82 
ref tikv#15082

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot added a commit that referenced this issue Sep 6, 2023
@ti-chi-bot @ekexium @ti-chi-bot
metrics: more logs and metrics for resolved-ts (#15416) (#15532)
6087ea6 
ref #15082

Add more logs and metrics for resolved-ts.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ekexium <eke@fastmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Oct 8, 2023
@ti-chi-bot @ekexium
metrics: more logs and metrics for resolved-ts (#15416) (#15533)
0173ee1 
ref #15082

Add more logs and metrics for resolved-ts.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: ekexium <eke@fastmail.com>

Co-authored-by: ekexium <eke@fastmail.com>
@ekexium ekexium closed this as completed Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ekexium ekexium

Labels
affects-6.5 affects-7.1 severity/moderate type/bug Type: Issue - Confirmed a bug type/enhancement Type: Issue - Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ekexium