metrics: show error time in changefeed error details panel (#5086)#5104
Open
ti-chi-bot wants to merge 1 commit into
Open
metrics: show error time in changefeed error details panel (#5086)#5104ti-chi-bot wants to merge 1 commit into
ti-chi-bot wants to merge 1 commit into
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added
do-not-merge/hold
|
This cherry pick PR is for a release branch and has not yet been approved by triage owners. To merge this cherry pick:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
Author
|
@wlwilliamx This PR has conflicts, I have hold it. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
size/XXL
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Prometheus metric and Grafana dashboard panel to display detailed changefeed error information, including error codes, messages, and occurrence times. However, the submission is currently broken as it contains Git merge conflict markers across multiple files, including Go source and JSON dashboard definitions. Additionally, the constant changefeedErrorMetricMsgLimit is referenced in coordinator/helper.go but is not defined, which will prevent the code from compiling.
Comment on lines
+17
to
+23
| <<<<<<< HEAD | ||
| ======= | ||
| "strings" | ||
| "time" | ||
|
|
||
| "github.com/pingcap/ticdc/pkg/config" | ||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The code contains Git merge conflict markers. These must be resolved before the code can be compiled.
"strings"
"time"
"github.com/pingcap/ticdc/pkg/config"
Comment on lines
+49
to
+126
| <<<<<<< HEAD | ||
| ======= | ||
|
|
||
| type changefeedErrorMetricLabels struct { | ||
| keyspace string | ||
| changefeed string | ||
| state string | ||
| errorTime string | ||
| code string | ||
| message string | ||
| } | ||
|
|
||
| func isUnchangedRuntimeState(info *config.ChangeFeedInfo, state config.FeedState, err *config.RunningError) bool { | ||
| if info == nil { | ||
| return true | ||
| } | ||
| if info.State != state { | ||
| return false | ||
| } | ||
| return sameRunningErrorSignature(info.Error, err) | ||
| } | ||
|
|
||
| func sameRunningErrorSignature(lhs *config.RunningError, rhs *config.RunningError) bool { | ||
| if lhs == nil || rhs == nil { | ||
| return lhs == rhs | ||
| } | ||
| return lhs.Addr == rhs.Addr && | ||
| lhs.Code == rhs.Code && | ||
| lhs.Message == rhs.Message | ||
| } | ||
|
|
||
| func (l changefeedErrorMetricLabels) labelValues() []string { | ||
| return []string{l.keyspace, l.changefeed, l.state, l.errorTime, l.code, l.message} | ||
| } | ||
|
|
||
| func normalizeChangefeedErrorMetricMessage(message string) string { | ||
| message = strings.Join(strings.Fields(message), " ") | ||
| if len(message) <= changefeedErrorMetricMsgLimit { | ||
| return message | ||
| } | ||
| return message[:changefeedErrorMetricMsgLimit-3] + "..." | ||
| } | ||
|
|
||
| func normalizeChangefeedErrorMetricTime(errorTime time.Time) string { | ||
| // Keep the label stable across nodes with different local time zones while remaining | ||
| // directly readable in Grafana's table view. | ||
| if errorTime.IsZero() { | ||
| return "" | ||
| } | ||
| return errorTime.UTC().Format(time.RFC3339) | ||
| } | ||
|
|
||
| func getChangefeedErrorMetricLabels(info *config.ChangeFeedInfo) (changefeedErrorMetricLabels, bool) { | ||
| if info == nil { | ||
| return changefeedErrorMetricLabels{}, false | ||
| } | ||
| if info.State != config.StateFailed && info.State != config.StateWarning { | ||
| return changefeedErrorMetricLabels{}, false | ||
| } | ||
|
|
||
| runningErr := info.Error | ||
| if runningErr == nil { | ||
| runningErr = info.Warning | ||
| } | ||
| if runningErr == nil { | ||
| return changefeedErrorMetricLabels{}, false | ||
| } | ||
|
|
||
| return changefeedErrorMetricLabels{ | ||
| keyspace: info.ChangefeedID.Keyspace(), | ||
| changefeed: info.ChangefeedID.Name(), | ||
| state: string(info.State), | ||
| errorTime: normalizeChangefeedErrorMetricTime(runningErr.Time), | ||
| code: runningErr.Code, | ||
| message: normalizeChangefeedErrorMetricMessage(runningErr.Message), | ||
| }, true | ||
| } | ||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The code contains Git merge conflict markers. Additionally, the constant changefeedErrorMetricMsgLimit is used in normalizeChangefeedErrorMetricMessage but is not defined in this file. Based on the dashboard description, it should be set to 256.
const changefeedErrorMetricMsgLimit = 256
type changefeedErrorMetricLabels struct {
keyspace string
changefeed string
state string
errorTime string
code string
message string
}
func isUnchangedRuntimeState(info *config.ChangeFeedInfo, state config.FeedState, err *config.RunningError) bool {
if info == nil {
return true
}
if info.State != state {
return false
}
return sameRunningErrorSignature(info.Error, err)
}
func sameRunningErrorSignature(lhs *config.RunningError, rhs *config.RunningError) bool {
if lhs == nil || rhs == nil {
return lhs == rhs
}
return lhs.Addr == rhs.Addr &&
lhs.Code == rhs.Code &&
lhs.Message == rhs.Message
}
func (l changefeedErrorMetricLabels) labelValues() []string {
return []string{l.keyspace, l.changefeed, l.state, l.errorTime, l.code, l.message}
}
func normalizeChangefeedErrorMetricMessage(message string) string {
message = strings.Join(strings.Fields(message), " ")
if len(message) <= changefeedErrorMetricMsgLimit {
return message
}
return message[:changefeedErrorMetricMsgLimit-3] + "..."
}
func normalizeChangefeedErrorMetricTime(errorTime time.Time) string {
// Keep the label stable across nodes with different local time zones while remaining
// directly readable in Grafana's table view.
if errorTime.IsZero() {
return ""
}
return errorTime.UTC().Format(time.RFC3339)
}
func getChangefeedErrorMetricLabels(info *config.ChangeFeedInfo) (changefeedErrorMetricLabels, bool) {
if info == nil {
return changefeedErrorMetricLabels{}, false
}
if info.State != config.StateFailed && info.State != config.StateWarning {
return changefeedErrorMetricLabels{}, false
}
runningErr := info.Error
if runningErr == nil {
runningErr = info.Warning
}
if runningErr == nil {
return changefeedErrorMetricLabels{}, false
}
return changefeedErrorMetricLabels{
keyspace: info.ChangefeedID.Keyspace(),
changefeed: info.ChangefeedID.Name(),
state: string(info.State),
errorTime: normalizeChangefeedErrorMetricTime(runningErr.Time),
code: runningErr.Code,
message: normalizeChangefeedErrorMetricMessage(runningErr.Message),
}, true
}
Comment on lines
+76
to
+88
| <<<<<<< HEAD | ||
| ======= | ||
| // ChangefeedErrorInfoGauge records the current warning or failed reason and its occurrence time | ||
| // for each changefeed. | ||
| ChangefeedErrorInfoGauge = prometheus.NewGaugeVec( | ||
| prometheus.GaugeOpts{ | ||
| Namespace: "ticdc", | ||
| Subsystem: "owner", | ||
| Name: "changefeed_error_info", | ||
| Help: "The current warning or failed reason and occurrence time of changefeeds", | ||
| }, []string{getKeyspaceLabel(), "changefeed", "state", "error_time", "code", "message"}) | ||
|
|
||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The code contains Git merge conflict markers.
// ChangefeedErrorInfoGauge records the current warning or failed reason and its occurrence time
// for each changefeed.
ChangefeedErrorInfoGauge = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: "ticdc",
Subsystem: "owner",
Name: "changefeed_error_info",
Help: "The current warning or failed reason and occurrence time of changefeeds",
}, []string{getKeyspaceLabel(), "changefeed", "state", "error_time", "code", "message"})
Comment on lines
+4594
to
+4733
| <<<<<<< HEAD | ||
| ======= | ||
| }, | ||
| { | ||
| "datasource": "${DS_TEST-CLUSTER}", | ||
| "description": "Current warning or failed reason of each changefeed. The metric message is normalized to a single line and truncated to 256 characters.", | ||
| "fieldConfig": { | ||
| "defaults": { | ||
| "custom": { | ||
| "align": null, | ||
| "filterable": false | ||
| }, | ||
| "links": [], | ||
| "mappings": [], | ||
| "thresholds": { | ||
| "mode": "absolute", | ||
| "steps": [ | ||
| { | ||
| "color": "green", | ||
| "value": null | ||
| }, | ||
| { | ||
| "color": "red", | ||
| "value": 80 | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "overrides": [ | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "namespace" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 120 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "changefeed" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "state" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 100 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "code" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "error_time" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 24, | ||
| "x": 0, | ||
| "y": 26 | ||
| }, | ||
| "id": 62010, | ||
| "options": { | ||
| "showHeader": true, | ||
| "sortBy": [] | ||
| }, | ||
| "pluginVersion": "7.5.17", | ||
| "targets": [ | ||
| { | ||
| "expr": "max by (namespace, changefeed, state, code, error_time, message) (ticdc_owner_changefeed_error_info{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", namespace=~\"$namespace\", changefeed=~\"$changefeed\"})", | ||
| "format": "time_series", | ||
| "instant": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Changefeed Error Details", | ||
| "transformations": [ | ||
| { | ||
| "id": "labelsToFields", | ||
| "options": {} | ||
| }, | ||
| { | ||
| "id": "organize", | ||
| "options": { | ||
| "excludeByName": { | ||
| "Metric": true, | ||
| "Time": true, | ||
| "Value": true, | ||
| "__name__": true | ||
| }, | ||
| "indexByName": { | ||
| "namespace": 0, | ||
| "changefeed": 1, | ||
| "state": 2, | ||
| "error_time": 3, | ||
| "code": 4, | ||
| "message": 5 | ||
| }, | ||
| "renameByName": {} | ||
| } | ||
| } | ||
| ], | ||
| "type": "table" | ||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The JSON file contains Git merge conflict markers, which makes it invalid.
Comment on lines
+4594
to
+4733
| <<<<<<< HEAD | ||
| ======= | ||
| }, | ||
| { | ||
| "datasource": "${DS_TEST-CLUSTER}", | ||
| "description": "Current warning or failed reason of each changefeed. The metric message is normalized to a single line and truncated to 256 characters.", | ||
| "fieldConfig": { | ||
| "defaults": { | ||
| "custom": { | ||
| "align": null, | ||
| "filterable": false | ||
| }, | ||
| "links": [], | ||
| "mappings": [], | ||
| "thresholds": { | ||
| "mode": "absolute", | ||
| "steps": [ | ||
| { | ||
| "color": "green", | ||
| "value": null | ||
| }, | ||
| { | ||
| "color": "red", | ||
| "value": 80 | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "overrides": [ | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "keyspace_name" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 120 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "changefeed" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "state" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 100 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "code" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "error_time" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 24, | ||
| "x": 0, | ||
| "y": 26 | ||
| }, | ||
| "id": 62010, | ||
| "options": { | ||
| "showHeader": true, | ||
| "sortBy": [] | ||
| }, | ||
| "pluginVersion": "7.5.17", | ||
| "targets": [ | ||
| { | ||
| "expr": "max by (keyspace_name, changefeed, state, code, error_time, message) (ticdc_owner_changefeed_error_info{k8s_cluster=\"$k8s_cluster\", sharedpool_id=\"$tidb_cluster\", keyspace_name=~\"$keyspace_name\", changefeed=~\"$changefeed\"})", | ||
| "format": "time_series", | ||
| "instant": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Changefeed Error Details", | ||
| "transformations": [ | ||
| { | ||
| "id": "labelsToFields", | ||
| "options": {} | ||
| }, | ||
| { | ||
| "id": "organize", | ||
| "options": { | ||
| "excludeByName": { | ||
| "Metric": true, | ||
| "Time": true, | ||
| "Value": true, | ||
| "__name__": true | ||
| }, | ||
| "indexByName": { | ||
| "keyspace_name": 0, | ||
| "changefeed": 1, | ||
| "state": 2, | ||
| "error_time": 3, | ||
| "code": 4, | ||
| "message": 5 | ||
| }, | ||
| "renameByName": {} | ||
| } | ||
| } | ||
| ], | ||
| "type": "table" | ||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The JSON file contains Git merge conflict markers, which makes it invalid.
Comment on lines
+2337
to
+2476
| <<<<<<< HEAD | ||
| ======= | ||
| }, | ||
| { | ||
| "datasource": "${DS_TEST-CLUSTER}", | ||
| "description": "Current warning or failed reason of each changefeed. The metric message is normalized to a single line and truncated to 256 characters.", | ||
| "fieldConfig": { | ||
| "defaults": { | ||
| "custom": { | ||
| "align": null, | ||
| "filterable": false | ||
| }, | ||
| "links": [], | ||
| "mappings": [], | ||
| "thresholds": { | ||
| "mode": "absolute", | ||
| "steps": [ | ||
| { | ||
| "color": "green", | ||
| "value": null | ||
| }, | ||
| { | ||
| "color": "red", | ||
| "value": 80 | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| "overrides": [ | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "keyspace_name" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 120 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "changefeed" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "state" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 100 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "code" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "matcher": { | ||
| "id": "byName", | ||
| "options": "error_time" | ||
| }, | ||
| "properties": [ | ||
| { | ||
| "id": "custom.width", | ||
| "value": 180 | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| }, | ||
| "gridPos": { | ||
| "h": 8, | ||
| "w": 24, | ||
| "x": 0, | ||
| "y": 26 | ||
| }, | ||
| "id": 62010, | ||
| "options": { | ||
| "showHeader": true, | ||
| "sortBy": [] | ||
| }, | ||
| "pluginVersion": "7.5.17", | ||
| "targets": [ | ||
| { | ||
| "expr": "max by (keyspace_name, changefeed, state, code, error_time, message) (ticdc_owner_changefeed_error_info{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", keyspace_name=~\"$keyspace_name\", changefeed=~\"$changefeed\"})", | ||
| "format": "time_series", | ||
| "instant": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Changefeed Error Details", | ||
| "transformations": [ | ||
| { | ||
| "id": "labelsToFields", | ||
| "options": {} | ||
| }, | ||
| { | ||
| "id": "organize", | ||
| "options": { | ||
| "excludeByName": { | ||
| "Metric": true, | ||
| "Time": true, | ||
| "Value": true, | ||
| "__name__": true | ||
| }, | ||
| "indexByName": { | ||
| "keyspace_name": 0, | ||
| "changefeed": 1, | ||
| "state": 2, | ||
| "error_time": 3, | ||
| "code": 4, | ||
| "message": 5 | ||
| }, | ||
| "renameByName": {} | ||
| } | ||
| } | ||
| ], | ||
| "type": "table" | ||
| >>>>>>> 7b68b7051 (metrics: show error time in changefeed error details panel (#5086)) |
There was a problem hiding this comment.
The JSON file contains Git merge conflict markers, which makes it invalid.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated cherry-pick of #5086
What problem does this PR solve?
Issue Number: close #5085
What is changed and how it works?
ticdc_owner_changefeed_error_infometric as anerror_timelabel.error_timein theChangefeed Error DetailsGrafana panel for both standard and next-generation dashboards.Check List
Tests
Questions
Will it cause performance regression or break compatibility?
No expected performance regression. This extends the current error-info metric with one stable label for the active error occurrence time. Existing selectors remain valid, while consumers that depend on the exact label set should account for the added
error_timelabel.Do you need to update user documentation, design documentation or monitoring documentation?
No separate documentation update is needed. The affected Grafana dashboard definitions are updated in this PR.
Release note
Summary by CodeRabbit
Release Notes
New Features
Tests