codec(ticdc): improve error logging for Debezium encoding failures#3963
Conversation
ti-chi-bot
Bot
added
do-not-merge/needs-linked-issue
do-not-merge/release-note-label-needed
|
Hi @takaidohigasi. Thanks for your PR. I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Welcome @takaidohigasi! |
|
|
Summary of ChangesHello @takaidohigasi, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the debugging experience for Debezium encoding issues by enriching error logs with crucial contextual details. Previously, error messages were generic, but now they provide specific information about the database, table, column, and value that caused the failure, along with the commit timestamp, making it much easier to pinpoint and resolve problems. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
ti-chi-bot
Bot
added
size/S
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request significantly improves error logging for Debezium encoding failures by adding valuable context like schema, table, column, and the problematic value. This will make debugging much easier.
My review includes one main suggestion to make the new logging code more robust. It addresses a potential panic when converting data to a string and improves the readability of logged binary data. Overall, this is a great enhancement.
ti-chi-bot
Bot
added
do-not-merge/needs-linked-issue
and removed
do-not-merge/needs-triage-completed
labels
takaidohigasi
force-pushed
the
improve-debezium-error-logging
branch
from
5cf8307 to
260de85
Compare
|
Thanks for the review! I've addressed both concerns:
The updated code now safely handles both edge cases. See commit 260de85. |
ti-chi-bot
Bot
added
size/M
|
/ok-to-test |
ti-chi-bot
Bot
added
ok-to-test
| datum := row.GetDatum(i, ft) | ||
| valueStr := func() (s string) { | ||
| defer func() { | ||
| if r := recover(); r != nil { |
There was a problem hiding this comment.
I think the recover is not necessary
Add detailed error logging when Debezium field value encoding fails. The error log now includes: - Schema name - Table name - Column name - Value (truncated to 1024 chars if too large, hex-encoded for binary types) - Commit timestamp This helps diagnose encoding issues like enum columns with invalid string values (e.g., issue #12474 in tiflow). Additionally, added panic recovery and hex encoding for binary types to ensure safe and readable error logging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
takaidohigasi
force-pushed
the
improve-debezium-error-logging
branch
from
260de85 to
0aed4e6
Compare
📝 WalkthroughWalkthroughEnhanced error logging in Debezium codec by adding Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Important Action Needed: IP Allowlist UpdateIf your organization protects your Git platform with IP whitelisting, please add the new CodeRabbit IP address to your allowlist:
Failure to add the new IP will result in interrupted reviews. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@pkg/sink/codec/debezium/codec.go`:
- Around line 66-75: The switch handling column types in
pkg/sink/codec/debezium/codec.go is missing mysql.TypeVarchar, so VARBINARY
columns aren't being hex-encoded like other binary types; update the switch case
that currently lists mysql.TypeTinyBlob, mysql.TypeMediumBlob,
mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString to
also include mysql.TypeVarchar and keep the existing
mysql.HasBinaryFlag(ft.GetFlag()) branch that sets valueStr =
hex.EncodeToString(datum.GetBytes()) (using ft, datum, valueStr) so behavior
matches the encoding logic used later (lines ~523-531).
- Around line 61-86: The value extraction and conversion logic (using
colInfo.FieldType, row.GetDatum(i, ft), datum.String(), and hex.EncodeToString)
can panic on corrupted datum; wrap the entire datum access and string/byte
conversion in a small panic-recovery block (defer + recover) so any panic sets
valueStr to a safe placeholder like "<corrupt value - panic recovered>" (and
optionally capture the panic error/stack into a separate log field), then
continue to apply truncation and call log.Error as before; ensure the recovery
scope covers both datum.GetBytes()/datum.String() paths and does not swallow
unrelated panics.
| // Get value for logging with truncation | ||
| ft := &colInfo.FieldType | ||
| datum := row.GetDatum(i, ft) | ||
| var valueStr string | ||
| // Use hex encoding for binary types for better readability | ||
| switch ft.GetType() { | ||
| case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | ||
| if mysql.HasBinaryFlag(ft.GetFlag()) { | ||
| valueStr = hex.EncodeToString(datum.GetBytes()) | ||
| } else { | ||
| valueStr = datum.String() | ||
| } | ||
| default: | ||
| valueStr = datum.String() | ||
| } | ||
| const maxValueLen = 1024 | ||
| if len(valueStr) > maxValueLen { | ||
| valueStr = valueStr[:maxValueLen] + "...(truncated)" | ||
| } | ||
| log.Error("failed to write Debezium field value", | ||
| zap.String("schema", tableInfo.GetSchemaName()), | ||
| zap.String("table", tableInfo.GetTableName()), | ||
| zap.String("column", colInfo.Name.O), | ||
| zap.String("value", valueStr), | ||
| zap.Uint64("commitTs", commitTs), | ||
| zap.Error(err)) |
There was a problem hiding this comment.
Potential panic from datum.String() on corrupted data.
The PR discussion mentions adding defer/recover around datum.String() to handle potential panics from corrupted data, but I don't see that protection in this code. If datum.String() panics, it will crash the encoding process.
Consider wrapping the value extraction in a recovery mechanism:
🛡️ Proposed fix to add panic recovery
// Get value for logging with truncation
ft := &colInfo.FieldType
datum := row.GetDatum(i, ft)
- var valueStr string
- // Use hex encoding for binary types for better readability
- switch ft.GetType() {
- case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString:
- if mysql.HasBinaryFlag(ft.GetFlag()) {
- valueStr = hex.EncodeToString(datum.GetBytes())
- } else {
+ var valueStr string
+ // Use hex encoding for binary types for better readability
+ func() {
+ defer func() {
+ if r := recover(); r != nil {
+ valueStr = fmt.Sprintf("<panic: %v>", r)
+ }
+ }()
+ switch ft.GetType() {
+ case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString:
+ if mysql.HasBinaryFlag(ft.GetFlag()) {
+ valueStr = hex.EncodeToString(datum.GetBytes())
+ } else {
+ valueStr = datum.String()
+ }
+ default:
valueStr = datum.String()
}
- default:
- valueStr = datum.String()
- }
+ }()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Get value for logging with truncation | |
| ft := &colInfo.FieldType | |
| datum := row.GetDatum(i, ft) | |
| var valueStr string | |
| // Use hex encoding for binary types for better readability | |
| switch ft.GetType() { | |
| case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | |
| if mysql.HasBinaryFlag(ft.GetFlag()) { | |
| valueStr = hex.EncodeToString(datum.GetBytes()) | |
| } else { | |
| valueStr = datum.String() | |
| } | |
| default: | |
| valueStr = datum.String() | |
| } | |
| const maxValueLen = 1024 | |
| if len(valueStr) > maxValueLen { | |
| valueStr = valueStr[:maxValueLen] + "...(truncated)" | |
| } | |
| log.Error("failed to write Debezium field value", | |
| zap.String("schema", tableInfo.GetSchemaName()), | |
| zap.String("table", tableInfo.GetTableName()), | |
| zap.String("column", colInfo.Name.O), | |
| zap.String("value", valueStr), | |
| zap.Uint64("commitTs", commitTs), | |
| zap.Error(err)) | |
| // Get value for logging with truncation | |
| ft := &colInfo.FieldType | |
| datum := row.GetDatum(i, ft) | |
| var valueStr string | |
| // Use hex encoding for binary types for better readability | |
| func() { | |
| defer func() { | |
| if r := recover(); r != nil { | |
| valueStr = fmt.Sprintf("<panic: %v>", r) | |
| } | |
| }() | |
| switch ft.GetType() { | |
| case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | |
| if mysql.HasBinaryFlag(ft.GetFlag()) { | |
| valueStr = hex.EncodeToString(datum.GetBytes()) | |
| } else { | |
| valueStr = datum.String() | |
| } | |
| default: | |
| valueStr = datum.String() | |
| } | |
| }() | |
| const maxValueLen = 1024 | |
| if len(valueStr) > maxValueLen { | |
| valueStr = valueStr[:maxValueLen] + "...(truncated)" | |
| } | |
| log.Error("failed to write Debezium field value", | |
| zap.String("schema", tableInfo.GetSchemaName()), | |
| zap.String("table", tableInfo.GetTableName()), | |
| zap.String("column", colInfo.Name.O), | |
| zap.String("value", valueStr), | |
| zap.Uint64("commitTs", commitTs), | |
| zap.Error(err)) |
🤖 Prompt for AI Agents
In `@pkg/sink/codec/debezium/codec.go` around lines 61 - 86, The value extraction
and conversion logic (using colInfo.FieldType, row.GetDatum(i, ft),
datum.String(), and hex.EncodeToString) can panic on corrupted datum; wrap the
entire datum access and string/byte conversion in a small panic-recovery block
(defer + recover) so any panic sets valueStr to a safe placeholder like
"<corrupt value - panic recovered>" (and optionally capture the panic
error/stack into a separate log field), then continue to apply truncation and
call log.Error as before; ensure the recovery scope covers both
datum.GetBytes()/datum.String() paths and does not swallow unrelated panics.
| switch ft.GetType() { | ||
| case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | ||
| if mysql.HasBinaryFlag(ft.GetFlag()) { | ||
| valueStr = hex.EncodeToString(datum.GetBytes()) | ||
| } else { | ||
| valueStr = datum.String() | ||
| } | ||
| default: | ||
| valueStr = datum.String() | ||
| } |
There was a problem hiding this comment.
Missing mysql.TypeVarchar in binary type check.
The switch statement doesn't include mysql.TypeVarchar, but this type can also be binary (VARBINARY). The main encoding logic at lines 523-531 includes mysql.TypeVarchar for binary handling. For consistency, VARBINARY columns should also be hex-encoded in error logs.
🔧 Proposed fix to include TypeVarchar
switch ft.GetType() {
- case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString:
+ case mysql.TypeVarchar, mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString:
if mysql.HasBinaryFlag(ft.GetFlag()) {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| switch ft.GetType() { | |
| case mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | |
| if mysql.HasBinaryFlag(ft.GetFlag()) { | |
| valueStr = hex.EncodeToString(datum.GetBytes()) | |
| } else { | |
| valueStr = datum.String() | |
| } | |
| default: | |
| valueStr = datum.String() | |
| } | |
| switch ft.GetType() { | |
| case mysql.TypeVarchar, mysql.TypeTinyBlob, mysql.TypeMediumBlob, mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString: | |
| if mysql.HasBinaryFlag(ft.GetFlag()) { | |
| valueStr = hex.EncodeToString(datum.GetBytes()) | |
| } else { | |
| valueStr = datum.String() | |
| } | |
| default: | |
| valueStr = datum.String() | |
| } |
🤖 Prompt for AI Agents
In `@pkg/sink/codec/debezium/codec.go` around lines 66 - 75, The switch handling
column types in pkg/sink/codec/debezium/codec.go is missing mysql.TypeVarchar,
so VARBINARY columns aren't being hex-encoded like other binary types; update
the switch case that currently lists mysql.TypeTinyBlob, mysql.TypeMediumBlob,
mysql.TypeLongBlob, mysql.TypeBlob, mysql.TypeVarString, mysql.TypeString to
also include mysql.TypeVarchar and keep the existing
mysql.HasBinaryFlag(ft.GetFlag()) branch that sets valueStr =
hex.EncodeToString(datum.GetBytes()) (using ft, datum, valueStr) so behavior
matches the encoding logic used later (lines ~523-531).
|
/check-issue-triage-complete |
|
/test kafka |
ti-chi-bot
Bot
added
lgtm
and removed
needs-1-more-lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3AceShowHand, wk989898 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
3 participants
What problem does this PR solve?
Issue Number: close #3964
When Debezium encoding fails, the current error log lacks context information, making it difficult to debug which table/column caused the issue.
Related tiflow PR: pingcap/tiflow#12484
What is changed and how it works?
Improve error logging for Debezium encoding failures by adding detailed context information.
When Debezium encoding fails, the error log now includes:
schema: database/schema nametable: table namecolumn: column name that caused the errorvalue: the actual value (truncated to 1024 bytes if too large)commitTs: the commit timestamp of the eventThe
commitTsin the log can be used to skip problematic events using checkpoint override.Before:
After:
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
No. This only affects error logging path which is not performance critical.
Do you need to update user documentation, design documentation or monitoring documentation?
No.
Release note
Summary by CodeRabbit