Tracking issue for Kafka handling large message without halting changefeeds

[3AceShowHand]

Background

Changefeed send messages to the Kafka server, if the message size is larger than the Kafka's configuration max.message.bytes, an error will be reported to the changefeed and it get stuck.

It's unexpected that an occasional large message cause the whole changefeed cannot make progress. So we propose that, if the message size is large than the message limit, only encode the primary key or the not null unique key of the event, and add a new flag to indicate this. The Kafka consumer once receive such an message, they should use the PK / UK to query the upstream TiDB to get the corresponding data.

Development

• add changefeed level configuration to support only encode handle key columns #9165
• open-protocol encoder support only encode handle key columns #9188
• decoder support decode message only have handle key columns #9189
• kafka consumer support query upstream tidb to get the whole row data #9190
• add large message handle option config #9346
• canal-json encoder support only encode handle key columns #9382
• avro encoder support only encode handle key columns #9403
• canal-json encoder support claim-check interface #9383
• open-protocol decoder support decode message only have handle key columns #9443
• canal-json decoder support decode message only have handle key columns #9442
• avro decoder support decode message only have handle key columns #9404
• kafka support claim-check functionality by send message to the external storage #9375
• add claim check related monitoring #9399
• avro encoder support encode claim check message #9405
• force-replicate is allowed when encode event only have handle key columns #9478
• canal-json consume handle key only message example #9490
• canal-json add unit test to cover claim-check functionality #9528
• open-protocol add unit test to cover claim-check functionality #9529
• cherry-pick large message handle key columns only feature to release-6.5 #9574
• kafka sink support compress the message value #9596
• large message handle functionality add more tests to cover it #9632
• cherry-pick large message handle key columns only feature to release-7.1 #9680
• embed the claim check into encoder to make send message to the external storage asynchronously #9693
• changefeed stucks for open protocol changefeed with claim-check on #9775

Docs

• update the document to describe the open-protocol with large message only handle columns enabled