Skip to content

kafka: improve topic validation, timeout checks, and logging #5819

Description

@3AceShowHand

Problem

Kafka sink has two validation gaps and several low-quality operational logs:

  1. After CreateTopic succeeds, waitUntilTopicVisible treats an empty metadata map with no error as proof that the topic is visible. The sink can proceed before brokers expose the requested topic.
  2. Kafka sink accepts explicitly configured non-positive dial-timeout, read-timeout, and write-timeout values.
  3. Kafka logging contains noisy normal-path messages, duplicated error logs, inconsistent messages and fields, and per-message payload diagnostics. Some integration tests depend on exact log text instead of validating behavior.

Expected behavior

  • Topic creation retries until metadata contains the requested topic, and success is logged only after visibility is confirmed.
  • Explicit Kafka timeout values must be greater than zero and invalid values return CDC:ErrKafkaInvalidConfig. Omitted values keep the existing 10-second defaults.
  • Kafka sink logs focus on lifecycle transitions, final configuration, and actionable failures:
    • normal control-flow and duplicated errors remain quiet;
    • producer failures preserve the original Kafka error and event context;
    • version detection emits one fallback warning without changing version selection;
    • messages, fields, and durations use consistent names and types;
    • DDL logs describe the specific child event that was skipped or sent.
  • Integration tests validate end-to-end behavior rather than grep implementation-specific Kafka log messages.

Related PR: #5818

Metadata

Metadata

Assignees

Labels

affects-8.5This bug affects the 8.5.x(LTS) versions.type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions