feat: add tabletopic e2e case

[Gezi-lzq]

Background

• Goal: Validate the full closed loop from Table Topic production (Avro) → Kafka write → Iceberg commit and query, ensuring record count consistency and providing diagnostic information on failure.
• Scenario: Use REST Iceberg Catalog and Schema Registry, start dependencies in a production-like deployment manner for end-to-end verification.

Components

• Kafka Cluster: KafkaService (KRaft mode, 3 nodes).
• External Dependencies: DockerComposeService to start Schema Registry (10.5.1.3:8081) and Iceberg Catalog (10.5.1.4:8181).
• Load Test Producer: AutoMQPerformanceService wraps bin/automq-perf-test.sh (org.apache.kafka.tools.automq.PerfCommand).

Core Flow

1. Startup Order: Iceberg Catalog → Kafka → Schema Registry (injecting KAFKA_BOOTSTRAP_SERVERS).
2. Health Checks: _wait_for_service polls Catalog /v1/config and Registry root path; proceeds after READY status.
3. Production: Use AutoMQPerformanceService to send Avro data; configure topic/producer parameters required by TableTopic.
4. Produced Records Counting: Use Kafka side "end offsets" (_sum_kafka_end_offsets_stable) as produced count; produceCountTotal from perf is only for log comparison.
5. Iceberg Verification: Poll REST metadata, parse total-records, compare with expected record count; on failure, print detailed context info.

New test cases added (asserting consistency: Kafka offsets == Iceberg total-records)

• Minimal test: automq_tabletopic_e2e_test.py::test_tabletopic_avro_e2e_flow
• Parameter matrix: commit.interval.ms ∈ {500, 2000, 10000}; topics ∈ {1, 10, 100}
• Periodic restart: automq_tabletopic_broker_restart_test.py::test_broker_periodic_restart_10m
• Schema evolution: automq_tabletopic_schema_evolution_test.py::test_schema_evolution_4_phases
• Partition migration: automq_tabletopic_partition_reassignment_test.py::test_partition_reassignment_during_produce

[Copilot]

Pull Request Overview

This PR adds comprehensive end-to-end testing capability for the TableTopic feature in AutoMQ. It introduces random Avro value generation, establishes a base test framework with external service management, and implements multiple test scenarios to validate TableTopic functionality under various conditions.

• Adds random Avro value generation to PerfCommand for schema-based testing without requiring predefined value files
• Creates a comprehensive TableTopic test framework with base services and helper methods
• Implements multiple test scenarios including schema evolution, partition reassignment, broker restarts, and matrix testing

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/src/main/java/org/apache/kafka/tools/automq/PerfCommand.java	Adds random Avro value generation functionality with comprehensive schema type support
tools/src/main/java/org/apache/kafka/tools/automq/perf/PerfConfig.java	Removes --values-random option, defaulting to random generation when no values file is provided
tests/kafkatest/tests/core/automq_tabletopic_base.py	Base test framework providing service lifecycle management and common test utilities
tests/kafkatest/tests/core/automq_tabletopic_e2e_test.py	Basic end-to-end test validating core TableTopic functionality
tests/kafkatest/tests/core/automq_tabletopic_schema_evolution_test.py	Tests schema evolution across multiple compatible Avro schema versions
tests/kafkatest/tests/core/automq_tabletopic_partition_reassignment_test.py	Tests partition reassignment during active data production
tests/kafkatest/tests/core/automq_tabletopic_broker_restart_test.py	Tests broker restart scenarios with continuous data flow
tests/kafkatest/tests/core/automq_tabletopic_matrix_test.py	Matrix testing for different configurations and topic counts
tests/kafkatest/services/performance/automq_performance.py	Performance service wrapper for AutoMQ PerfCommand with Avro support
tests/kafkatest/services/external_services.py	Docker Compose service management for external dependencies
tests/docker/	Docker configuration files for Schema Registry and Iceberg Catalog services
tests/docker/ducker-ak	Modified to enable Docker-in-Docker for ducker01 node
tests/docker/Dockerfile	Adds Docker installation to test container

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Using shell=True with user-controlled input in subprocess.check_call is a security risk. Consider using subprocess.run with a list of arguments instead of shell=True.