roachtest: cdc/sink-chaos failed

[cockroach-teamcity]

roachtest.cdc/sink-chaos failed with artifacts on release-24.1 @ 7185e6c311160061e0753e9ef1d66e8bb828fc98:

(cdc.go:560).newChangefeed: failed to create changefeed: pq: kafka server: Replication-factor is invalid
(cluster.go:2348).Run: context canceled
(cdc.go:385).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/sink-chaos/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-38021

[wenyihu6]

Not a release blocker - this looks like the same infra flake as #121346. We might want to deflake it by waiting for the new brokers to be properly started before creating topics.

[2024-04-19 11:00:15,235] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2024-04-19 11:00:15,256] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='item', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,257] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='district', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='order_line', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='history', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='warehouse', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='stock', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='new_order', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,258] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='order', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,259] INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='customer', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
[2024-04-19 11:00:15,299] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use node teamcity-14898299-1713505946-26-n4cpu16-0004.c.cockroach-ephemeral.internal:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2024-04-19 11:00:15,333] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use node teamcity-14898299-1713505946-26-n4cpu16-0004.c.cockroach-ephemeral.internal:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)