Kafka auth test fails with dropped instead of stalled

[def-]

What version of Materialize are you using?

0716439

What is the issue?

Seen in https://buildkite.com/materialize/tests/builds/69712#018c11e1-c004-4451-bbaf-dc16d421a8be

> ALTER CONNECTION kafka_backup SET (BROKER 'kafka:9093') WITH (VALIDATE = false);
rows match; continuing at ts 1701108752.3339665
Ingesting data into Kafka topic testdrive-avro-data-3510298630 with start_iteration = 0, repeat = 1
> SELECT * FROM avro_data
rows didn't match; sleeping to see if dataflow catches up 50ms 75ms 113ms 169ms 253ms 380ms
rows match; continuing at ts 1701108753.6897357
> SELECT status FROM mz_internal.mz_sink_statuses WHERE name = 'snk_backup';
rows didn't match; sleeping to see if dataflow catches up 50ms 75ms 113ms 169ms 253ms 380ms 570ms 854ms 1s 2s 3s 4s 6s 10s 15s 22s 33s 22s
^^^ +++
test-schema-registry-ssl-basic.td:129:1: error: non-matching rows: expected:
[["stalled"]]
got:
[["dropped"]]
Poor diff:
+ dropped
- stalled

@sploiselle Could this be related to your alter connection changes?

[sploiselle]

I suspect this is more likely related to #23222, cc'ing @guswynn but I'll take a look

[guswynn]

@sploiselle this was found by @def- before that pr merged :(

even so, i dont see the codepath that would call healthcheck::drop_sinks when altering a sink connection...

[sploiselle]

@guswynn PTAL at my understanding:

• @def- identified this bug in 0716439
• The parent of that commit is 59f74a7, which looks a lot like 77aaed0, which is in storage: write status updates in the controller, and add the "paused" status #23222

[def-]

Sorry for the incomplete information when opening this issue. I saw this previously on main before @guswynn 's change: https://buildkite.com/materialize/tests/builds/69697#018c11a7-fd87-4ad8-972e-6ca1c2134d20

[sploiselle]

I just ran test-schema-registry-basic.td test for 6 hours in a loop without hitting this issue. Seems like it might need a broader set of tests to run to trigger.

[def-]

I reproduced it locally in a few minutes with

while true; do bin/mzcompose --find kafka-auth down && bin/mzcompose --find kafka-auth run default || break; done

[sploiselle]

This might also be #23450

[sploiselle]

This is probably #23507

[sploiselle]

This should be fixed as of #23528