Schemas get deleted after compaction

[yogeshbelur]

Hi

I have a scenario where system is running for long time, but after somedays i run console consumer on _schemas and i see 0 msgs. But when i query subjects from SR I get all the registered schema. I am afraid now that when i restart SR all schemas will vanish as currenlty SR has schemas in cache.

What is the reason schemas got deleted from topic.

Below are the topics settings

Topic:_schemas PartitionCount:1 ReplicationFactor:3 Configs:cleanup.policy=compact Topic: _schemas Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1001,1003,1002

[OneCricketeer]

Are there still log files on the Kafka brokers for that topic?

[jung-hyunchul]

I have same issue.
After restarting schema registry, I can't see any schema using schema-registry ui. so I consumed _schemas topic, but there were no schema at all.
I have 3 zookeeper ( using docker ), 5 brokers ( using docker ), 1 schema registry ( using docker )
Kafka and schema registry are all same version 5.0.0

[yogeshbelur]

Hi sorry i had lost the logs. But i am trying to reproduce. I see _schemas logs and index delete message. Is it anything to do with log compact? Will not see messages on kafka topic when compact happened?

[chaicesan]

Any updates on this. We have experienced the same issue.

[keithhawkridge]

Any updates on this. We have experienced the same issue.

It seems that our schemas are deleted from the _schemas topic after a period of time

[ArturKT]

Any update? I have experienced this issue too.

[ethanttbui]

It baffles me that the is no response to such a serious issue..

[renatocron]

Any updates on this? I think I experienced a similar issue while running version confluentinc/cp-schema-registry:7.0.1.

I upgraded my cluster to larger machines, and everything was running fine for about 200 days, even after the upgrade. However, when I shut off the smaller pods, the schema registry pods relocated to the newer nodes. Somehow, the _schemas topic was the only one that appeared 'clean' after this.

This caused my producer to start creating new messages with new IDs, leading to a very cumbersome process to restore the messages back to their original IDs. I had to publish directly on _schemas while all producers, consumers, and the schema registry were scaled to 0, insert the duplicate magic numbers, restart the cluster, and then process the messages. I was fortunate to encounter only one collision where a previous magic ID matched a newer one, and this happened during a maintenance window.

I'm running 2 instances of the SR, without anything set besides SCHEMA_REGISTRY_HOST_NAME/SCHEMA_REGISTRY_LISTENERS/SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS so maybe some kind of race condition? maybe related to #3127 but I think that running in the same gruop is the default behavior, as I just one have 2 SR for HA