Confluent schema-registry crashes on relaunch

[hagmonk]

Confluent's schema-registry creates a compacted topic called _schemas. On initial launch this works fine, but if the schema-registry is restarted, we get the following crash:

java.lang.NullPointerException
     at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:263)
     at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:171)
     at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:118)
     at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:295)
     at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:73)
     at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:88)
     at io.confluent.rest.Application.configureHandler(Application.java:255)
     at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:227)
     at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
     at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

Looking at KafkaStore.java:263, this occurs in verifySchemaTopic which is only called if the topic doesn't exist (see createOrVerifySchemaTopic)

This problem persists even if the topic is manually created with --compact:

rpk topic create _schemas --replicas 3 --compact

Or modified with set-config:

rpk topic set-config _schemas cleanup.policy compact

rpk debug info

Error initializing: The cloud vendor couldn't be detected
  Version                                            v21.2.1 (rev c74b7048)                                      
  CPU Usage %                                        0.500                                                       
  Free Memory (MB)                                   219245.828                                                  
  Free Space  (MB)                                   16206.422                                                   
  OS                                                 x86_64����������������������������������������������������������� 5.4.77-7.el7pie�������������������������������������������������� #1 SMP Sat Nov 21 01:16:27 UTC 2020������������������������������  
  CPU Model                                          Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz                   
  config_file                                        /etc/redpanda/redpanda.yaml                                 
  license_key                                                                                                    
  organization                                                                                                   
  redpanda.admin                                     0.0.0.0:9644                                                
  redpanda.data_directory                            /var/lib/redpanda/data                                      
  redpanda.developer_mode                            false                                                       
  redpanda.kafka_api                                 0.0.0.0:9092                                                
  redpanda.node_id                                   0                                                           
  redpanda.rpc_server                                0.0.0.0:33145                                               
  rpk.coredump_dir                                   /var/lib/redpanda/coredump                                  
  rpk.enable_memory_locking                          false                                                       
  rpk.enable_usage_stats                             false                                                       
  rpk.overprovisioned                                true                                                        
  rpk.tune_aio_events                                false                                                       
  rpk.tune_clocksource                               false                                                       
  rpk.tune_coredump                                  false                                                       
  rpk.tune_cpu                                       false                                                       
  rpk.tune_disk_irq                                  false                                                       
  rpk.tune_disk_nomerges                             false                                                       
  rpk.tune_disk_scheduler                            false                                                       
  rpk.tune_disk_write_cache                          false                                                       
  rpk.tune_fstrim                                    false                                                       
  rpk.tune_network                                   false                                                       
  rpk.tune_swappiness                                false                                                       
  rpk.tune_transparent_hugepages                     false                                                       
  rpk.well_known_io                                                                                              
                                                                                                                 
  Redpanda Cluster Status                                                                                        
  Node ID (IP)                                       Partitions                                                  
  0 (redpanda-0.redpanda.schmoo-cabal-dev.svc:9092)  Leader: _schemas: [0]                                       
                                                                                                                 
                                                     Replica:                                                    
                                                                                                                 
  1 (redpanda-1.redpanda.schmoo-cabal-dev.svc:9092)  Leader:                                                     
                                                                                                                 
                                                     Replica: _schemas: [0]                                      
                                                                                                                 
  2 (redpanda-2.redpanda.schmoo-cabal-dev.svc:9092)  Leader:                                                     
                                                                                                                 
                                                     Replica: _schemas: [0]                                      
                                                                                                                 

[emaxerrno]

@mmaslankaprv i think this has to do with alter configs ?

[dotnwat]

ah this may be redpanda not reporting all topic configuration properties

[dotnwat]

I was able to reproduce this. The issue is that redpanda is not reporting the clean-up policy in the describe configs api. Should be an easy fix. @hagmonk i'll update this ticket when a fix drops.

[dotnwat]

hey @hagmonk a fix for this has merged. can you recommend any sort of workload / tests to run against the schema registry? given the feedback loop to get a new release cut with this change (@senior7515 ?) it could be useful for us to do a little testing locally in case there are other incompatibilities lurking in there.