[Bug]: Kafka image 3.6.1 is failing with cat: /opt/kafka/custom-config/metadata.state: No such file or directory

[infa-vdas]

Bug Description

We are using strimzi operator version 0.39.0 and upgraded strimzi Kafka image from 3.5.2 to 3.6.1
Post deploying the kafka image 3.6.1, zookeeper pods came up successfully but broker pods are failing with following exception

[centos@mgmt-node-abcd ~]$ kubectl logs abcd-kafka-kafka-0
removed directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/custom-external-9094.keystore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=0
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing store for custom-external-9094 listener
Creating keystore /tmp/kafka/custom-external-9094.keystore.p12
Preparing store for custom external listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################

This file is automatically generated by the Strimzi Cluster Operator

Any changes to this file will be ignored and overwritten!

##############################
##############################

##########

Node / Broker ID

##########
broker.id=0
node.id=0

##########

Kafka message logs configuration

##########
log.dirs=/var/lib/kafka/data/kafka-log0

##########

Control Plane listener

##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########

Replication listener

##########
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.replication-9091.ssl.keystore.password=[hidden]
listener.name.replication-9091.ssl.keystore.type=PKCS12
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.replication-9091.ssl.truststore.password=[hidden]
listener.name.replication-9091.ssl.truststore.type=PKCS12
listener.name.replication-9091.ssl.client.auth=required

##########

Listener configuration: INTERNAL-9093

##########
listener.name.internal-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.internal-9093.ssl.keystore.password=[hidden]
listener.name.internal-9093.ssl.keystore.type=PKCS12

##########

Listener configuration: EXTERNAL-9094

##########
listener.name.external-9094.ssl.keystore.location=/tmp/kafka/custom-external-9094.keystore.p12
listener.name.external-9094.ssl.keystore.password=[hidden]
listener.name.external-9094.ssl.keystore.type=PKCS12

##########

Common listener configuration

##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,INTERNAL-9093:SSL,EXTERNAL-9094:SSL
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,INTERNAL-9093://0.0.0.0:9093,EXTERNAL-9094://0.0.0.0:9094
advertised.listeners=CONTROLPLANE-9090://abcd-kafka-kafka-0.abcd-kafka-kafka-brokers.abcd-qa-kafka.svc:9090,REPLICATION-9091://abcd-kafka-kafka-0.abcd-kafka-kafka-brokers.abcd-qa-kafka.svc:9091,INTERNAL-9093://abcd-kafka-kafka-0.abcd-kafka-kafka-brokers.abcd-qa-kafka.svc:9093,EXTERNAL-9094://b0-abcd-qa-mrel.kafka.infacloudops.net:9094
inter.broker.listener.name=REPLICATION-9091
control.plane.listener.name=CONTROLPLANE-9090
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########

User provided configuration

##########
inter.broker.protocol.version=3.6
log.message.format.version=3.6
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3

##########

Zookeeper

##########
zookeeper.connect=abcd-kafka-zookeeper-client:2181
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
zookeeper.ssl.keystore.password=[hidden]
zookeeper.ssl.keystore.type=PKCS12
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
zookeeper.ssl.truststore.password=[hidden]
zookeeper.ssl.truststore.type=PKCS12
Configuring Java heap: -Xms5368709120 -Xmx5368709120
cat: /opt/kafka/custom-config/metadata.state: No such file or directory
[centos@mgmt-node-abcd ~]$

On downgrading the kafka image to 3.5.2, broker pods are coming up successfully. Observing the issue when upgrading kafka image to 3.6.1.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

Strimzi Kafka image 3.6.1

Kubernetes version

Kubernetes v1.27

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response