After 1.1.0 Operator + Kafka 4.2.1 Update - MM2 fails to retrieve secret #13095
Replies: 1 comment · 4 replies
|
I think for a start you should share the complete custom resource and the complete logs from the Cluster Operator and Connect/MM2 Pods. |
All reactions
|
okay thank you for your help. Attached the logs etc. I use Strimzi 1.2.0 and Kafka 1.3..1 Operator Logs: 2026-08-28 07:24:50 INFO StrimziPodSetController:350 - Reconciliation #3549(watch) StrimziPodSet(kafka-archive/mm2-archive-mirrormaker2): StrimziPodSet will be reconciled
2026-08-28 07:24:50 INFO StrimziPodSetController:388 - Reconciliation #3549(watch) StrimziPodSet(kafka-archive/mm2-archive-mirrormaker2): reconciled
2026-08-28 07:24:50 INFO AbstractOperator:498 - Reconciliation #3550(watch) Kafka(env3/cluster): Kafka cluster in namespace env3 was MODIFIED
2026-08-28 07:24:50 INFO CrdOperator:147 - Reconciliation #3532(watch) Kafka(env3/cluster): Status of Kafka cluster in namespace env3 has been updated
2026-08-28 07:24:50 INFO AbstractOperator:521 - Reconciliation #3532(watch) Kafka(env3/cluster): reconciled
2026-08-28 07:24:50 INFO AbstractOperator:242 - Reconciliation #3540(watch) Kafka(env3/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:52 INFO CrdOperator:147 - Reconciliation #3533(watch) Kafka(env1/cluster): Status of Kafka cluster in namespace env1 has been updated
2026-08-28 07:24:52 INFO AbstractOperator:521 - Reconciliation #3533(watch) Kafka(env1/cluster): reconciled
2026-08-28 07:24:52 INFO AbstractOperator:498 - Reconciliation #3551(watch) Kafka(env1/cluster): Kafka cluster in namespace env1 was MODIFIED
2026-08-28 07:24:52 INFO AbstractOperator:242 - Reconciliation #3542(watch) Kafka(env1/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:53 INFO CrdOperator:147 - Reconciliation #3535(watch) Kafka(env2/cluster): Status of Kafka cluster in namespace env2 has been updated
2026-08-28 07:24:53 INFO AbstractOperator:521 - Reconciliation #3535(watch) Kafka(env2/cluster): reconciled
2026-08-28 07:24:53 INFO AbstractOperator:498 - Reconciliation #3552(watch) Kafka(env2/cluster): Kafka cluster in namespace env2 was MODIFIED
2026-08-28 07:24:53 INFO AbstractOperator:242 - Reconciliation #3543(watch) Kafka(env2/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:53 INFO CrdOperator:147 - Reconciliation #3534(watch) Kafka(kafka-archive-backup/archive-backup): Status of Kafka archive-backup in namespace kafka-archive-backup has been updated
2026-08-28 07:24:53 INFO AbstractOperator:498 - Reconciliation #3553(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup in namespace kafka-archive-backup was MODIFIED
2026-08-28 07:24:53 INFO AbstractOperator:521 - Reconciliation #3534(watch) Kafka(kafka-archive-backup/archive-backup): reconciled
2026-08-28 07:24:53 INFO AbstractOperator:242 - Reconciliation #3541(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup will be checked for creation or modification
2026-08-28 07:24:54 INFO CrdOperator:147 - Reconciliation #3536(watch) Kafka(env4/cluster): Status of Kafka cluster in namespace env4 has been updated
2026-08-28 07:24:54 INFO AbstractOperator:521 - Reconciliation #3536(watch) Kafka(env4/cluster): reconciled
2026-08-28 07:24:54 INFO AbstractOperator:498 - Reconciliation #3554(watch) Kafka(env4/cluster): Kafka cluster in namespace env4 was MODIFIED
2026-08-28 07:24:54 INFO AbstractOperator:242 - Reconciliation #3544(watch) Kafka(env4/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:54 INFO CrdOperator:147 - Reconciliation #3539(watch) Kafka(kafka-archive/archive): Status of Kafka archive in namespace kafka-archive has been updated
2026-08-28 07:24:54 INFO AbstractOperator:498 - Reconciliation #3555(watch) Kafka(kafka-archive/archive): Kafka archive in namespace kafka-archive was MODIFIED
2026-08-28 07:24:54 INFO AbstractOperator:521 - Reconciliation #3539(watch) Kafka(kafka-archive/archive): reconciled
2026-08-28 07:24:54 INFO AbstractOperator:242 - Reconciliation #3547(watch) Kafka(kafka-archive/archive): Kafka archive will be checked for creation or modification
2026-08-28 07:24:54 INFO AbstractOperator:498 - Reconciliation #3556(watch) Kafka(env5/cluster): Kafka cluster in namespace env5 was MODIFIED
2026-08-28 07:24:54 INFO AbstractOperator:498 - Reconciliation #3557(watch) Kafka(default/cluster): Kafka cluster in namespace default was MODIFIED
2026-08-28 07:24:54 INFO CrdOperator:147 - Reconciliation #3537(watch) Kafka(env5/cluster): Status of Kafka cluster in namespace env5 has been updated
2026-08-28 07:24:54 INFO CrdOperator:147 - Reconciliation #3538(watch) Kafka(default/cluster): Status of Kafka cluster in namespace default has been updated
2026-08-28 07:24:54 INFO AbstractOperator:521 - Reconciliation #3537(watch) Kafka(env5/cluster): reconciled
2026-08-28 07:24:54 INFO AbstractOperator:521 - Reconciliation #3538(watch) Kafka(default/cluster): reconciled
2026-08-28 07:24:54 INFO AbstractOperator:242 - Reconciliation #3545(watch) Kafka(env5/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:54 INFO AbstractOperator:242 - Reconciliation #3546(watch) Kafka(default/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:56 INFO AbstractOperator:498 - Reconciliation #3558(watch) Kafka(env3/cluster): Kafka cluster in namespace env3 was MODIFIED
2026-08-28 07:24:56 INFO CrdOperator:147 - Reconciliation #3540(watch) Kafka(env3/cluster): Status of Kafka cluster in namespace env3 has been updated
2026-08-28 07:24:56 INFO AbstractOperator:521 - Reconciliation #3540(watch) Kafka(env3/cluster): reconciled
2026-08-28 07:24:57 INFO AbstractOperator:242 - Reconciliation #3550(watch) Kafka(env3/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:24:57 INFO CrdOperator:147 - Reconciliation #3541(watch) Kafka(kafka-archive-backup/archive-backup): Status of Kafka archive-backup in namespace kafka-archive-backup has been updated
2026-08-28 07:24:57 INFO AbstractOperator:521 - Reconciliation #3541(watch) Kafka(kafka-archive-backup/archive-backup): reconciled
2026-08-28 07:24:57 INFO AbstractOperator:498 - Reconciliation #3559(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup in namespace kafka-archive-backup was MODIFIED
2026-08-28 07:24:57 INFO AbstractOperator:242 - Reconciliation #3548(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup will be checked for creation or modification
2026-08-28 07:24:58 INFO AbstractOperator:498 - Reconciliation #3560(watch) Kafka(env1/cluster): Kafka cluster in namespace env1 was MODIFIED
2026-08-28 07:24:58 INFO CrdOperator:147 - Reconciliation #3542(watch) Kafka(env1/cluster): Status of Kafka cluster in namespace env1 has been updated
2026-08-28 07:24:58 INFO AbstractOperator:521 - Reconciliation #3542(watch) Kafka(env1/cluster): reconciled
2026-08-28 07:24:58 INFO AbstractOperator:242 - Reconciliation #3551(watch) Kafka(env1/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:25:00 INFO AbstractOperator:498 - Reconciliation #3561(watch) Kafka(env2/cluster): Kafka cluster in namespace env2 was MODIFIED
2026-08-28 07:25:00 INFO CrdOperator:147 - Reconciliation #3543(watch) Kafka(env2/cluster): Status of Kafka cluster in namespace env2 has been updated
2026-08-28 07:25:00 INFO AbstractOperator:521 - Reconciliation #3543(watch) Kafka(env2/cluster): reconciled
2026-08-28 07:25:00 INFO AbstractOperator:242 - Reconciliation #3552(watch) Kafka(env2/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:25:00 INFO AbstractOperator:498 - Reconciliation #3562(watch) Kafka(kafka-archive/archive): Kafka archive in namespace kafka-archive was MODIFIED
2026-08-28 07:25:00 INFO CrdOperator:147 - Reconciliation #3547(watch) Kafka(kafka-archive/archive): Status of Kafka archive in namespace kafka-archive has been updated
2026-08-28 07:25:00 INFO AbstractOperator:521 - Reconciliation #3547(watch) Kafka(kafka-archive/archive): reconciled
2026-08-28 07:25:00 INFO AbstractOperator:242 - Reconciliation #3555(watch) Kafka(kafka-archive/archive): Kafka archive will be checked for creation or modification
2026-08-28 07:25:00 INFO CrdOperator:147 - Reconciliation #3544(watch) Kafka(env4/cluster): Status of Kafka cluster in namespace env4 has been updated
2026-08-28 07:25:00 INFO AbstractOperator:498 - Reconciliation #3563(watch) Kafka(env4/cluster): Kafka cluster in namespace env4 was MODIFIED
2026-08-28 07:25:00 INFO AbstractOperator:521 - Reconciliation #3544(watch) Kafka(env4/cluster): reconciled
2026-08-28 07:25:01 INFO AbstractOperator:242 - Reconciliation #3554(watch) Kafka(env4/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:25:01 INFO AbstractOperator:498 - Reconciliation #3564(watch) Kafka(env5/cluster): Kafka cluster in namespace env5 was MODIFIED
2026-08-28 07:25:01 INFO AbstractOperator:498 - Reconciliation #3565(watch) Kafka(default/cluster): Kafka cluster in namespace default was MODIFIED
2026-08-28 07:25:01 INFO CrdOperator:147 - Reconciliation #3545(watch) Kafka(env5/cluster): Status of Kafka cluster in namespace env5 has been updated
2026-08-28 07:25:01 INFO AbstractOperator:521 - Reconciliation #3545(watch) Kafka(env5/cluster): reconciled
2026-08-28 07:25:01 INFO CrdOperator:147 - Reconciliation #3546(watch) Kafka(default/cluster): Status of Kafka cluster in namespace default has been updated
2026-08-28 07:25:01 INFO AbstractOperator:521 - Reconciliation #3546(watch) Kafka(default/cluster): reconciled
2026-08-28 07:25:01 INFO AbstractOperator:242 - Reconciliation #3557(watch) Kafka(default/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:25:01 INFO AbstractOperator:242 - Reconciliation #3556(watch) Kafka(env5/cluster): Kafka cluster will be checked for creation or modification
2026-08-28 07:25:02 INFO AbstractOperator:498 - Reconciliation #3566(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup in namespace kafka-archive-backup was MODIFIED
2026-08-28 07:25:02 INFO CrdOperator:147 - Reconciliation #3548(watch) Kafka(kafka-archive-backup/archive-backup): Status of Kafka archive-backup in namespace kafka-archive-backup has been updated
2026-08-28 07:25:02 INFO AbstractOperator:521 - Reconciliation #3548(watch) Kafka(kafka-archive-backup/archive-backup): reconciled
2026-08-28 07:25:02 INFO AbstractOperator:242 - Reconciliation #3553(watch) Kafka(kafka-archive-backup/archive-backup): Kafka archive-backup will be checked for creation or modificationCR: ## please keep this in sync with archive-recovery-mm2 config
apiVersion: kafka.strimzi.io/v1
kind: KafkaMirrorMaker2
metadata:
name: 'mm2-archive'
labels:
app.kubernetes.io/part-of: 'kafka-archive'
spec:
version: '4.3.1'
replicas: 1
target:
alias: 'archive-kafka-ohs'
bootstrapServers: 'archive-kafka-bootstrap.kafka-archive:9095'
authentication:
type: scram-sha-512
username: 'mirrormaker-archive-local'
passwordSecret:
secretName: 'mirrormaker-archive-password'
password: password
groupId: 'archive-mirrormaker'
# # Use topic names which reflect the specifc mirrormaker instance in order to avoid
# conflicts between different mirrormaker procceses (e.g. geo-replication and archive).
# Default naming is required when having multiple replication flows in a single mirrormaker instance.
offsetStorageTopic: 'archive-mirrormaker-offsets.DC-archive-kafka-ohs.internal'
configStorageTopic: 'archive-mirrormaker-configs.DC-archive-kafka-ohs.internal'
statusStorageTopic: 'archive-mirrormaker-status.DC-archive-kafka-ohs.internal'
config:
# # Kafka Connect coordination timeout values
worker.sync.timeout.ms: 30000
task.shutdown.graceful.timeout.ms: 10000
## Due to KAFKA-13365 support for consumer and producer settings is very limited and won't work for all connect workers.
# Set max message size
max.partition.fetch.bytes: 15728640 # 15 MB
consumer.max.partition.fetch.bytes: 15728640 # 15 MB
max.request.size: 52428800 # 50 MB
producer.max.request.size: 52428800 # 50 MB
# Default flush timeout of 5s is too low.
offset.flush.timeout.ms: 30000
# Set connection idle time to 180s which is less than any popular load balancer / ingress uses as default
connections.max.idle.ms: 180000
consumer.connections.max.idle.ms: 180000
producer.connections.max.idle.ms: 180000
admin.connections.max.idle.ms: 180000
# Always use zstd compression for mirrored data
producer.compression.type: 'zstd'
replication.factor: 3
# # The replication factor for mm2 internal topics.
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offset-syncs.topic.replication.factor: 3
checkpoints.topic.replication.factor: 3
# The replication factor for connect internal topics.
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
config.storage.replication.factor: 3
offset.storage.replication.factor: 3
status.storage.replication.factor: 3
config.storage.min.insync.replicas: 2
offset.storage.min.insync.replicas: 2
status.storage.min.insync.replicas: 2
replication.policy.class: 'org.apache.kafka.connect.mirror.IdentityReplicationPolicy'
mirrors:
- source:
alias: 'eon-experimental-env1'
bootstrapServers: 'cluster-kafka-bootstrap.env1:9095'
authentication:
type: scram-sha-512
username: 'mirrormaker-archive-env1-remote'
passwordSecret:
secretName: 'mirrormaker-archive-env1-password'
password: password
sourceConnector:
tasksMax: 5
autoRestart:
enabled: true
config:
replication.policy.class: 'org.apache.kafka.connect.mirror.IdentityReplicationPolicy'
replication.factor: 3
# # The replication factor for mm2 internal topics.
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offset-syncs.topic.replication.factor: 3
checkpoints.topic.replication.factor: 3
offset-syncs.topic.location: 'target'
refresh.topics.enabled: 'true'
sync.topic.configs.enabled: 'false'
sync.topic.acls.enabled: 'false'
# checkpointConnector = disable checkpoints
topicsPattern: 'test|.*\.\Qarchive.topic1\E\..*|.*\.\Qarchive.topic2\E\..*|.*\.\Qarchive.topic3\E\..*'
groupsPattern: '.*'
logging:
type: inline
loggers:
rootLogger.level: 'WARN'
readinessProbe:
timeoutSeconds: 15
failureThreshold: 5
livenessProbe:
initialDelaySeconds: 60
timeoutSeconds: 15
failureThreshold: 5
periodSeconds: 60
jvmOptions:
'-XX':
MaxRAMPercentage: '75'
resources:
requests:
ephemeral-storage: '10Mi'
memory: '2Gi'
cpu: '200m'
limits:
ephemeral-storage: '100Mi'
memory: '2Gi'
template:
pod:
priorityClassName: 'criticality-level1'
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9404'
prometheus.io/path: '/metrics'
prometheus.io/container: 'mm2-archive-mirrormaker2'
prometheus.io/scrapetype: 'hostname'
tolerations:
- effect: NoExecute
key: DataComponentsOnly
operator: Equal
value: 'true'
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: strimzi.io/kind
operator: In
values:
- KafkaMirrorMaker2
topologyKey: kubernetes.io/hostname
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: 'node.custom/name'
operator: 'In'
values: [ 'data1.0' ]
- key: zone.custom/datanode
operator: In
values:
- '1'
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: metrics-mirrormaker
key: metrics_mirrormaker.yaml
Kafka Logs: |
All reactions
|
I tried to reproduce it, but it worked fine for me during the upgrade. RBAC issues are normally clear from the error message. So I do not think this would be an RBAC issue. There are some strange things in the logs you shared:
|
All reactions
|
Hi @scholzj, thank you very much for your help. I found the cause of the problem: my netpols blocks all ingress and egress by default. Until 1.0.0 the passwords that I use for authentication were mounted into the MM2 pods. Starting with 1.1.0 they the passwords are retrieved from the Kube API. After allowing egress from the MM2 pod to the Kube API, everything works fine. |
All reactions
|
Great. Glad you figured it out. |
Uh oh!
There was an error while loading. Please reload this page.
Hello!
I would appreciate help with the following issue. After updating to 1.1.0 (but the same problem occurs for 1.2.0 too), my MM2 crashes with the following error message:
The relevant section of the MM2 CR is:
In the
podSpec I see:so I tested whether the Token in the file has the necessary permissions:
and it does not seem to be a problem of permissions. Can you help to investigate further what might cause this Problem?
Thank you very much!
Malte
PS: I did not make any changes to the CR from 1.0.0 to 1.1.0, so this is not the culprit in my view
All reactions