[Helm] Pulsar 4.0.x incompatible with milvus-helm 5.0.x — bookkeeper.conf and OTEL config not updated for 4.x class paths — cross-ref: milvus-helm#318

[chands08]

Context

Posting here to get broader visibility, as the original issue on the milvus-helm
repo has not received a response in over a week.

Original issue: zilliztech/milvus-helm#318

---

Summary

Upgrading the Pulsar image tag to any 4.0.x release causes two independent
CrashLoopBackOff failures in a milvus-helm 5.0.x deployment on EKS. This is
a blocker for CVE remediation — Pulsar 3.0.x LTS reached security EOL on
May 2, 2026, and the only supported LTS branch with a patched OpenSSL base
image is 4.0.x (supported through October 2027).

---

Environment

Component	Version
milvus-helm chart	5.0.23
Pulsar image	apachepulsar/pulsar:4.0.13
Milvus	2.6.19
Kubernetes	AWS EKS

---

Failures Observed

1. Bookie — ClassNotFoundException

org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider

The class path was renamed to:

org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider

in Pulsar 4.0.10 (ref: apache/pulsar PR #1868). The chart's bookkeeper.conf
still references the old path, causing the Bookie pod to crash on startup.

2. Broker — Unrecognized OTEL propagator

Unrecognized value for otel.propagators: xray

The AWS X-Ray OTEL propagator class was removed from the Pulsar 4.x classpath.
The chart's broker OTEL config still expects it, causing the Broker pod to
crash on startup.

---

Security Context

This is not a cosmetic upgrade. Pulsar 3.0.x LTS is past its 36-month security
support window (EOL: May 2, 2026). CVE-2026-45447 (OpenSSL heap use-after-free,
USN-8414-1, disclosed June 9, 2026) cannot be remediated on any 3.0.x base image.
Pulsar 4.0.x is the only currently supported LTS branch with a patched OpenSSL
base. The incompatibility in milvus-helm effectively blocks security compliance
for any production Milvus deployment using the Pulsar message store.

---

Ask

1. Update bookkeeper.conf statsProviderClass to the Pulsar 4.x class path
2. Update broker OTEL config to drop or make the xray propagator optional/configurable
3. Publish an official compatibility statement for Pulsar 4.0.x with milvus-helm 5.x

Any interim workaround (e.g., a values override that patches these configs without
forking the chart) would also be very welcome while a fix is in progress.

---

Cross-posted from milvus-helm#318. Happy to test any proposed fix or PR against
our EKS environment.

[jmrplens]

Both can be worked around from values, without forking the chart. I reproduced both crashes with apachepulsar/pulsar:4.0.13, using the ConfigMaps and container commands that milvus-helm 5.0.23 renders (it bundles the Apache Pulsar chart 3.3.0), and with these overrides the bookie, autorecovery, broker and proxy all start and the broker takes messages:

pulsarv3:
  bookkeeper:
    configData:
      PULSAR_PREFIX_statsProviderClass: org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider
  autorecovery:
    configData:
      PULSAR_PREFIX_statsProviderClass: org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider
  broker:
    configData:
      PULSAR_EXTRA_OPTS: "-Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacityPerThread=4096 -Dotel.propagators=tracecontext,baggage"
  proxy:
    configData:
      PULSAR_EXTRA_OPTS: "-Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacityPerThread=4096 -Dotel.propagators=tracecontext,baggage"

Bookie. The bundled chart hardcodes statsProviderClass in the bookie and autorecovery ConfigMaps. Putting the same key in configData renders it a second time in the ConfigMap, which only works because the last duplicate happens to win; the prefixed variable doesn't depend on that. The image's apply-config-from-env.py applies PULSAR_PREFIX_* variables after the plain ones and overwrites keys that already exist, and the bookie log shows Applying config statsProviderClass = org.apache.bookkeeper... followed by Updating config statsProviderClass = org.apache.pulsar.... Autorecovery needs it too, because it reads the same setting and dies with the same ClassNotFoundException. The rename came with the branch-4.0 Jetty 12 upgrade that shipped in 4.0.10 (apache/pulsar#25534). The Apache chart stopped setting the class in apache/pulsar-helm-chart#654 (chart 4.6.0), so the image's own value applies there, but milvus-helm still bundles 3.3.0. ZooKeeper's metrics provider was renamed in the same commit, but the chart doesn't set that one, so it needs nothing.

Broker. Neither milvus-helm nor the bundled Pulsar chart sets anything OpenTelemetry-related, so the xray value comes from an OTEL_PROPAGATORS variable that something in your cluster adds to the pod: kubectl get pod <broker-pod> -o yaml | grep -A1 OTEL_ will show it (kubectl exec won't work while it's crash-looping). On EKS my guess would be the ADOT / OpenTelemetry operator's auto-instrumentation injection. Pulsar 3.0 had no OpenTelemetry setup, so it never read the variable. 4.0 runs the OpenTelemetry SDK autoconfiguration at startup, which reads it and fails on any propagator that isn't on Pulsar's classpath, and the X-Ray one isn't. A -Dotel.propagators system property takes precedence over the environment variable, so it works however the variable is injected, whereas an OTEL_PROPAGATORS entry in configData would lose to one injected into the container's env. Setting PULSAR_EXTRA_OPTS replaces the default from conf/pulsar_env.sh, which is why its two flags are repeated. The 4.0 proxy has the same OpenTelemetry setup and crashes the same way when it gets the variable, hence the same line for it.

[chands08]

Hi @jmrplens , thanks for the update. I will get this applied in the environment and let you know the update. Meanwhile, I have a query for you. I understand you had provided the details for pulsar 3.0.7. Currently I had upgraded it to 3.0.17 to fix another CVE ID. Will this version bump will have any impact with the solution provided by you? I would like to reverify once before I perform this in the environment.

[jmrplens]

The version you start from doesn't change anything: the overrides are for the 4.0.x image, and that's what I tested them against (4.0.13). What matters is to apply them in the same helm upgrade that moves the Pulsar images to 4.0.x, not before, because the bookie one breaks 3.0.x.

I repeated the test with apachepulsar/pulsar:3.0.17 and the same milvus-helm 5.0.23 ConfigMaps:

• Without the overrides, the bookie, autorecovery and broker all start.
• With PULSAR_PREFIX_statsProviderClass, the bookie and autorecovery exit with ClassNotFoundException: org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider. 3.0.17 still ships the old class in its own bookkeeper.conf, and the new one only exists from 4.0.10 on.
• The -Dotel.propagators flag does nothing on 3.0.17. That version has no OpenTelemetry setup, so the broker started and took messages with OTEL_PROPAGATORS=tracecontext,baggage,xray set, with and without the flag.

So while you stay on 3.0.17 you don't need any of this, and the broker and proxy line is safe to add at any time.

[chands08]

Thanks for the insight. I ran it now in the system and below are the events actually happened during the upgrade attempt

Worked with your overrides (came up 1/1 on 4.0.13):

milvus-pulsarv3-bookie-0/1/2 — PULSAR_PREFIX_statsProviderClass was correctly rendered second in the ConfigMap and applied.
milvus-pulsarv3-recovery-0 (autorecovery) — same.
Zookeeper — no override needed, came up on 4.0.13.
Did NOT come up (both crash-looped with a different error, not OTEL):

milvus-pulsarv3-broker-1, milvus-pulsarv3-proxy-1 failed with:

sh: can't create status: Permission denied
Their StatefulSet arg is bin/apply-config-from-env.py conf/{broker,proxy}.conf; ... echo "OK" > status; .... WORKDIR=/pulsar in the 4.0.13 image, but /pulsar is root:root drwxr-xr-x while the container runs as uid=10000(pulsar) — so echo "OK" > status fails before the JVM (with your -Dotel.propagators fix) is ever invoked.

Verified inside a running 4.0.13 bookie: touch /pulsar/testfile → Unknown system error (RO filesystem semantics for that dir). Bookie doesn't hit it because its args don't contain echo "OK" > status — only broker/proxy do.

The ConfigMap overrides are correct — they render as expected — but there's a third issue the values patch doesn't cover: the broker/proxy pod command touches /pulsar/status from a non-writable dir on the 4.0.13 image.

[jmrplens]

Thanks for testing it. You're right, and I missed it because my broker test didn't run the chart's full command. Two details on how it fails: the broker still starts, because the echo is followed by ;, but the chart also sets statusFilePath: /pulsar/status, so /status.html returns 404 and the probes restart it. The proxy chains the echo with &&, so it exits before starting. The 4.x images no longer let the pulsar user (uid 10000) write to /pulsar, while 3.0.x did (chmod g+w /pulsar in its Dockerfile). The Apache chart fixed this on its side by moving the file to /pulsar/logs/status, which that user owns, but milvus-helm still bundles 3.3.0.

Values can't reach the proxy's command, so the least invasive fix I found without forking the chart is a Helm post-renderer that makes the same two edits as the upstream chart:

#!/bin/sh
# pulsar-status-fix.sh
exec sed -e 's#echo "OK" > status#echo "OK" > logs/status#' \
         -e 's#statusFilePath: "/pulsar/status"#statusFilePath: "/pulsar/logs/status"#'

With Helm 3, make it executable and add --post-renderer ./pulsar-status-fix.sh to your helm upgrade. With Helm 4, --post-renderer takes a plugin name instead, so put the script in a directory next to this plugin.yaml, run helm plugin install <that directory> and pass --post-renderer pulsar-status-fix:

apiVersion: v1
name: pulsar-status-fix
version: 0.1.0
type: postrenderer/v1
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: "${HELM_PLUGIN_DIR}/pulsar-status-fix.sh"

It changes exactly four lines, the two echos and the two statusFilePaths, which you can check by diffing helm template with and without it. I ran the chart's literal broker and proxy commands on 4.0.13 again: without it I get your can't create status: Permission denied, the proxy exits and the broker's /status.html returns 404; with it both stay up, /status.html returns 200, and producing through the proxy, which is what Milvus connects to, works.

[chands08]

Thanks @jmrplens for the update. Since, the earlier one errored out, I had reverted back to the older version. I will check this and keep you posted.

[chands08]

Also this approach needs attention while progressing to higher environment by incorporating these changes in the IaC pipeline.

[chands08]

Hi @jmrplens - I had tested it with the fix provided by you. It had fixed the issue surfaced earlier, but ended up with new one. Below is the update.

FAILED on a 4th workaround we hadn't anticipated
What worked (3 of 4 workarounds held)
✅ PULSAR_PREFIX_statsProviderClass fix — bookies + autorecovery on 4.0.13 Ready 1/1
✅ -Dotel.propagators=tracecontext,baggage fix — brokers on 4.0.13 Ready 1/1
✅ Post-renderer /pulsar/status → /pulsar/logs/status fix — proxies on 4.0.13 Ready 1/1 (dry-render verified all 4 line substitutions)
What broke (the 4th issue)
Zookeeper probe compatibility. Chart 3.3.0 renders ZK readiness + liveness probes as:

echo ruok | nc -q 1 localhost 2181 | grep imok
Pulsar 3.0.17 image shipped a nc that accepted -q. Pulsar 4.0.13 ships BusyBox 1.37.0 nc, which rejects -q → every probe fails with nc: unrecognized option: q. ZK-2 was functionally healthy (joined quorum, broadcast state), but K8s marked it 0/1, so the StatefulSet rolling update never progressed to ZK-1/ZK-0. Helm --wait was going to time out at 20 min.

[chands08]

Hi @jmrplens - Did you had a chance to check on the above issue?

[jmrplens]

Your understanding is right on all three points. I'd do 1 first and then 2: the dry render costs nothing and catches a sed that didn't match, and the one thing it can't show, the probe answering on a 4.0.13 ZooKeeper, can also be checked without touching the release.

For 1, I rendered milvus-helm 5.0.23 with the Pulsar images at 4.0.13 again and piped it through the script: exactly 6 lines change, and these are the counts on the result:

grep -c 'nc -q 1' fixed.yaml                                # 0
grep -c '/pulsar/status"' fixed.yaml                        # 0
grep -c 'echo "OK" > logs/status' fixed.yaml                # 2 (broker, proxy)
grep -c 'statusFilePath: "/pulsar/logs/status"' fixed.yaml  # 2 (broker and proxy ConfigMaps)
grep -c 'nc 127.0.0.1 2181' fixed.yaml                      # 2 (ZooKeeper readiness and liveness)

For the probe itself, a throwaway pod with the 4.0.13 image is enough, since the probe only talks to the ZooKeeper in its own container:

kubectl -n <ns> run zk-probe-test --image=apachepulsar/pulsar:4.0.13 --restart=Never --env="PULSAR_MEM=-Xms128m -Xmx128m" --command -- sh -c 'exec bin/pulsar zookeeper'
kubectl -n <ns> exec zk-probe-test -- bash -c '(echo ruok; sleep 1) | nc 127.0.0.1 2181 | grep imok'   # imok
kubectl -n <ns> exec zk-probe-test -- bash -c 'echo ruok | nc -q 1 localhost 2181 | grep imok'       # nc: unrecognized option: q
kubectl -n <ns> delete pod zk-probe-test

Give it a few seconds to start before the first exec. I ran the same thing with Docker today: the new line prints imok and the current one fails with that error.

If both come out like that, go for 2. The rollback path is the same as last time, and ZooKeeper still rolls one pod at a time starting with zookeeper-2, so watching that pod reach 1/1 tells you within a couple of minutes whether anything new turns up.

[chands08]

Hi @jmrplens - Good news. Now I had applied the fix and it works well. All the pods are up and running now with the latest pulsar version. I need the application team to perform some sanity checks and confirm if the e2e functionalities are working without any issues. I will keep you posted.

Meanwhile, I have a query for you. Have you tested the BK 4.14 → 4.17 ledger format compatibility specifically — their 14-probe test proved the pod lifecycle, but didn't test persistent-data upgrade. Also, this environment is an empty database. If I apply this in an environment with data, will it have any impact? if yes, do I need to perform any pre-requisites or precautions

[chands08]

Hi @jmrplens - Quick update. After the successful fix applied for CVE-2026-45447 last night, today as part of the security findings, it had generated the below new findings (CVE IDs). LOL.

CVE-2026-18798
CVE-2026-63076
CVE-2026-63075
CVE-2026-63073
CVE-2026-63072
CVE-2026-54874
CVE-2026-15308
CVE-2026-75803
CVE-2026-14457
CVE-2026-7210
CVE-2026-82209
CVE-2026-80229
CVE-2026-82208
CVE-2026-80255
CVE-2026-80231
CVE-2026-80230
CVE-2026-18924
CVE-2026-19931
CVE-2026-13608
CVE-2026-86140
CVE-2026-86139
CVE-2026-86144
CVE-2026-86143
CVE-2026-86138
CVE-2026-86142

All these are consolidated and below are the findings and related fixes. Do you have any other better alternatives? Just wanted to check with you.

libxml2: 2.13.9-r2   - fixed in 2.15.4
curl: 8.21.0-r0 - fixed in 8.22.0
python3: 3.14.5-r0  -  fixed in 3.14.7-r0
openssl: 3.5.7-r0 - fixed in 4.0.2, 3.6.4, 3.5.8

[jmrplens]

Great to hear it's up. No, I haven't tested that: my lab started empty too, so it says nothing about ledgers that already hold data. Which upgrade you're facing depends on what the data environment runs, though. BookKeeper 4.14 is what Pulsar 2.9.5 ships, the image of milvus-helm's v2 subchart (pulsar); the default image of pulsarv3, 3.0.7, ships 4.16.6, and 4.0.13 ships 4.17.3.

• If it runs the v2 subchart, moving to pulsarv3 doesn't upgrade those bookies at all. pulsarv3 is a new set of StatefulSets with its own PVCs, so nothing ever reads the 4.14 ledgers. Milvus documents this move in Upgrading Pulsar: stop all writes, persist what is still only in Pulsar with /api/v1/persist until /api/v1/persist/state returns "flushed":true, uninstall, delete the v2 Pulsar PVCs, and install again with pulsarv3. Our overrides go in that values file and nothing else changes.
• If it already runs pulsarv3 on 3.0.7, it is an in-place upgrade from 4.16.6 to 4.17.x on the same volumes, which neither of us has tested with data. I'd still start with the same two steps from that guide, stop writes and persist: once everything is flushed, Milvus has what it needs in object storage, which is why the guide can delete Pulsar's volumes outright. That also gives you a way out if the bookies misbehave after the roll.

On the CVEs, I checked the four packages, not the IDs one by one. 4.0.14 came out on 5 October with curl 8.22.0, OpenSSL 3.5.9 and Python 3.14.8, so moving the five image tags from 4.0.13 to 4.0.14 clears three of the four. It's the same line (BookKeeper 4.17.3 to 4.17.4), and the overrides and the post-renderer still apply: same statsProviderClass, same read-only /pulsar, same BusyBox nc.

libxml2 is still 2.13.9-r2 in 4.0.14, and that is the newest build in Alpine 3.24, the base of both images, so no Pulsar image or apk upgrade fixes it today. Nothing in the image needs it except bind-tools (dig, host, nslookup), and I wouldn't remove that package to get rid of it: the chart's init jobs and the broker and proxy commands call nslookup, and the image's nslookup is the one from bind-tools. So until Alpine ships a fixed build, it's either an accepted finding or a derived image with apk upgrade once that build exists.

[chands08]

Hi @jmrplens - Thanks for your incredible support. I really appreciate your patience to help me out here. I am keeping this on hold because the libxml2 had introduced 6 different CVEs. I will have this working solution on hold until the libxml2 issue gets fixed. I will keep you posted on the further updates.