Skip to content

build(docker): default to jemalloc; install mimalloc; add MS jaz java launcher#35410

Merged
wezell merged 8 commits intomainfrom
issue-32320-jemalloc-is-back
Apr 28, 2026
Merged

build(docker): default to jemalloc; install mimalloc; add MS jaz java launcher#35410
wezell merged 8 commits intomainfrom
issue-32320-jemalloc-is-back

Conversation

@wezell
Copy link
Copy Markdown
Member

@wezell wezell commented Apr 21, 2026

Installing jemalloc for its profiling — we can see what is eating off-heap mem when an OOM is issued.

Summary

This PR bundles two related runtime-image changes:

1. Allocator: jemalloc as default, mimalloc still installed

  • Install both libjemalloc2 and libmimalloc2.0 in the runtime image (dotCMS/src/main/docker/original/Dockerfile) and the dev-env image (docker/dev-env/Dockerfile).
  • Switch the default LD_PRELOAD in setenv.sh from mimalloc to libjemalloc.so.2. LD_PRELOAD is still overridable at runtime, so operators can swap back to mimalloc via env var without an image change.

Context: original issue #32320 adopted mimalloc after jemalloc's upstream archival; since then jemalloc has returned to active development, and we want it back as the default while keeping mimalloc installed for easy A/B.

2. JVM launcher: Microsoft jaz (Azure Command Launcher for Java)

  • Add Microsoft's apt repo (packages-microsoft-prod.deb) and install jaz in the runtime image.
  • In setenv.sh, default _RUNJAVA to jaz when present. Tomcat honors _RUNJAVA in place of $JRE_HOME/bin/java. jaz is a transparent shim that invokes java from PATH, adding crash-dump capture and arg validation.
  • Override path: set _RUNJAVA=/java/bin/java to bypass jaz.

Both changes target diagnostics for native memory and JVM crashes — the same problem space — which is why they ship together. Either piece can be disabled at runtime without rebuilding.

why jemalloc over mimalloc

No clear winner in general. They're both excellent and roughly comparable on real workloads.

  • Raw performance: For pure allocation throughput, mimalloc frequently edges out jemalloc.
  • Fragmentation / RSS: Roughly a tie. Both are far better than glibc. Mimalloc has a slight theoretical edge on small-object workloads.
  • Battle-testing on JVMs: jemalloc wins. It's been the de-facto choice for Cassandra, Elasticsearch, Kafka, Hadoop, and countless JVMs.
  • Tooling: jemalloc wins decisively. Its heap profiler is the reason I suggested it — it's the best open-source tool for answering "what's eating my native memory?" Mimalloc's diagnostics are minimal by comparison.

Ref: #32320

Test plan

  • Confirm image builds successfully (./mvnw install -pl :dotcms-core -DskipTests + docker build)
  • Start container and verify ldd / /proc/$PID/maps on the Tomcat process shows libjemalloc.so.2 preloaded
  • Verify both /usr/lib/<arch>-linux-gnu/libjemalloc.so.2 and /usr/lib/<arch>-linux-gnu/libmimalloc.so.2 are present in the final image
  • Smoke test: run dotCMS, confirm startup is clean and no allocator-related warnings in logs
  • Verify LD_PRELOAD=/usr/lib/.../libmimalloc.so.2 override still works for A/B testing
  • Confirm jaz is on PATH and Tomcat starts via _RUNJAVA=$(command -v jaz) (check ps -ef shows the jaz wrapper)
  • Verify _RUNJAVA=/java/bin/java override bypasses jaz cleanly

🤖 Generated with Claude Code

This PR fixes: #32320

…D to jemalloc

Install both libjemalloc2 and libmimalloc2.0 in the runtime and dev-env images,
and switch the default LD_PRELOAD in setenv.sh to libjemalloc.so.2. LD_PRELOAD
remains overridable at runtime, so operators can still swap back to mimalloc
via env var without an image change.

Ref: #32320

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 21, 2026

Claude finished @wezell's task in 1m 27s —— View job


PR Review

  • Read changed files and diff
  • Analyze for issues

Issues

1. No integrity check on the Microsoft .deb download — supply chain risk

dotCMS/src/main/docker/original/Dockerfile line 70:

wget -q "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo $VERSION_ID)/packages-microsoft-prod.deb" -O packages-microsoft-prod.deb && \
dpkg -i packages-microsoft-prod.deb && \

The .deb is downloaded and installed without any checksum or GPG signature verification. The file installs a new APT key + repo, so a compromised or MITM'd response would add an attacker-controlled repo to the image. At minimum, pin the expected SHA256 and sha256sum -c before dpkg -i. Alternatively, pin the key fingerprint after install. Fix this →

2. jaz has no version pin

dotCMS/src/main/docker/original/Dockerfile line 76:

apt-get install -y --no-install-recommends \
    postgresql-client-18 \
    jaz && \

postgresql-client-18 is pinned to a major version; jaz is not. An unversioned install means any update Microsoft ships between builds is silently picked up. Either pin jaz=<version> or document this as an intentional "always-latest" choice. Fix this →

3. Dev/prod parity gap — jaz not installed in the dev-env image

docker/dev-env/Dockerfile only picks up libjemalloc2 but does not install jaz or add the Microsoft repo. The setenv.sh jaz detection will silently fall back to plain java in dev. Any regression from jaz (arg validation quirks, crash-dump path issues) will only surface in production. Either install it in both images or add a comment explaining the intentional difference.

4. libarchive-tools listed twice in dev-env/Dockerfile (pre-existing, not cleaned up)

docker/dev-env/Dockerfile line 39 (unchanged):

vim libarchive-tools postgresql-common libmimalloc2.0 libjemalloc2 libarchive-tools

The PR touched this line but didn't remove the duplicate. Harmless but worth fixing while the line is open. Fix this →

5. command -v jaz called twice in setenv.sh

dotCMS/src/main/resources/container/tomcat9/bin/setenv.sh lines 115–117:

if [ -z "$_RUNJAVA" ] && command -v jaz >/dev/null 2>&1; then
  export _RUNJAVA="$(command -v jaz)"
fi

Minor: command -v jaz is invoked twice — once to test, once to resolve the path. Cache it in a variable to avoid the double lookup and the silent discard of any transient error on the second call. Fix this →


The allocator switch (jemalloc → default, mimalloc kept) and the LD_PRELOAD override pattern are clean. The setenv.sh jaz detection logic is correct in structure. The main concern worth addressing before merge is item 1 (integrity of the Microsoft package download).

@wezell wezell requested a review from yolabingo April 21, 2026 23:30
@wezell wezell requested a review from swicken April 21, 2026 23:33
@wezell wezell enabled auto-merge April 21, 2026 23:33
wezell and others added 5 commits April 22, 2026 08:52
Install Microsoft's Azure Command Launcher for Java (jaz) in the runtime
Dockerfile alongside the existing postgres-client. jaz is a transparent
shim over `java` that adds crash-dump capture and arg validation.

In setenv.sh, set _RUNJAVA to jaz when present so Tomcat invokes it in
place of $JRE_HOME/bin/java. _RUNJAVA remains overridable at runtime, so
operators can bypass jaz by setting _RUNJAVA=/java/bin/java.

Ref: #32320

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restores java=21.0.8-ms. The bump in 2ce5b08 was not intended to be
pushed; runtime/base image stays on 21 for this PR.

Ref: #32320

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first apt install block lost its cleanup when the postgresql-common
install was consolidated, so apt lists were being baked into the layer.
Re-append && rm -rf /var/lib/apt/lists/* to that RUN.

Ref: #32320

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The file was removed in 2ce5b08 unintentionally; restoring the prior
version (enabled=false) so build-cache config remains tracked.

Ref: #32320

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wezell wezell changed the title build(docker): install jemalloc + mimalloc; default LD_PRELOAD to jemalloc build(docker): default to jemalloc; install mimalloc; add MS jaz java launcher Apr 28, 2026
… install

Address PR review feedback by collapsing the postgres/jaz install steps into
a single RUN block:

- Remove packages-microsoft-prod.deb after dpkg install instead of leaving it
  in the image layer.
- Install postgresql-common in the same RUN where it is purged so it is not
  baked into intermediate layers.
- Drop apt-get update cache (rm -rf /var/lib/apt/lists/*) inside the same
  RUN where update was issued, instead of a later layer.
- Strip extraneous blank lines.

No functional change to the resulting image (same packages installed, same
purges); just smaller layers and tighter cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wezell wezell added this pull request to the merge queue Apr 28, 2026
Merged via the queue into main with commit f9ec566 Apr 28, 2026
49 checks passed
@wezell wezell deleted the issue-32320-jemalloc-is-back branch April 28, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

jemalloc is dead, long live mimalloc

2 participants