Exit JVM and dump heap on OutOfMemoryError for server containers

[jcschaff]

Summary

Add -XX:+ExitOnOutOfMemoryError, -XX:+HeapDumpOnOutOfMemoryError, and -XX:HeapDumpPath=/dump to the long-running server JVMs (data, api, submit, sched, db, rest). On OutOfMemoryError the JVM now writes a heap dump to /dump and exits cleanly so K8s recreates the container, instead of limping along in undefined state.

Why

Confirmed via 213 days of Loki retention scan that the prod data-pod wedges on 2026-05-04 and 2026-05-06 were both precipitated by Java heap space errors corrupting the JVM-wide static InactivityMonitor.READ_CHECK_TIMER mid-TimerTask:

06:39:39  ERROR  Java heap space
06:39:54  ERROR  Java heap space
06:40:38  ERROR  Java heap space (×4)
06:41:27.157  ERROR  Scheduled task error              ← TimerThread's task hit OOM, thread died
06:41:27.188  WARN   FailoverTransport ... after: 1 attempt(s) with Timer already cancelled.
[wedge from this point on; manual restart required]

With ExitOnOutOfMemoryError, the JVM aborts on the first OOM before the InactivityMonitor's TimerThread can be touched, eliminating the OOM-driven path to the wedge entirely. The companion JmsFailoverWatchdog in #1681 remains as defense-in-depth for non-OOM wedge causes.

HeapDumpOnOutOfMemoryError writes a .hprof file to /dump immediately before the JVM aborts — keeps a postmortem artifact for analysis (Eclipse MAT, VisualVM, JXRay, etc.) so we can find what's actually consuming the heap.

Companion PR (already merged)

virtualcell/vcell-fluxcd adds a /dump emptyDir volume mount (sizeLimit 4Gi, default node-disk backing) to all six affected Deployments and bumps the data container's resources.limits.memory from 3000Mi to 8000Mi. That PR is merged; without these JVM flags it's simply a latent mount and extra heap headroom — no behavior change. This PR activates the actual exit-on-OOM behavior on the next image roll.

Scope

Dockerfile	Change
docker/build/Dockerfile-data-dev	3 flags after -XX:MaxRAMPercentage=80
docker/build/Dockerfile-api-dev	same
docker/build/Dockerfile-submit-dev	same
docker/build/Dockerfile-sched-dev	same
docker/build/Dockerfile-db-dev	same
vcell-rest/src/main/docker/Dockerfile.jvm	appended to JAVA_OPTS
vcell-rest/src/main/docker/Dockerfile.legacy-jar	appended to JAVA_OPTS

Skipped intentionally: Dockerfile-batch-dev (per-job SLURM lifecycle, no /dump volume), Dockerfile-clientgen-dev and docker/build/admin (build/utility tools), Dockerfile.native and Dockerfile.native-micro (Quarkus native image — -XX: flags don't apply to GraalVM builds).

Test plan

• git diff --stat confirms 7 files, 20 inserts / 2 deletes
• grep verification that all 3 flags landed in each of the 7 Dockerfiles
• CI image build succeeds for each Dockerfile (relies on existing CI)
• After release tag (probably 7.7.0.76), fluxcd auto-rolls dev. Verify on the new data pod:

  kubectl -n dev exec deployment/data -- cat /proc/1/cmdline | tr '\0' ' ' | grep -oE 'ExitOnOutOfMemoryError|HeapDumpOnOutOfMemoryError|HeapDumpPath=/dump'

  Should print all three flags.
• kubectl -n dev exec deployment/data -- ls -la /dump shows the directory empty and writable.
• Promote to stage, then prod.
• On the next OOM event in any environment, confirm:
  • JVM stderr in Loki: Aborting due to java.lang.OutOfMemoryError
  • Heap dump file present at /dump/java_pid1.hprof (use kubectl cp to retrieve)
  • K8s pod restart event in kubectl describe pod and in Loki's kubectl container stream

🤖 Generated with Claude Code