[Bug] Docker containers never restart when Java process crashes — broken cron monitor + tail -f /dev/null supervision

[bitflicker64]

[Bug] Docker containers never restart when Java process crashes — broken cron monitor + tail -f /dev/null supervision

Before submit

• I have confirmed and searched that there are no similar problems in the historical issue and documents

---

Environment

• HugeGraph version: 1.7.0+
• Deployment: Docker / docker-compose
• Related PR: perf(docker): improve all images build cache efficiency #3025

---

Expected & Actual behavior

Expected:
When the HugeGraph Java process crashes inside a Docker container, the container should exit and Docker's restart policy (restart: unless-stopped) should automatically bring it back up.

Actual:
The container stays in Up state permanently even after Java crashes. docker ps shows green. Users get Connection refused. The container never restarts on its own. Manual intervention is required every time.

# Java crashes at T=0:30
$ docker ps
CONTAINER ID   IMAGE              STATUS               PORTS
abc123         hugegraph/server   Up 2 hours           0.0.0.0:8080->8080/tcp

# Container looks healthy but:
$ curl http://localhost:8080/versions
curl: (7) Failed to connect to localhost port 8080: Connection refused

---

Root Cause Analysis

There are three compounding problems:

1. crond is never started — the watchdog is completely dead

cron is installed in all four Dockerfiles but dumb-init only launches docker-entrypoint.sh. Nobody starts crond. So even if start-hugegraph.sh -m true were called, start-monitor.sh registers the crontab job but since crond is not running, monitor-hugegraph.sh never fires. The entire watchdog silently does nothing in containers.

What happens on a VM:          What happens in Docker:
  crond reads crontab every      crond is NOT running
  minute                         monitor-hugegraph.sh NEVER fires
  monitor-hugegraph.sh fires     HugeGraph stays dead forever
  HugeGraph gets restarted

2. tail -f /dev/null means zero supervision

All three docker-entrypoint.sh files background the Java process then sleep forever:

# hugegraph-server entrypoint (current)
./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
# ... post-startup checks ...
tail -f /dev/null   # ← keeps container alive with NO watchdog

When Java crashes, tail -f /dev/null keeps running. The container never exits. Docker's restart: unless-stopped only triggers on container exit — since the container never exits, the restart policy never fires. The container stays Up (unhealthy) forever.

3. HEALTHCHECK only exists in docker-compose.yml, not in the Dockerfiles

Health checks are defined per service in docker-compose.yml but none of the four Dockerfiles have a HEALTHCHECK instruction. So docker run without compose has no health reporting at all. depends_on: condition: service_healthy only works because compose injects the check at runtime — it is not baked into the image.

4. Foreground mode is broken in start-hugegraph.sh

start-hugegraph.sh has a -d false foreground flag but it is broken. $!, pid file write, trap, wait_for_startup, disown, and OPEN_MONITOR all run unconditionally after the daemon/foreground if/else block — meaning in foreground mode they all execute after Java has already exited, with empty/stale values. Java's exit code is lost and the script always exits 0.

5. No foreground mode exists at all in start-hugegraph-pd.sh and start-hugegraph-store.sh

Both scripts always background Java unconditionally with exec java ... & regardless of any flag. There is no -d flag and no foreground path.

---

Impact

CURRENT — Java crashes inside container:
  T=0:30  Java crashes (OOM, segfault, deadlock, etc.)
  T=0:30  tail -f /dev/null keeps running
  T=0:30  Container stays "Up" — Docker sees nothing wrong
  T=1:00  HEALTHCHECK marks container "unhealthy" (compose only)
  T=∞     Container stays unhealthy forever, never restarts
          docker ps shows: Up 2 hours (unhealthy)
          Users get: Connection refused

AFTER FIX — Java crashes inside container:
  T=0:30  Java crashes
  T=0:30  Entrypoint exits → dumb-init exits → container exits
  T=0:30  Docker restart policy fires immediately
  T=0:31  New container starts
  T=1:41  docker ps shows: Up 1 min (healthy)

---

Additional bug found during investigation

The shipped default conf/rest-server.properties has:

restserver.url=127.0.0.1:8080

No http:// scheme. On macOS, curl fails immediately with "Protocol not supported" causing wait_for_startup to always time out and start-hugegraph.sh to exit 1 even though the server starts fine. Every other config in the repo uses http:// explicitly — raft CI configs, the Dockerfile sed patch, cluster test templates, and the Java ServerOptions default. The shipped default is inconsistent and breaks local macOS development.

---

Related

• Follow-up to perf(docker): improve all images build cache efficiency #3025
• Deferred from #3025 discussion (comment by @imbajin ): "Either drop cron or start it in docker-entrypoint.sh" → decision was to keep cron in perf(docker): improve all images build cache efficiency #3025 (comment) and fix properly in a follow-up

[dosubot]

This is a confirmed architectural issue in HugeGraph's Docker setup. The root cause is clear from the entrypoint scripts:

All three components (Server, Store, PD) use tail -f /dev/null as the final command in their entrypoint scripts, keeping the container's PID 1 alive indefinitely even after Java crashes [1] [2] [3].

The startup scripts (start-hugegraph-store.sh, start-hugegraph-pd.sh) background the Java process with & [4] [5], so even though exec is used, the shell continues running since & prevents the process replacement. Docker sees PID 1 (tail -f /dev/null) as alive → container stays "Up" → restart policies never trigger.

Workaround until this is fixed upstream: Add a healthcheck to your docker-compose.yml and combine it with restart: unless-stopped:

services:
  hugegraph-server:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/versions"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 30s
    restart: unless-stopped

Note: Docker's restart policy alone won't help here since the container never exits. The healthcheck will mark it unhealthy, but you'd need an external tool (like autoheal) to actually restart unhealthy containers, or use Docker Swarm/Kubernetes which natively restart unhealthy containers.

Proper fix would involve modifying the entrypoint scripts to:

1. Remove tail -f /dev/null
2. Run Java as PID 1 (foreground, no &) so the container exits when Java dies
3. Use dumb-init (already installed in Store/PD images) for signal handling

There's also an unused cron-based monitor in the Server component (enabled via -m flag in start-hugegraph.sh), but it's not activated in the Docker entrypoint and only covers Server — not Store or PD.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[bitflicker64]

Implementation Plan

Breaking the work into independently reviewable and testable chunks. Each chunk has its own test script, can be reviewed in isolation, and builds on the previous one.

---

Chunk 1 — Fix start-hugegraph.sh foreground mode

File: hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh

Problem: All post-branch logic (PID="$!", pid file write, trap, wait_for_startup, disown, OPEN_MONITOR) runs unconditionally after the daemon/foreground if/else block. In foreground mode the script blocks at hugegraph-server.sh until Java exits, then all those lines run with $! empty/stale, wait_for_startup immediately fails (Java already dead), and the script exits 0 losing Java's exit code entirely.

Fix: Move all post-branch logic inside the DAEMON == "true" branch. Add exit $? to the foreground branch so Java's exit code propagates out.

Test: hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph.sh

---

Chunk 2 — Add -d flag to start-hugegraph-pd.sh

File: hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh

Problem: No foreground mode exists — Java is always backgrounded unconditionally with exec java ... &. No -d flag. wait_for_startup exists in pd's util.sh but is never called — daemon startup has no health polling at all, callers compensate with hardcoded sleep 10.

Fix: Add DAEMON="true" default (consistent with existing style in this file), add -d to getopts, split the exec java block into daemon/foreground branches, wire up wait_for_startup in daemon mode, add exit $? to foreground branch.

Test: hugegraph-pd/hg-pd-dist/src/assembly/travis/test-start-hugegraph-pd.sh

---

Chunk 3 — Add -d flag to start-hugegraph-store.sh

File: hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh

Problem: Identical to chunk 2 — no foreground mode, always backgrounds Java unconditionally.

Fix: Identical structure to chunk 2.

Test: hugegraph-store/hg-store-dist/src/assembly/travis/test-start-hugegraph-store.sh

---

Chunk 4 — Fix pd entrypoint

File: hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh

Problem:

./bin/start-hugegraph-pd.sh -j "${JAVA_OPTS:-}"
tail -f /dev/null   # ← zero supervision

Fix: Pass -d false now that pd has foreground mode. Remove tail -f /dev/null. When Java exits the script exits, container exits, restart policy fires.

./bin/start-hugegraph-pd.sh -d false -j "${JAVA_OPTS:-}"
# no tail needed — blocks until Java exits, exits with Java's exit code

---

Chunk 5 — Fix store entrypoint

File: hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh

Problem: Same as chunk 4.

Fix: Same as chunk 4 but for store.

---

Chunk 6 — Fix server entrypoint

File: hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh

Problem:

./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
# post-startup checks including wait-partition.sh for hstore...
tail -f /dev/null   # ← zero supervision

Why -d false alone doesn't work here: wait-partition.sh must run after the server is already up in the background. The structure must stay: start in background → post-startup checks → then supervise.

Fix: Keep the background start and all post-startup checks unchanged. Replace only tail -f /dev/null with wait $(cat ./bin/pid). When Java exits wait returns immediately, entrypoint exits with non-zero, container exits, restart policy fires.

./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120

# post-startup checks preserved exactly as-is
ACTUAL_BACKEND=$(...)
if [[ "${ACTUAL_BACKEND}" == "hstore" ]]; then
    ./bin/wait-partition.sh || log "WARN: partitions not assigned yet"
fi

# replace tail -f /dev/null:
wait "$(cat ./bin/pid)"
exit $?

---

Chunk 7 — Add HEALTHCHECK to all four Dockerfiles

Files: hugegraph-server/Dockerfile, hugegraph-server/Dockerfile-hstore, hugegraph-pd/Dockerfile, hugegraph-store/Dockerfile

Problem: No HEALTHCHECK in any Dockerfile. Health checks only exist in docker-compose.yml. Plain docker run has no health reporting. depends_on: condition: service_healthy relies on compose injecting the check at runtime rather than it being baked into the image.

Fix: Add HEALTHCHECK to each Dockerfile before ENTRYPOINT, mirroring the exact values already in docker-compose.yml:

# hugegraph-pd/Dockerfile
HEALTHCHECK --interval=10s --timeout=5s --retries=12 --start-period=30s \
    CMD curl -fsS http://localhost:8620/v1/health >/dev/null || exit 1

# hugegraph-store/Dockerfile
HEALTHCHECK --interval=10s --timeout=10s --retries=30 --start-period=60s \
    CMD curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1

# hugegraph-server/Dockerfile + Dockerfile-hstore
HEALTHCHECK --interval=10s --timeout=5s --retries=30 --start-period=60s \
    CMD curl -fsS http://localhost:8080/versions >/dev/null || exit 1

---

Chunk 8 — Remove cron from all four Dockerfiles

Files: All four Dockerfiles

Problem: cron is installed but crond is never started — it has no purpose in the image now that supervision is Docker-native.

Fix: Remove cron from the apt-get install step in all four Dockerfiles. Keep dumb-init, procps, curl, lsof.

What stays untouched:

• monitor-hugegraph.sh — preserved, still valid for VM deployments
• start-monitor.sh — preserved, still valid for VM deployments
• docker-compose.yml healthcheck definitions — already correct, no changes needed

---

Chunk 9 — Fix default restserver.url scheme

File: hugegraph-server/hugegraph-dist/src/assembly/static/conf/rest-server.properties

Problem: Shipped default is restserver.url=127.0.0.1:8080 — no http:// scheme. wait_for_startup passes this raw value to curl. On macOS, curl fails with "Protocol not supported" causing start-hugegraph.sh to always exit 1 on local development even though the server starts correctly. On Linux, curl happens to handle scheme-less URLs by defaulting to HTTP — masking the bug.

Every other config in the repo uses http:// explicitly: raft CI configs, the Dockerfile sed patch, cluster test templates, and Java ServerOptions default. The shipped default is the only inconsistency.

Fix: Change line 3 of rest-server.properties from 127.0.0.1:8080 to http://127.0.0.1:8080.

---

Chunk 10 — Docs

Update relevant README or docker/ docs to explain:

• Docker deployments use native process supervision — the entrypoint watches the Java process and exits when it dies, triggering restart: unless-stopped
• HEALTHCHECK is now baked into the Dockerfiles so docker ps reports health without needing compose
• start-hugegraph.sh -m true (cron-based monitor) is for VM/bare-metal deployments only — unchanged and fully functional

---

Blast radius

All existing callers of start-hugegraph.sh, start-hugegraph-pd.sh, and start-hugegraph-store.sh pass no -d flag and rely on the default DAEMON="true". Daemon mode behavior is completely unchanged. Only new callers that explicitly pass -d false (the Docker entrypoints after this PR) are affected by chunks 1-3.

[bitflicker64]

@dosu what do u think about the implementation plan

[dosubot]

The plan is well-structured and technically sound. A few observations after verifying against the current code:

Chunk 1 — Looks correct. The server's start-hugegraph.sh already has a DAEMON variable and -d flag [1], but the post-branch logic (PID="$!", trap, wait_for_startup, disown, monitor) does run unconditionally after the if/else block [2]. Moving that into the daemon branch and adding exit $? to the foreground branch is the right fix.

Chunks 2 & 3 — Confirmed. PD and Store both unconditionally background Java with & and have no -d flag [3] [4]. Adding a daemon/foreground split mirrors the pattern already established in the server script.

Chunk 6 — One nuance worth noting. The plan uses wait "$(cat ./bin/pid)" after the post-startup checks. This works because start-hugegraph.sh writes the PID to $BIN/pid [5]. However, since dumb-init is the ENTRYPOINT in PD/Store Dockerfiles [6] but not in the server Dockerfile, you'll want to confirm that wait on a child PID works correctly under the shell that runs the entrypoint. If the server Dockerfile doesn't use dumb-init, the backgrounded Java process is a direct child of the entrypoint bash shell, so wait $PID should work fine. Just noting it's a difference from the PD/Store containers.

Chunk 8 — Confirmed. Both PD and Store Dockerfiles do install cron via apt-get [7] [8], and the service is started during init. Removing it is clean since supervision is now Docker-native.

Chunk 9 — Confirmed. The shipped default is indeed 127.0.0.1:8080 with no scheme [9]. Adding http:// aligns it with every other config in the repo.

Overall assessment: The blast radius analysis is accurate — all existing callers use the default DAEMON="true" so daemon mode is untouched. The chunking is well-sequenced (scripts first, then entrypoints that depend on them, then Dockerfiles, then docs). Each chunk is independently testable.

One suggestion: in Chunk 6, consider adding a comment in the entrypoint explaining why the server uses the start-in-background → post-checks → wait $PID pattern instead of -d false like PD/Store, since future maintainers might otherwise "simplify" it and break wait-partition.sh.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[imbajin]

Thanks for opening this. I think this is a valid follow-up to #3025, not something already fixed by it.

#3025 intentionally kept runtime behavior unchanged and focused on Docker build/cache hygiene. This issue is about a different layer: Docker runtime lifecycle and process supervision.

Core Context

The main problem is that the Java process lifecycle is currently decoupled from the container lifecycle.

At the moment, the Docker entrypoints start the HugeGraph Java process in the background, then keep the container alive with tail -f /dev/null.

That means when Java dies, Docker still sees the container as running, so restart: unless-stopped does not trigger.

Current behavior

┌──────────────────────────────┐
│ dumb-init / docker-entrypoint │
└──────────────┬───────────────┘
               │
               ▼
       start-*.sh starts Java &
               │
               ▼
       tail -f /dev/null keeps running
               │
               ▼
        Java crashes or exits
               │
               ▼
       container still appears Up
               │
               ▼
       Docker restart policy is not triggered
               │
               ▼
       users get connection refused

Expected Docker-native behavior should be closer to:

Expected behavior

┌──────────────────────────────┐
│ dumb-init / docker-entrypoint │
└──────────────┬───────────────┘
               │
               ▼
       supervise actual Java process
               │
               ▼
        Java crashes or exits
               │
               ▼
          entrypoint exits
               │
               ▼
          container exits
               │
               ▼
       Docker restart policy restarts it

Conclusion

I agree this should be treated as a real Docker runtime lifecycle bug.

The key points are:

• tail -f /dev/null hides Java process failure from Docker.
• Docker restart policies only trigger when the container exits.
• A Compose/Docker healthcheck helps observability, but ordinary Docker does not restart a container just because it is unhealthy.
• The legacy cron monitor is not a good Docker supervision model unless cron is actually started, and even then it is less aligned with Docker-native lifecycle handling.
• PD and Store need a real foreground mode.
• Server needs extra care because the hstore path has post-start checks such as wait-partition.sh; it likely needs the pattern: start in background, run post-start checks, then wait on the actual Java PID.

Suggested Fix Direction

A clean fix should probably be split like this:

1. Fix startup script foreground semantics

   • Keep daemon mode as the default for existing callers.
   • Fix start-hugegraph.sh -d false so it propagates the Java exit code correctly.
   • Add foreground mode to start-hugegraph-pd.sh.
   • Add foreground mode to start-hugegraph-store.sh.

2. Fix Docker entrypoints

   • PD / Store: run the Java process in foreground mode and remove tail -f /dev/null.
   • Server: preserve the current post-start hstore checks, then wait on the real HugeGraphServer PID instead of tailing forever.

3. Document the lifecycle model

   • HEALTHCHECK reports health.
   • restart: unless-stopped restarts only after container exit.
   • Docker deployments should rely on container lifecycle, not a hidden background Java process plus tail.

Validation

The fix should include a runtime restart validation, not only a Docker build check.

Example validation flow:

docker build -f hugegraph-server/Dockerfile -t hugegraph/server:test .

HUGEGRAPH_VERSION=test docker compose -f docker/docker-compose.yml up -d

# Kill the Java process, not the container or entrypoint.
docker exec hg-server pkill -f HugeGraphServer

# The container should exit and be restarted by Docker policy.
docker inspect hg-server \
  --format "{{.RestartCount}} {{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}"

# Service should eventually recover.
curl -fsS http://localhost:8080/versions

Expected result after the fix:

• The Java process death causes the entrypoint to exit.
• The container exits instead of staying Up with a dead service.
• RestartCount increases.
• /versions becomes reachable again after restart.

The same lifecycle check should be covered for Server, PD, and Store. For Server with hstore, we should also verify that the existing post-start partition wait logic is not skipped.

[bitflicker64]

@imbajin Thanks for the validation and the runtime restart test suggestion. I'll start with the startup script changes (chunks 1–3), get the foreground mode behavior and tests in place first, then move on to the Docker entrypoint/runtime lifecycle changes in a follow-up PR.

[bitflicker64]

hey @imbajin chunk 1 PR up: #3044
fixes foreground mode in start-hugegraph.sh, adds test script (all green on CI). chunks 2–3 next.