ServerInfo is missing WARN floods logs continuously after startup with HBase backend

[linmengmeng-1314]

Description

After starting HugeGraph Server with HBase as the backend, the task-scheduler-1 thread continuously prints ServerInfo is missing WARN logs every second, and the situation never recovers.

Environment

• HugeGraph Server: 1.7.0
• Backend: HBase 2.1.2
• OS: CentOS Stream 9 (kernel 5.14.0-686.el9.x86_64)
• Deployment: Docker

Steps to Reproduce

1. Deploy HugeGraph Server with HBase backend via Docker
2. Start the server container
3. Observe the logs after startup completes

Expected Behavior

The ServerInfo should be successfully persisted to HBase during initialization and readable by task-scheduler, or the WARN should stop after a successful retry.

Actual Behavior

The server starts normally and the REST API works fine, but logs are flooded with WARN messages every second:

2026-05-20 02:51:39 [task-scheduler-1] [WARN] o.a.h.t.ServerInfoManager - ServerInfo is missing: DEFAULT-aikg/server-1
2026-05-20 02:51:39 [task-scheduler-1] [WARN] o.a.h.t.ServerInfoManager - ServerInfo is missing: DEFAULT-aikg/server-1, may be cleared before

This repeats indefinitely and never recovers.

Relevant Logs

The startup shows the ServerInfo init attempt:

2026-05-20 02:51:34 [main] [INFO] o.a.h.StandardHugeGraph - Init server info [server-1-MASTER] for graph 'DEFAULT-aikg'...
2026-05-20 02:51:34 [main] [INFO] o.a.h.t.ServerInfoManager - Init server info: HugeServerInfo(DEFAULT-aikg/server-1){server_load=0, server_role=MASTER, ...}

Then the WARN starts ~5 seconds later and never stops.

Workaround

Suppress the WARN by setting the log level to ERROR in log4j2.xml:

<AsyncLogger name="org.apache.hugegraph.task.ServerInfoManager" level="ERROR" additivity="false"/>

Notes

• Core functionality (graph API, Gremlin queries) is not affected.
• The issue appears to be specific to the HBase backend — the ServerInfo write during initialization may not be properly persisted or there may be a timing/race condition with task-scheduler.
• The message "may be cleared before" suggests the code is aware of this scenario but does not handle recovery properly.

[dosubot]

This is a recently reported issue that matches what was tracked in #3007 [1]. The root cause is a race condition between ServerInfo initialization and backend readiness — the task-scheduler heartbeat tries to read the ServerInfo system vertex before the backend (HBase in your case, hstore in #3007) has fully persisted it, causing the read to return null and the WARN to loop indefinitely.

Key details from the code: The heartbeat in ServerInfoManager (lines 184–213) checks selfServerInfo() every second. On the master node, if it returns null, it only logs the warning and does not attempt to re-save — unlike worker nodes which have a lazy re-init path. This is why the warning never recovers on its own for your master node.

Fixes have been merged to master addressing this:

• PR #1767 added lazy re-initialization during heartbeat when ServerInfo is missing and introduced defensive null checks.
• PR #3027 fixed server ID collisions when different graphs share the same backend by scoping node IDs by graph name — which matches your log showing DEFAULT-aikg/server-1.

Recommendations:

1. Rebuild from the latest master branch — the fixes should resolve this. The same fix resolved it for the user in hugegraph-server 启动不正常，查看日志有报错 #3007.
2. If you're using a released 1.7.0 binary (not built from latest master), the fix may not be included yet.
3. Your log4j2 workaround is safe in the meantime since, as you noted, core functionality is unaffected — the ServerInfo is only used for task scheduling/load balancing metadata.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[linmengmeng-1314]

Update:

I am using a custom image built from the latest master branch (tagged as hugegraph/hugegraph:1.7.0-amd-fixed for Docker deployment), not the 1.7.0 release. So the version in the issue description should be master (latest) rather than 1.7.0.

This also means the fixes mentioned in #1767 and #3007 are already included in my build, but the issue still occurs with HBase 2.1.2 backend. The fix may not fully cover the HBase backend scenario.

[dosubot]

Thanks for clarifying — this is helpful. Since your build already includes the PR #1767 fixes, the issue is likely HBase-specific and stems from a different root cause than the hstore bug in #3007.

Looking at the code, the HBase backend's commit() uses table.batch(rows, results) which is synchronous but does not guarantee immediate read visibility [1] — particularly during startup when HBase regions may still be initializing. The HBase backend also has no true transaction support (beginTx() is a no-op) [2], so there's a timing window where:

1. initServerInfo() successfully writes the ServerInfo vertex via table.batch()
2. The task-scheduler heartbeat fires ~1 second later and queries the same vertex
3. The read returns null because HBase hasn't made the write fully visible yet
4. Once it enters this "missing" state, the master node heartbeat path only logs the warning — it doesn't attempt to re-save [3], so it never recovers

This is fundamentally different from RocksDB (immediate in-memory visibility) or MySQL (ACID commit guarantees). The fix from PR #1767 added lazy re-initialization for worker nodes but the master node path still just logs and waits.

Potential fix approaches:

1. Add retry/backoff logic in ServerInfoManager.serverInfo() to handle transient HBase visibility delays
2. Have the master heartbeat path re-save ServerInfo (similar to what workers already do) instead of only logging
3. Add a startup delay or readiness check before the task-scheduler begins heartbeating — giving HBase time to make the initial write visible

As a more robust workaround than suppressing logs, you could try adding a small startup delay before the graph is registered with the task scheduler. But the real fix likely needs a code change to make the master heartbeat re-initialize ServerInfo when it's missing, rather than only warning about it.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[vaijosh]

Hi @imbajin,
Would you like me to investigate this issue?
Please feel free to assign any HBase backend related issues to me; I would be glad to assist with them.

[imbajin]

Hi @imbajin, Would you like me to investigate this issue? Please feel free to assign any HBase backend related issues to me; I would be glad to assist with them.

Get it, thanks a lot