[Improvement] Re-register HStore schema cache clear listener after a Meta reconnect

[dpol1]

• Follow-up to fix(server): sync hstore schema cache clears #3011.

That PR made CachedSchemaTransactionV2 listen for schema-cache-clear meta
events and drop the matching local V2 caches, so a schema change on one server
no longer leaves other nodes serving stale schema in HStore / multi-server mode.

The catch is the listener registers exactly once per JVM. The gRPC watch behind
it is process-wide and there is no unlisten. If the Meta transport reconnects
and the old watch gets dropped, the listener goes deaf and nothing brings it
back. There is already a manual recovery hook,
resetMetaListenerForReconnect(), but nothing calls it, because Meta exposes no
reconnect signal to hang it on.

The two halves of the gap, both in CachedSchemaTransactionV2:

• The listener-lifetime comment explaining why recovery is not automatic today:
  https://github.com/apache/hugegraph/blob/master/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java#L52-L58
• The resetMetaListenerForReconnect() hook that has no caller and the javadoc
  saying it must be wired to a Meta reconnect callback:
  https://github.com/apache/hugegraph/blob/master/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java#L254-L270

What should happen

After the Meta transport reconnects, the JVM-wide schema-cache-clear listener
comes back on its own and keeps delivering events for every graph. No operator
action.

What happens now

MetaManager / MetaDriver expose listen and keepAlive but no reconnect
callback. A dropped watch goes unnoticed, events stop arriving, and the node can
keep serving stale schema until someone calls resetMetaListenerForReconnect()
by hand. Nothing does.

Proposed fix

Two parts, same theme, easiest to do in one go:

1. Give MetaManager / MetaDriver a reconnect callback (something like
   listenReconnect / onTransportReconnect) that fires when the transport
   reconnects and the previous watch is gone.
2. Point CachedSchemaTransactionV2 at that callback so it re-registers the
   listener. resetMetaListenerForReconnect() stops being a manual entry point
   and becomes the callback target.

How to verify

Two servers against HStore/Meta. Force a Meta reconnect (restart Meta or kill
the connection), then change a schema on server A and assert server B clears its
V2 caches and stops returning stale schema.

[dpol1]

@dosu — EtcdMetaDriver.listen()/listenPrefix() register the jetcd watch with the Consumer<WatchResponse> overload only — no onError/onCompleted. Does that watch stream auto-reestablish after a gRPC drop / Meta reconnect, or die silently? Same for PdMetaDriver. If it self-heals the issue is moot; if it dies silently, that teardown is where a reconnect callback must hook.

[dosubot]

Short answer: EtcdMetaDriver watches can die silently; PdMetaDriver watches self-heal.

---

EtcdMetaDriver.listen() / listenPrefix() pass only a Consumer<WatchResponse> to jetcd's Watch client — no onError, no onCompleted [1]. Whether the watch auto-reestablishes after a gRPC drop depends entirely on jetcd's internal retry behaviour, which is not validated or documented anywhere in the HugeGraph codebase. There is no application-level reconnection, no leader-change detection, and no health monitoring. If jetcd silently gives up (which it can, depending on version and channel config), events stop arriving with zero indication.

PdMetaDriver is a different story. Its underlying KvClient registers a full StreamObserver<WatchResponse> with explicit callbacks [2]:

• onError → resets the client ID and re-invokes listen()/listenPrefix() via a BiConsumer wrapper (fixed 1 s backoff) [3].
• onNext with Leader_Changed state → same reconnect path.
• onCompleted → no-op, but defined.

So PD watches do self-heal after transport failures and leader elections.

---

Implication for issue #3036: The issue is not moot. For HStore/etcd mode, a gRPC drop will silently kill the schema-cache-clear watch registered in CachedSchemaTransactionV2, and nothing in EtcdMetaDriver brings it back. That teardown point — jetcd's implicit onError / channel-close path — is exactly where a reconnect callback must hook. The fix has two angles:

1. Immediate (EtcdMetaDriver): Switch from the Consumer-only overload to jetcd's Watch.Listener (or equivalent) that exposes onError/onCompleted, then re-register the watch on error — mirroring what KvClient already does for PD [2].
2. API-level (MetaDriver interface): Surface a listenReconnect / onTransportReconnect callback so CachedSchemaTransactionV2.resetMetaListenerForReconnect() can be wired as a reconnect target without depending on driver internals.

As of today there are no PRs addressing this in EtcdMetaDriver [4], and no tests covering watch failure/recovery for either driver.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.