Skip to content

fix(server): wait for GRAPH_CREATE event when creating graph on PD path - #3138

Draft
bitflicker64 wants to merge 2 commits into
apache:masterfrom
bitflicker64:fix/graph-create-wait-event
Draft

fix(server): wait for GRAPH_CREATE event when creating graph on PD path#3138
bitflicker64 wants to merge 2 commits into
apache:masterfrom
bitflicker64:fix/graph-create-wait-event

Conversation

@bitflicker64

Copy link
Copy Markdown
Contributor

Purpose of the PR

In distributed mode (PD + HStore), POST /graphspaces/{space}/graphs/{name} returns 200 before the creating server has actually bound the new graph into its embedded Gremlin server. GraphManager.createGraph (PD path) fires GRAPH_CREATE via this.eventHub.notify(...), which is asynchronous, so the REST response can be written before ContextGremlinServer injects the graph and its TraversalSource into the Gremlin global bindings. An immediate follow-up Gremlin/Cypher request to the same server then fails with HTTP 400: Could not rebind [g] to [__g_<name>] as [__g_<name>] could not be found in the Graph or TraversalSource global bindings.

The local path already handles this correctly: createGraphLocal calls this.notifyAndWaitEvent(Events.GRAPH_CREATE, graph), which blocks on the event future before returning. The PD path simply lost that parity.

Main Changes

One question for reviewers, as raised in #3137: notifyAndWaitEvent (GraphManager.java, around line 1774) waits on the event future but swallows listener failures with only a LOG.warn, so a failed bindings injection would still return 200. Should a listener failure fail the create instead? I left that behavior unchanged here to keep this PR a pure parity fix with createGraphLocal.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:

Does this PR potentially affect the following parts?

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

The PD-backed createGraph fired GRAPH_CREATE without awaiting it, so the
REST 200 could be written before ContextGremlinServer injected the graph
into the Gremlin global bindings, and an immediate Gremlin/Cypher request
to the creating server could fail with "Could not rebind [g]".
createGraphLocal already waits via notifyAndWaitEvent; this applies the
same call on the PD path.
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Aug 3, 2026
@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 28.94737% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.42%. Comparing base (1716c77) to head (9a4ac02).

Files with missing lines Patch % Lines
...n/java/org/apache/hugegraph/core/GraphManager.java 28.94% 26 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (1716c77) and HEAD (9a4ac02). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (1716c77) HEAD (9a4ac02)
3 1
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3138      +/-   ##
============================================
- Coverage     39.18%   32.42%   -6.76%     
  Complexity      264      264              
============================================
  Files           770      770              
  Lines         65779    65811      +32     
  Branches       8726     8728       +2     
============================================
- Hits          25774    21341    -4433     
- Misses        37244    42023    +4779     
+ Partials       2761     2447     -314     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The PD create path now waits for GRAPH_CREATE listeners, but listener failures remain non-fatal and the current head has a failing codecov/project check. Evidence: EventHub.notify() catches listener Throwable at hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/event/EventHub.java:200-205; final gate reports codecov/project FAILURE.


// Let gremlin server and rest server context add graph
this.eventHub.notify(Events.GRAPH_CREATE, graph);
this.notifyAndWaitEvent(Events.GRAPH_CREATE, graph);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ This wait only guarantees that the listener task completed, not that graph registration succeeded. EventHub.notify() catches every listener Throwable at hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/event/EventHub.java:200-205, while notifyAndWaitEvent() catches future.get() failures at GraphManager.java:1777-1780. A failing ContextGremlinServer.injectGraph() can therefore leave the new graph absent from the Gremlin graph/traversal/global bindings while this PD create still returns success. Please propagate GRAPH_CREATE registration failures or verify the required bindings before responding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, fixed in 9a4ac02. EventHub.notify resolves with the count of listeners that returned normally, so notifyAndWaitEvent now compares that against the registered listener count and fails the create when one is missing.

Rolling back addGraphConfig / notifyGraphAdd would let this replica's binding failure delete a graph the others bound fine, so the graph is instead bound locally before it is published to meta; the failure path unregisters and closes it, like a failed backend init.

The count is an inference, not the real exception. Happy to switch to verifying the bindings directly, or to having EventHub surface listener failures, if you prefer either.


// Let gremlin server and rest server context add graph
this.eventHub.notify(Events.GRAPH_CREATE, graph);
this.notifyAndWaitEvent(Events.GRAPH_CREATE, graph);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Making this path synchronous also introduces an unbounded request wait: notifyAndWaitEvent() calls future.get() without a timeout at GraphManager.java:1777, and its catch(Throwable) swallows InterruptedException without restoring the interrupt status. A hung listener or shutdown can now hold the create request indefinitely, or return before registration after interruption. Please use a bounded wait and handle InterruptedException explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 9a4ac02: bounded future.get(30s), and InterruptedException is caught on its own and restores the interrupt status before failing. The drop path gets the same bounded wait but stays lenient, since the data is already gone when the event fires and TinkerPop's removeGraph throws for a graph the Gremlin server never bound.

30s is a constant rather than an option, as ServerOptions has nothing comparable; happy to promote it if you want it tunable.

One call I would rather leave to you: a timeout currently fails the create, so a merely slow listener breaks a create that actually worked. Treating a timeout as unknown (log loudly, do not fail) is the alternative.

Waiting for the GRAPH_CREATE future was not enough to prove that the graph
was actually registered: EventHub swallows every throwable raised by a
listener and resolves the future with the number of listeners that returned
normally, so a listener that blew up looked exactly like a successful one.
The create now compares the notified count with the registered listener
count and fails when a listener did not complete.

The wait is also bounded now instead of blocking forever, and an
InterruptedException restores the thread's interrupt status before the
failure is reported.

Ordering is fixed along with it. On the PD path the graph is bound in the
local gremlin/rest server context before its config is written to meta and
broadcast, so a failed binding cannot leave a graph behind in meta for the
other servers to converge on. A binding failure now unregisters the graph
locally and closes it, the same cleanup a failed backend init already does,
rather than dropping data that other servers may have bound successfully.
On the local path the notify moved inside the existing try, which now also
unregisters the graph before dropping it, so a failed binding leaves no
closed graph behind in the context.

The drop path keeps the lenient behaviour: the data is already gone when
the event fires, so failing the request cannot undo anything and the
listener state may legitimately be absent already.
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Aug 4, 2026
@bitflicker64

Copy link
Copy Markdown
Contributor Author

Both points addressed in 9a4ac02, details in the inline replies.

Two things left out on purpose:

There is no automated coverage for the new failure semantics: nothing exercises notifyAndWaitEvent today, and the race itself needs a multi-server deployment. A unit test registering a deliberately throwing listener on a scratch EventHub would cover the count check and the cleanup cheaply, if you want it here.

codecov/project looks like a base-report artifact: it reports +11 files and +1150 lines against base, which this diff cannot produce, while codecov/patch passes.

@bitflicker64
bitflicker64 marked this pull request as draft August 4, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants