refactor(kt-graph,worker-nodes): route auto_build through WorkerGraphEngine#298
Closed
charlie83Gs wants to merge 1 commit intomainfrom
Closed
refactor(kt-graph,worker-nodes): route auto_build through WorkerGraphEngine#298charlie83Gs wants to merge 1 commit intomainfrom
charlie83Gs wants to merge 1 commit intomainfrom
Conversation
…Engine auto_build was reaching into WriteNode/Edge/Dimension repos and Qdrant directly, duplicating key derivation, edge-rekey logic and Qdrant plumbing. Add batch + lifecycle methods on WorkerGraphEngine and have auto_build go through them so the engine remains the single source of write-routing truth. New on WorkerGraphEngine: - bulk_create_nodes, bulk_upsert_nodes_to_qdrant - delete_node, delete_node_qdrant, delete_edge_by_key - absorb_node (composite: dims + edges-rekey + facts + delete-loser) All new methods leave transaction control to the caller (no commit, no savepoint) and keep Qdrant calls non-transactional, matching today's per-seed semantics. auto_build: - _promote_seeds builds NodeCreateSpec list per batch, calls bulk_create_nodes inside per-seed savepoints, then a single batched bulk_upsert_nodes_to_qdrant outside. - _absorb_merged_nodes calls engine.absorb_node + engine.delete_node_qdrant. Tests: new libs/kt-graph/tests/test_worker_engine_bulk.py (17 cases) for the engine surface; test_merge_absorption rewritten to mock the engine seam instead of three repo classes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Merged locally; switching to local-iteration mode (no remote PRs for now). |
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
auto_build_graphwas reaching intoWriteNode/Edge/Dimensionrepos and Qdrant directly, duplicating key derivation, edge-rekey logic and Qdrant plumbing. The rest of the worker-nodes pipeline already goes throughWorkerGraphEngine— auto_build was the lone divergent path and risked silently bypassing future engine side-effects (hooks, counters, telemetry).WorkerGraphEngineso the engine remains the single source of write-routing truth, then routeauto_buildthrough them.New on
WorkerGraphEngine(libs/kt-graph/src/kt_graph/worker_engine.py)bulk_create_nodes(specs)— acceptslist[NodeCreateSpec], upserts each into write-db, populates the in-memory node cache, returnsBulkCreateResultper spec. No commit, no Qdrant.bulk_upsert_nodes_to_qdrant(items)— single batched Qdrant call (chunked at 200 internally byQdrantNodeRepository.upsert_batch).delete_node(node_id)/delete_node_qdrant(node_uuid)— write-db node delete + best-effort Qdrant cleanup.delete_edge_by_key(edge_key)— write-db edge delete with cache eviction.absorb_node(loser_id, winner_id)— composite: dimension transfer, edge re-key, fact merge, loser delete. ReturnsAbsorbResultso caller can do Qdrant cleanup outside any savepoint.All new methods leave transaction control to the caller (no
commit, nobegin_nested) and keep Qdrant calls non-transactional — matches the per-seed semantics auto_build was already relying on.auto_build(services/worker-nodes/src/kt_worker_nodes/workflows/auto_build.py)_promote_seedsbuilds aNodeCreateSpecper active seed, callsbulk_create_nodesinside a per-seed savepoint, marks the seed promoted, then issues one batchedbulk_upsert_nodes_to_qdrantoutside savepoints. Drops direct imports ofWriteNodeRepository,make_node_key,key_to_uuid,QdrantNodeRepository._absorb_merged_nodesresolves loser/winner UUIDs and callsengine.absorb_nodeinside a savepoint, thenengine.delete_node_qdrantoutside. Drops direct imports ofWriteEdgeRepository,WriteDimensionRepository,make_edge_key._check_fact_stale_nodesunchanged.Seed lifecycle (
get_promotable_seeds,mark_seed_promoted,get_merged_promoted_seeds,clear_promoted_node_key,get_seed_fact_ids) stays inWriteSeedRepository— seeds are not graph entities.Tests
libs/kt-graph/tests/test_worker_engine_bulk.py— 17 unit tests covering all 6 new methods, including absorb-node happy path, missing loser/winner, self-edge, no-commit / no-Qdrant invariants.services/worker-nodes/tests/test_merge_absorption.pyrewritten — mocks the engine seam (WorkerGraphEngine.absorb_node,delete_node_qdrant) instead of three repo classes. Engine internals are now covered by the engine's own tests.Test plan
uv run --project libs/kt-graph pytest libs/kt-graph/tests/ --ignore=libs/kt-graph/tests/integration— 43 passeduv run --project services/worker-nodes pytest services/worker-nodes/tests/ --ignore=services/worker-nodes/tests/integration— 122 passedruff check+ruff formatclean on changed files🤖 Generated with Claude Code