feat(graphs): pick a database when creating a graph

[charlie83Gs]

Summary

Drops the confusing "schema vs database" storage-mode toggle in the graph creation UI and replaces it with a single Database picker. The first option is always default (the system DB); other options come from a new GET /api/v1/graphs/database-connections endpoint that lists rows from the database_connections table.

Conceptually: every non-default graph is a schema in some database. The only choice the user makes is which database. Multiple graphs in the same DB are schema-isolated by convention.

Closes the gap reported on prod where the new graph UI had no database options to pick from.

What changes for users

Before — confusing dropdown:

Storage Mode: [ Schema (same DB, separate schema) | Database (different DB) ]

…and even when "Database" was selected, there was no way to pick which DB.

After — clear single dropdown:

Database: [ default | shared | ...other provisioned DBs ]

Backend

• New GET /api/v1/graphs/database-connections (admin-only). Synthetic default entry is always first; rows from database_connections follow in name order.
• Drop storage_mode from CreateGraphRequest. Server derives it from database_connection_config_key (None or "default" → schema mode in system DB; otherwise → database mode in the referenced external DB).
• Fix _provision_graph to honor database_connection_id: resolves the graph/write/qdrant URLs from settings.graph_databases[config_key], creates schemas via one-off engines pointed at the correct DBs, runs alembic with DATABASE_URL / WRITE_DATABASE_URL env overrides (Pydantic Settings re-reads them in the subprocess), and provisions Qdrant collections against the per-graph URL when it differs.
• New make_qdrant_client() factory in kt-qdrant for non-singleton clients pointed at arbitrary URLs.

Frontend

• New DatabaseConnectionResponse type and listDatabaseConnections() API helper.
• frontend/src/app/graphs/page.tsx: Storage Mode <select> replaced with Database <select> populated from the new endpoint; default selection is "default".
• frontend/src/app/graphs/[slug]/page.tsx: "Storage" field renamed to "Database", showing the connection name (looked up by database_connection_id), falling back to default / external.

Tests

• services/api/tests/test_graph_schemas.py — drop the storage_mode-specific cases; add a test_default_connection_key_accepted for the magic string.
• New services/api/tests/test_database_connections_endpoint.py — covers synthetic-default ordering, name-ordered rows, and string-UUID id field. All 3 tests pass locally against real Postgres.
• kt-config (26), kt-qdrant (64), services/api graph schemas (15), and frontend (123) tests all green.

Out of scope (follow-ups)

• Dropping the storage_mode column entirely (would need an alembic migration; column kept for now)
• Per-graph runtime use of qdrant_url (the resolver in graph_sessions.py already plumbs it through; no consumer reads it yet)

Test plan

• Merge & wait for CI to publish the new image
• Reconcile flux on prod
• Hit GET /api/v1/graphs/database-connections as admin → expect [{default}, {shared}]
• In the UI: open Graphs → Create Graph → confirm dropdown shows default and Shared
• Create a graph with slug prov_test against Shared
• Verify schema lands in knowledge-tree-shared-graph-db (NOT in the system graph-db)

  kubectl exec -n knowledge-tree knowledge-tree-shared-graph-db-1 -- \
    psql -U postgres -d knowledge_tree_shared -c "\dn" | grep prov_test

• Verify Qdrant collections exist on knowledge-tree-shared-qdrant
• Verify the graph card shows DB: Shared instead of Separate DB / Shared DB

Risks

• _provision_graph external-DB code path is exercised in prod for the first time. If alembic migrations fail against a fresh external DB it likely means search_path isn't being honored — fixable, may need a follow-up.
• DDL through PgBouncer transaction mode for the write-db schema creation should work for a single CREATE SCHEMA. If it fails, plan B is to point provisioning at the underlying CNPG -rw service directly.

🤖 Generated with Claude Code

[charlie83Gs]

Pushed a fresh take addressing all review feedback. Force-pushed because the previous branch was diverged from main and the rebase had multiple conflicts — easier to start clean from the latest main.

Changes vs the original PR

• Rebased onto main — discovered that v0.36.0/v0.37.0 already landed: _discover_extra_databases, qdrant_url on GraphDatabaseConfig, the database picker UI scaffolding, and database_connection_name on GraphResponse. So most of the original diff was duplicating upstream work.
• Conceptual simplification per follow-up feedback: drop the "schema vs database" toggle entirely. Schema is now the only isolation strategy. The user just picks a database — to get DB-level isolation, just don't put another graph in the same connection.
• graph_sessions.py: collapsed the storage_mode == "database" branch in _build_and_cache. Non-default graphs now route by database_connection_id IS NULL only.

Review feedback addressed

1. ✅ Reserved default config_key — GraphRepository.create_database_connection now raises ValueError if you try to insert a row with config_key="default". Covered by test_create_database_connection_rejects_default_key.
2. ✅ asyncpg URL normalization — added a field_validator on GraphDatabaseConfig that auto-rewrites postgresql:// → postgresql+asyncpg://. Covered by libs/kt-config/tests/test_graph_databases.py::TestGraphDatabaseConfigValidator.
3. ✅ storage_mode grep — confirmed nothing else branches on it at runtime: graph_sessions.py:200 was the only consumer and it's been refactored away. The column stays for backward-compat with old rows but is no longer load-bearing.
4. ✅ Admin auth coverage — services/api/tests/integration/test_database_connections_endpoint.py::test_blocked_for_non_admin uses httpx.AsyncClient + dependency_overrides[require_auth] to assert non-superusers get 403 from the route.
5. ✅ make_qdrant_client cosmetic — only calls get_settings() when timeout is None.
6. (skipped per request — no CI smoke test for the external-DB provisioning path)

Nits

• ✅ from kt_config.settings import get_settings moved to module top in graphs.py.
• 🟡 Did not extract a dbLabel(g) helper in the frontend — the badge is now a one-line nullish coalesce: g.database_connection_name ?? "default". No ternary chain to extract.

Test plan

• kt-config (34), kt-qdrant (64), kt-db graph models (8), api graph schemas (15), api permissions (14), api database-connections endpoint (4) — all green locally.
• Frontend lint clean; 123 frontend tests passing.
• Wait for CI build, deploy, then exercise create-graph-against-shared end-to-end on prod.

[charlie83Gs]

Round 2 fixes pushed (e07b6ca):

Lint failure

Fixed — moved DatabaseConnection and delete imports out of the cleanup_connections fixture body to module-level so ruff I001 passes. Confirmed locally with uv run --frozen ruff check ..

Review feedback addressed

1. ✅ application_name on provisioning engines — both one-off engines now pass connect_args={"server_settings": {"application_name": f"kt-provision-{graph.slug}"}}. They show up in pg_stat_activity tagged.
2. ✅ NullPool on one-off engines — added poolclass=NullPool so we don't keep an idle pool around for a single CREATE SCHEMA.
3. (acknowledged — DatabaseConnectionResponse duplication is pre-existing)
4. ✅ Follow-up issue opened: Drop graphs.storage_mode column (alembic migration) #172 to drop the storage_mode column with a proper alembic migration.
5. (FYI — test path is correct)
6. ✅ Disable submit on dbError — <Button disabled={creating || dbError !== null}> so a failed db-connections fetch can't silently fall through to the system DB.
7. (verified) — checked libs/kt-db/alembic/env.py: it honors ALEMBIC_SCHEMA by setting version_table_schema and running SET search_path TO {schema}, public in do_run_migrations. Combined with the DATABASE_URL / WRITE_DATABASE_URL env override in the subprocess, the alembic-against-external-DB path should work end-to-end. Will still verify with the manual smoke test in the test plan after merge.

Test status

• ruff check . clean
• test_graph_schemas.py (15) ✓
• test_database_connections_endpoint.py (4) ✓
• test_permissions.py (14) ✓
• Frontend lint clean, 123 frontend tests ✓

[charlie83Gs]

Round 3 fixes pushed (7b21f76):

Items addressed

#1 — Orphan-schema risk (defended + documented)
After re-reading the actual control flow, I think the original concern was based on a misread of the order. The Graph row commits at graphs.py:313 before _provision_graph runs at graphs.py:317, so there's no "orphan row" possibility. There IS a possible "orphan schema in the external DB after a partial provisioning failure" scenario, but every step is idempotent (CREATE SCHEMA IF NOT EXISTS, alembic upgrade head, ensure_collection), so a failure marks the row "error" and POST /api/v1/graphs/{slug}/retry-provision is the safe recovery path. I added a "Failure recovery" section to the _provision_graph docstring explaining this so future readers don't have to re-derive it. Open to adding cleanup-on-failure if you still think it's worth the complexity, but the cleanup itself can fail and would muddy the error path.

#3 — Fragile qdrant_url comparison ✅
Now strips trailing slashes before comparing: qdrant_url.rstrip("/") != settings.qdrant_url.rstrip("/"). So http://h:6333 and http://h:6333/ no longer needlessly spawn a fresh client.

#4 — "default" magic string ✅

• Extracted DEFAULT_DB_CONFIG_KEY = "default" in libs/kt-config/src/kt_config/settings.py with a docstring explaining the contract.
• All four sites now import and use it: API handler (graphs.py:288), synthetic list entry (graphs.py:355), repo guard (repositories/graphs.py).
• Added a startup check in kt_api.main._assert_default_db_key_unreserved() that logs an error at API startup if any real database_connections row holds the reserved key. Catches anything that may have slipped in via raw SQL or a previous version that lacked the repo guard.

#6 — Test coverage gap ✅
Two new test files:

• services/api/tests/test_provision_graph_routing.py — 3 unit tests that mock create_async_engine, subprocess.run, and the Qdrant client factories to assert _provision_graph routes to:
  • the external DB URLs when database_connection_id is set,
  • the system DB URLs when it's NULL,
  • raises a clear RuntimeError if the config_key is missing from Settings.graph_databases.
• test_database_connections_endpoint.py::test_create_graph_with_default_key_uses_system_db — POSTs a graph with database_connection_config_key="default" and asserts the resulting row has database_connection_id=None. Added a stub_users_in_db fixture to insert the test users into the User table so the graphs.created_by FK doesn't trip.

#7 — Frontend storage_mode references ✅
grep -r "storage_mode" frontend/ returns one hit: GraphResponse.storage_mode in types/index.ts. That's the read-side type (kept for backward compat with old graph rows). Nothing references CreateGraphRequest.storage_mode.

Skipped per reviewer note

• fix: resolve cross-worker test imports and enforce unit/integration test boundaries #2 Alembic env override — reviewer self-resolved.
• fix: add pytest.skip guards for tests requiring external API keys #5 _admin rename — cosmetic, no action needed.

Test results

• ruff check . clean (auto-fixed I001 import-order issues from the new code)
• 94 api tests pass, including the 3 new _provision_graph routing tests and the new test_create_graph_with_default_key_uses_system_db
• 34 kt-config tests pass
• 8 kt-db graph models tests pass

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}