Skip to content

Migrate storage layer from Snowflake/DynamoDB/SQS to Postgres/NATS (graph backend fixes)#159

Open
jonathanhaaswriter wants to merge 2 commits intomainfrom
fix/pr-158-graphdb-backends-main
Open

Migrate storage layer from Snowflake/DynamoDB/SQS to Postgres/NATS (graph backend fixes)#159
jonathanhaaswriter wants to merge 2 commits intomainfrom
fix/pr-158-graphdb-backends-main

Conversation

@jonathanhaaswriter
Copy link
Copy Markdown
Collaborator

Summary

  • start from feat/postgres-nats-migration and keep Postgres scoped to storage and warehouse responsibilities instead of treating it as a graph backend
  • move read-only SQL validation into backend-agnostic warehouse helpers and update API, agent, and CLI call sites accordingly
  • wire in the pgx stdlib driver plus vendored dependencies required for Postgres runtime support, and refresh generated config env docs

Testing

  • CEREBRO_DEVEX_BASE_REF=writer/main python3 ./scripts/devex.py run --mode changed --base-ref writer/main

Test added 2 commits March 25, 2026 13:23
Rebased onto current main, resolved merge conflicts from upstream
service seam extractions. Fixed import cycle (warehouse -> snowflake),
updated sync service and test stubs to use warehouse interfaces,
and adapted remaining snowflake references to postgres types.
@jonathanhaaswriter
Copy link
Copy Markdown
Collaborator Author

I found two blocking issues in the current head:

  • internal/app/app_init_core.go:initPostgres still couples DATABASE_URL to a.Warehouse, so storage DB bootstrap overrides WAREHOUSE_BACKEND and prevents the configured warehouse from being initialized. initWarehouse also still lacks a snowflake branch.
  • internal/jobs/worker.go switched retry backoff back to ExtendVisibility(delay), but internal/jobs/queue_nats.go:ExtendVisibility ignores the delay argument and only calls Msg.InProgress(). Retryable failures therefore won't honor exponential backoff under JetStream.

I think both need to be resolved before this is ready.

@jonathanhaaswriter
Copy link
Copy Markdown
Collaborator Author

I think we still have two problems here:

  • effectivePostgresURL() prefers JOB_DATABASE_URL over DATABASE_URL, so if both are set the app repos silently connect to the jobs DB instead of the normal app DB
  • the local findings fallback changed from SQLite to in-memory, which drops the persisted local-mode behavior we had before

Can we keep app storage on DATABASE_URL and preserve the SQLite fallback when Postgres isn't configured?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant