Proposal: move CAIPE's default persistence from MongoDB to DocumentDB

[sriaradhyula]

Proposal

Move CAIPE's default persistence provider from MongoDB Community Server to DocumentDB in a future release, while continuing to support MongoDB as an operator choice.

DocumentDB is a MongoDB-compatible document database built on PostgreSQL and licensed under MIT. MongoDB Community Server is distributed under the Server Side Public License. For an OSS-first distribution, MIT is the simpler and more permissive default.

This is a staged change:

1. Draft PR #2418 adds DocumentDB as an opt-in provider across Docker Compose, setup-caipe.sh, and Helm.
2. MongoDB remains the default in that PR. Existing installs are not migrated or switched.
3. After compatibility, migration, TLS, performance, and rollback gates are met, a follow-up change would make DocumentDB the default while preserving MongoDB as an explicit choice.

Why discuss this now?

• CAIPE already talks to persistence through the MongoDB wire protocol and shared MONGODB_URI / MONGODB_DATABASE settings.
• Prior compatibility fixes removed assumptions that do not work across MongoDB-compatible implementations: #886, #1321, and #1924.
• An opt-in release gives operators time to validate real workloads before the default changes.
• We need an explicit community decision because persistence is a platform-wide dependency, not just chat history.

Complete MongoDB dependency inventory

The audit covered application code, integration services, deployment manifests, setup automation, tests, and migration tooling. These are the places that must remain compatible:

Area	MongoDB-backed responsibilities and collections
UI/BFF chat and user state	users, user_settings, user_preferences, conversations, messages, turns, conversation_bookmarks, sharing_access, feedback, in_app_notifications, auth_token_cache; conversation sharing, pin/archive/delete, audit views, statistics, and admin cleanup
Agent and platform configuration	dynamic_agents, mcp_servers, llm_models, platform_config, review_configs, policies, workflow_runs, publication_requests
Skills and catalogs	agent_skills, builtin_skill_scans, skill_hubs, hub_skills, skill_revisions, catalog_api_keys; catalog loading, hub crawling, scan results/overrides, and skill revision history
Credentials and service accounts	service_accounts, credential_secret_refs, credential_encrypted_payloads, oauth_connectors, provider_connections, credential_migration_previews
Teams, identity sync, and ReBAC control-plane data	teams, identity_providers, identity_group_sync_rules, identity_group_sync_runs, external_groups, external_group_team_links, team_membership_sources, idp_sync_settings, idp_sync_runs, rebac_resources, rebac_relationships, policy_rules, policy_change_sets, rebac_enforcement_status, rebac_drift_findings; schema/data migration bookkeeping and self-checks
Dynamic Agents runtime	PyMongo reads, LangGraph MongoDBSaver collections (checkpoints_conversation, checkpoint_writes_conversation), GridFS (agent_files.files, agent_files.chunks), TTL handling, file APIs, conversation deletion, and health checks
Scheduler and autonomous agents	schedules, schedule_one_off_runs, autonomous_tasks, autonomous_runs, trigger_instances, webex_thread_map; dynamic-agent lookups and publishing completed runs into conversations / messages
Slack integration	channel_team_mappings, slack_channel_agent_routes, slack_channel_grants, slack_link_nonces, slack_user_metrics, plus teams, dynamic_agents, service_accounts, conversations/messages, route resolution, auto-assignment, identity linking, admin sync, and feedback
Webex integration	webex_space_team_mappings, webex_space_agent_routes, webex_space_grants, webex_direct_user_routes, webex_link_nonces, webex_user_metrics, plus teams, dynamic_agents, conversations/messages, route resolution, auto-assignment, identity linking, and admin sync
RAG control-plane metadata	rag_ingestion_sources, rag_collections, team_rag_tools, and legacy team_kb_ownership. Vector/document content remains in Milvus/Neo4j; this proposal changes the MongoDB-backed ownership/configuration metadata only
Operational packaging	docker-compose.yaml, docker-compose.dev.yaml, .env.example, setup-caipe.sh, the caipe-ui-mongodb subchart and umbrella values, and the UI, Dynamic Agents, scheduler, autonomous-agents, Slack, and Webex chart Secret/config paths
Validation and lifecycle tooling	Node MongoDB driver, PyMongo, Motor, LangGraph MongoDB checkpointer, GridFS tests, index creation/deduplication, RBAC migrations, backup/restore guidance, setup tests, Compose rendering, and Helm rendering/linting

Mongo Express is MongoDB-specific operational tooling. It remains attached only to the MongoDB profile and is not started with DocumentDB.

What draft PR #2418 adds

• caipe-documentdb Compose profiles for production and development stacks.
• Interactive setup choice, --database=mongodb|documentdb, and DATABASE_PROVIDER support.
• A Helm provider: mongodb|documentdb switch and an opt-in DocumentDB values example.
• Shared URI wiring so UI/BFF, Dynamic Agents, scheduler, autonomous agents, Slack, and Webex select the same provider.
• Provider-specific health checks, persistent volume paths, ports, credentials, and cleanup behavior.
• Documentation that switching providers does not migrate existing data.
• DocumentDB Local pinned to pg17-0.113.0. The upstream 0.114 release warning says active connections can close after two hours and recommends waiting for a fixed release for longer-running workloads.

Compatibility evidence so far

• Docker Compose DocumentDB service reached healthy state.
• Node MongoDB driver and PyMongo connectivity passed.
• CRUD, compound unique indexes, aggregation, and deletion passed.
• A 17 MiB GridFS put/get/delete round trip passed, covering payloads above MongoDB's single-document limit.
• LangGraph MongoDBSaver checkpoint persistence passed.
• A validation migration restored 978 documents; collection counts and index definitions matched the source, which was retained for rollback.
• Both providers render successfully through the subchart and umbrella chart; both production and development Compose files validate for either profile.

This evidence supports opt-in use. It is not yet sufficient to claim complete MongoDB behavioral parity or to change the default immediately.

Known constraints

• MongoDB API compatibility is broad but not identical. New queries, aggregation operators, index options, or driver behavior need a compatibility test before adoption.
• retryWrites=false and direct connection settings are required for the bundled DocumentDB path.
• The bundled local image generates a self-signed TLS certificate. tlsAllowInvalidCertificates=true is acceptable only for local/in-cluster evaluation; production needs a trusted CA and strict verification.
• No automatic data migration is included. Operators must back up, restore, compare counts/indexes, validate application behavior, and retain a rollback source.
• The current pg17-0.113.0 pin must stay until an upstream release fixes the 0.114 long-lived-connection regression and passes CAIPE's matrix.

Gates before changing the default

• Add a CI matrix that exercises MongoDB and DocumentDB across UI/BFF, Dynamic Agents, scheduler, autonomous agents, Slack, Webex, skills, RBAC, RAG metadata, LangGraph, and GridFS.
• Validate every required index type and aggregation used by the inventory above, including TTL, sparse/partial, compound unique, $lookup, $group, and deduplication paths.
• Publish a supported-version matrix and upgrade policy.
• Provide documented, repeatable migration and rollback tooling with count/index verification and an upgrade guide.
• Provide production TLS configuration using a trusted CA.
• Run performance, connection-lifetime, restart/recovery, backup/restore, and load tests.
• Keep DocumentDB opt-in for at least one release and collect operator feedback.
• Confirm that MongoDB remains a supported explicit provider after the default flips.

Feedback requested

• Are there MongoDB features or deployment patterns in your CAIPE installation that are missing from this inventory?
• Should MongoDB remain a supported option indefinitely after DocumentDB becomes the default?
• What migration/rollback guarantees would you require before adopting DocumentDB?
• Who can test the draft PR against a production-like workload, especially long-running connections, GridFS, schedules, identity sync, or Slack/Webex routing?