CBG-5226: Opt-in use of _system._mobile for databases and bootstrap

[bbrks]

CBG-5226

• Adds support for the dual-collection MetadataStore wrapper on DatabaseContext in SG based on opt-in configuration - as well as similar fallback read logic in the bootstrap code.
• Adds support for _system._mobile storage for all bootstrap metadata (registry, dbconfigs) for:
  • New deployments that have opted in at the bootstrap config level (Capella), or have their first database opting in to use the system metadata collection.
  • Existing deployments that have had all databases in the bucket fully migrated with the opt-in.
• Uses a migration status tracking doc _sync:metadata_migration_status in _system._mobile to track per-bucket and per-database migration state. Used to bypass the dual metadata store wrapper post-migration, and also used to determine the last migrated database for bootstrap/registry migration.
• Does not perform DatabaseContext migration - Upcoming PR as CBG-5228 - though all writes get routed to the new location after opting in, and reads do fall back to the fallback datastore even without the migration run.

Diagrams

Logic to determine which collection to use for bootstrap (not gated behind any opt-in - we only have the opt-in for databases). We optimistically write to _system if we find this is a new deployment. We write to _default if there is an existing deployment, but have a migration job to move bootstrap-level stuff over once the last database has been fully migrated.

  flowchart TD
      A[First bootstrap-doc op on bucket] --> C[Read _sync:registry]
      C --> D{Where is it?}
      D -- _system._mobile --> E[Target: _system._mobile]
      D -- _default._default --> F[Target: _default._default]
      D -- not found --> G{Cluster flag<br/>OR per-DB opt-in?}
      G -- yes --> E
      G -- no --> F

Loading

Logic to determine whether db is new and can bypass metadata migration and write directly to _system, or if we need to run migrate.

  flowchart TD
      A[Create or update db] --> B{resolveUseSystemMetadataCollection<br/>per-DB wins over cluster flag}
      B -- no --> J[MetadataStore = _default._default<br/>no wrapper]
      B -- yes --> C[Wrap MetadataStore<br/>base.NewMetadataStore]
      C --> D[probeLegacyPerDBMetadata]
      D --> E{_sync:seq<br/>or _sync:m_id:seq<br/>exists in _default._default?}
      E -- found --> F[Arm migration:<br/>not_started entry in<br/>migration_status]
      E -- none --> G[SetMigrationComplete<br/>on wrapper immediately]
      G --> H[shouldRunMetadataMigration<br/>short-circuits — no manager arm]
      F --> I[MetadataMigrationManager<br/>processes entry]

Loading

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/714/
  • TestChangeIndexPartitionsStartStopAndRestart flake
  • TestMetadataMigrationStartsAfterAllNodesApplyConfig Fixed w/ last commit

[Copilot]

Pull request overview

Implements CBG-5226 by adding an opt-in path for storing Sync Gateway bootstrap metadata and per-database metadata in _system._mobile, with read-fallback to legacy _default._default during the migration window. This wires migration status tracking via a bucket-level _sync:metadata_migration_status document and adds rosmar + Couchbase bootstrap-connection support for dual-collection reads/writes.

Changes:

• Add per-DB base.MetadataStore wrapping to target _system._mobile when opted in, including a “new DB” fast path and per-bucket bootstrap-migration completion trigger wiring.
• Extend base.BootstrapConnection (CouchbaseCluster + RosmarCluster) to support dual-collection bootstrap metadata operations, migration status doc CRUD, and bucket-target caching.
• Add/adjust tests to exercise dual-collection bootstrap semantics and metadata migration gating behavior.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rest/server_context.go	Routes DB metadata to _system._mobile via MetadataStore, adds migration-status stamping/refresh logic, and adds bucket-level bootstrap migration completion flow.
rest/metadatamigrationtest/metadata_migration_test.go	Adjusts test setup to seed legacy metadata so migration gating behavior can be exercised.
rest/main.go	Plumbs startup flag into bootstrap connection creation to enable bootstrap dual-collection behavior.
rest/admin_api.go	Adds per-bucket bootstrap-target hinting prior to initial registry/dbconfig writes for new deployments.
db/database.go	Wires new migration status hooks into DatabaseContext and refines metadata migration arming behavior.
db/background_mgr_metadata_migration.go	Implements status-doc updates for per-DB migration lifecycle (stubbed copy step for follow-up PR).
base/bootstrap.go	Extends BootstrapConnection interface and CouchbaseCluster implementation for dual-collection bootstrap metadata + migration status handling.
base/rosmar_cluster.go	Implements the new dual-collection bootstrap behavior and migration-status APIs for rosmar.
base/metadata_migration_status.go	Introduces the bucket-level metadata migration status document model and helpers.
base/bootstrap_test.go	Adds dual-collection bootstrap tests (insert/write/touch/delete fallback semantics) and updates cluster constructors.
go.mod	Bumps rosmar dependency to a newer pseudo-version.
go.sum	Updates checksums for the rosmar bump.

[gregns1]

Happy for these to be done in follow up PR/ticket so approving but everything else looks good to me.