Repository navigation
v3.0.0
3.0.0 — 2026-05-12 — Migration Squashing (all 5 providers)
This release ships the destructive-model migration squash feature plus
the universal script-format resource form and the CLI extensibility
contract that lets each provider plug into hyperbee-migrations squash.
v3.0 is a major release because of two breaking changes around
IMigrationRecordStore and provider record-store schemas — both protected by
safe back-compat paths so existing v2 migrations and consumers keep working
unchanged.
The full release-readiness audit is closed: 5 release-blockers + 17
Redesigns + the F-tier deferred items are resolved.
Library tier ships at 1335 unit tests per target framework (.NET 8, 9, 10);
the five Squash provider packages
(Hyperbee.Migrations.Providers.{Provider}.Squash) ship as separate
NuGet packages so production deployments do not pay the Testcontainers /
Docker runtime cost.
Highlights
- Squash migrations — declare a migration that subsumes a contiguous range
of prior versions via[Migration(version, Replaces = new[] { ... })]or
[Migration(version, ReplacesRange = "1000-1500")]. The runner reconciles
per environment: mature environments auto-mark the squash without running
its body; fresh environments run the squash as a baseline. - Squash codegen for all five providers — every provider ships its
own snapshot strategy in v1: PostgresPgDumpSnapshotStrategy(canonical
.sqlfrompg_dump --schema-only), AerospikeInfoSnapshotStrategy,
OpenSearchRestStateDiffStrategy, MongoDBIntrospectionSnapshotStrategy,
CouchbaseHybridStrategy. Shipping all five together is deliberate:
the strategy abstraction is only proven correct by being implemented
against the full provider matrix. Outcome: the 5-interface contract
held for all five providers without modification. - Universal
.pqlscript form — the recommended shape for resource
migrations. All four NoSQL providers accept multi-statement.pql
(Provider Query Language) files with--/////* */comments and
;terminators; Postgres accepts its native.sqland.pql. The
legacy.statements.jsonJSON-array form continues to apply unchanged
(backward-compatible, not recommended for new work). - Reversible migrations via
.down.pql(OpenSearch) — pair a
<name>.pqlUp script with a sibling<name>.down.pqlDown script;
the down script is dispatched in written order (author-owned teardown),
preserving the R-19 partial-rollback ledger semantics. The legacy
per-entryrollbackfield in.statements.json(auto-reverse) remains
supported. Missing.down.pql⇒ loudRollbackNotSupportedException
before any mutation. Squashes are up-only, so generated squashes carry
no down script. - Fleet readiness gate — the squash CLI refuses generation while any
registered fleet member is mid-range (MidRangeFleetException), and the
runner refuses a mid-range environment loudly at apply time
(MidRangeSquashException, with arecover from-mid-rangeescape hatch).
A deploy-time fleet-staleness gate was specified during design but cut
as redundant before ship (the apply-time refusal already makes the
dangerous case loud and recoverable). - Recovery acknowledgement token for the
MidRangeSquashExceptionescape hatch — deterministic per
(env, squash, missing-versions)so retries reproduce the same token,
but accidental copy-paste from a sibling environment is rejected.
Added
[Migration(version, Replaces = ..., ReplacesRange = ...)]— squash declaration.MigrationRecordKindenum (Migration/Squash/Baseline).MigrationRecord.Checksum+Kind+Replaces(long[]).MigrationLedgerIntegrityException— refuses inconsistentKind/Replacesrows.MidRangeSquashException— partial-coverage refusal with three documented recovery paths.MigrationApplyModeenum (Fresh/PartialCatchUp) plusMigrationContextambient context withIsFreshInstallback-compat sugar.IChecksumStrategy+DefaultChecksumStrategy— deterministic SHA-256 over(typeof.FullName, Version).WritePrecondition(None/MustNotExist) +WriteOutcome
(Created/AlreadyExistsBenign/PreconditionFailed).IMigrationRecordStoregains
WriteAsync(MigrationRecord, WritePrecondition, CancellationToken) → WriteOutcome,
IntersectWithAppliedAsync, andIntersectWithSquashedAsync— all with safe
default-interface-method implementations so v2 record stores compile and run
unchanged.Hyperbee.Migrations.Squashnamespace with the strategy contract
(ISquashStrategy,SquashGenerationResult,ITopologySignature,
IDataOpClassifier,ISnapshotCanonicalizer,ISquashVerifier,
SquashStrategyDescriptor).Hyperbee.Migrations.Resourcesnamespace withResourceFormat,
ResourceFormatDetector, andScriptStatementSplitterfor the universal
script form.- Postgres squash components (
PostgresTopologySignature,
PostgresStatementClassifier+PostgresStatementSplitter,
PostgresSnapshotCanonicalizer,PostgresDataOpClassifier,
PgDumpSnapshotStrategy,PostgresSquashVerifier,
PostgresMigrationSourceScanner). - Aerospike squash components (
AerospikeTopologySignature,
AerospikeStatementClassifier+AerospikeStatementKind,
AerospikeSnapshotCanonicalizer,AerospikeDataOpClassifier,
InfoSnapshotStrategy+AerospikeSquashGenerationContext+
AerospikeSnapshotCapture,AerospikeSquashVerifier,
AerospikeMigrationSourceScanner). - OpenSearch squash components (
OpenSearchTopologySignature,
OpenSearchStatementClassifier,
OpenSearchSnapshotCanonicalizer-- JSON-section canonical form with
opaque painless preservation,
OpenSearchDataOpClassifier,
RestStateDiffStrategy+OpenSearchSquashGenerationContext+
OpenSearchSnapshotCapture,OpenSearchSquashVerifier,
OpenSearchMigrationSourceScanner). - MongoDB squash components (
MongoDBTopologySignature,
MongoDBStatementClassifier+MongoDBStatementKind,
MongoDBSnapshotCanonicalizer-- JSON-section canonical form with
ephemeral strip catalog (uuid/readOnly/v/ns),
MongoDBDataOpClassifier,
IntrospectionSnapshotStrategy+MongoDBSquashGenerationContext+
MongoDBSnapshotCapture,MongoDBSquashVerifier,
MongoDBMigrationSourceScanner). - Couchbase squash components (
CouchbaseTopologySignature,
CouchbaseStatementClassifier+CouchbaseStatementKind,
CouchbaseSnapshotCanonicalizer-- JSON-section canonical form with
deferred-build GSI state preservation (R-P3 OQ resolution:state=online
dropped,state=deferredpreserved, transient states throw at squash-time),
CouchbaseDataOpClassifier-- parameterized N1QLQueryAsync/
AnalyticsQueryAsyncdefault-deny (R-P3 OQ resolution),
HybridStrategy+CouchbaseSquashGenerationContext+
CouchbaseSnapshotCapture,CouchbaseSquashVerifier,
CouchbaseMigrationSourceScanner). [DataMigration]and[StructuralOnly]attributes -- the
Roslyn-based source scanners refuse squash generation if a migration class
matches the data-op heuristic without an explicit annotation.- Generation-time fleet gate types:
SquashMetadata,SquashFleetGate
(EnsureGenerable),MidRangeFleetException. (The deploy-time half --
EnsureDeployable,StaleFleetMemberException,
UnregisteredEnvironmentException-- was cut before ship.) RecoveryAcknowledgement— deterministic 12-char token for the
recover from-mid-rangeescape hatch.- Per-provider
MigrationRunnersubclasses (PostgresMigrationRunner,
MongoDBMigrationRunner,CouchbaseMigrationRunner,
OpenSearchMigrationRunner,AerospikeMigrationRunner). Each provides
a unique DI handle so multi-provider hosts can resolve and run each
provider's runner independently. See the multi-provider
hosts operator guide. IEphemeralProvisionerabstraction (+ Couchbase sibling-container variant).
The per-provider squash CLI capture orchestrators consume an
IEphemeralProvisionerfor container provisioning, decoupling lifecycle
from the apply/capture pipeline. Each Squash package ships a default
Testcontainers-backed provisioner; the Couchbase package additionally
shipsCouchbaseSiblingContainerProvisionerfor the case where the CLI
itself runs inside a Docker container (CI pipelines, containerized
operator tooling). Provider provisioners are DI-overridable: the
default{Provider}SquashProvider()ctor wires the default
Testcontainers impl; a second(IEphemeralProvisioner)ctor accepts
a caller-supplied provisioner for integration tests and third-party
embeddings.- 5 end-to-end SquashProvider integration tests + CLI binary E2E.
One SquashProvider integration test per provider (Postgres,
Aerospike, OpenSearch, MongoDB, Couchbase). Each test loads the
corresponding sample assembly by path (Assembly.LoadFrom),
discoversIMigrationHost, builds aSquashRequest, and invokes
provider.GenerateAsyncend-to-end. Determinism gate (C12): the
Postgres variant runsGenerateAsynctwice against the same sample
and asserts byte-equal output. Couchbase taggedLocalOnlyper F-1
v3.0.1 follow-up (sibling-container model).CliBinaryEndToEndTests
spawns the actualhyperbee-migrations.exechild process against
the Postgres sample + a live Postgres Testcontainer and verifies the
emitted.sql,.metadata.json, and.summary.mdartifacts. All
pass on net8.0, net9.0, and net10.0. - Plugin-style
AssemblyLoadContextisolation for the CLI binary
.MigrationAssemblyLoadernow defers shared-type
identity to the Default ALC (soIServiceCollection,
IServiceProvider, etc. type-match across the host/plugin boundary)
AND probes the NuGet cache directly via the migration assembly's
.deps.jsonfor transitive packages that library projects don't
carry in their bin folder.SquashProviderRegistry.Discover
supplements the metadata reference closure with a directory scan so
<ProjectReference>packages whose types the migration project
doesn't directly use (a common shape for the Squash packages)
still surface through discovery. Without this, the CLI binary
reported "Discovered providers: " or threw
MissingMethodExceptionat the first cross-ALC call. Set
HYPERBEE_CLI_ALC_TRACE=1to surface every plugin-ALC resolution
step on stderr when diagnosing an operator's load failure. - OpenSearch resource-runner: leaf-filename dashes preserved.
OpenSearchResourceRunner.LoadBodyFromResourceno longer over-
sanitizes leaf filenames. MSBuild's manifest-name rule converts
dashes to underscores in folder segments but preserves them in leaf
filenames; the prior shared-helper sanitization treated all dashes
uniformly and silently failed to find resources like
WITH BODY @bodies/common-mappings-component.jsonwhose embedded
manifest entry is...bodies.common-mappings-component.json. - MongoDB test container: mapped public port (not fixed 28017).
MongoDbTestContainerpreviously bound28017:27017as a fixed
host port; the binding got retained by Windows HNS after Docker
container teardown and surfaced as "port is already allocated" on
the next test run. Mapped ports are allocated fresh per container
and avoid the retention path entirely; downstream consumers read
viaMongoDbTestContainer.ConnectionStringrather than assuming a
fixed host:port. - OpenSearch ISM lifecycle DSL —
DROP POLICY+DETACH POLICY FROM INDEX.
Closes the CREATE/APPLY/DETACH/DROP symmetry for ISM policy management.DROP POLICY <id> [IF EXISTS]deletes
the policy viaDELETE _plugins/_ism/policies/<id>(the cluster rejects
with 409 if any index still references the policy -- run DETACH first).
DETACH POLICY FROM INDEX <pattern> [NO WAIT("<reason>")]calls
POST _plugins/_ism/remove/<pattern>and reportsupdated_indices
count; zero-match is treated as an idempotent no-op (informational, not
failure) so operator teardown scripts stay rerunnable. The legacy
_opendistro/_ismendpoint prefix is honored automatically via the
existingIsmEndpointCapabilitybootstrap. Both verbs participate in
the data-op classifier as structural ops (squash-replaceable).
Changed (back-compat preserved)
- F-1 partial close -- CouchbaseRunnerTest now runs in CI. The
prior LocalOnly tag onCouchbaseRunnerTestis removed; the test
now passes in CI on net8/9/10. Two fixes:
(1) Pincouchbase:community-7.6.2(was: Testcontainers.Couchbase
default of 7.0.2-community). 7.0.2 had a planner-catalog refresh
issue for new scopes/collections that surfaced as
IndexFailureException 12021 "Scope not found in CB datastore"on
CREATE PRIMARY INDEX; 7.6.x ships the fix.
(2) BumpretryCountfrom 3 to 60 on
CouchbaseTestContainer.ConfigureCouchbaseAsync's admin-API wait
(and from 1 to 150 on the bucket-ready wait).retryCountis the
real ceiling -- previously capped the budget at 15 seconds, too
tight for 7.6.2's ~12s warmup on a CI runner. - F-1 v3.0.1 -- 6 Couchbase squash tests remain LocalOnly for a
SEPARATE issue: the host-side cluster-map redirect. The Couchbase
SDK bootstraps via the host-mapped mgmt port, receives a cluster
map advertising internal Docker addresses (172.17.0.2:11210), tries
to connect there, and gets "response ended prematurely". The
?network=externalquery parameter requires
setupAlternateAddressesto be configured on the server -- the
Testcontainers.Couchbase library default setup callback does NOT
configure alt-addresses, so host-side SDK connections to an
isolated Couchbase container fail. The two unblocked paths are
(a)CouchbaseSiblingContainerProvisioner(scheduled for v3.0.1)
where the test/CLI process runs as a container on the same Docker
network, or (b) callingsetupAlternateAddresseson the server.
Tests gated:CouchbaseSquashDeterminismTests,
CouchbaseSquashVerificationTests,
CouchbaseSquashProviderIntegrationTests. Squash correctness
is byte-tested by 192 Couchbase unit tests in
Hyperbee.Migrations.Squash.Tests. - Per-provider integration matrix in CI (run_tests.yml). Each
Postgres/Aerospike/MongoDB/OpenSearch/Couchbase job spawns ONLY its
own provider's containers (viaHYPERBEE_TESTS_PROVIDERS_ONLY) and
runs ONLY tests targeting that provider (viaFullyQualifiedName
filter). Eliminates the resource pressure that came from one job
spinning up all 5 provider containers simultaneously on a
4-CPU / 16 GB GitHub-hosted runner -- the pressure was amplifying
eventual-consistency races inside Couchbase Server and surfacing
them as test flakiness. Unit tests run separately because they need
no containers at all.MultiProviderHostIntegrationTestsgets its
own job with Postgres + MongoDB. - Couchbase Squash provider package ships -- the fifth and final
ISquashProvider implementation for the v3.0 CLI extensibility
cascade.CouchbaseSquashProviderspins ephemeral Couchbase Server
containers viaTestcontainers.Couchbase, applies migrations through
the discoveredIMigrationHost, and captures via the shared
CouchbaseSnapshotCapturehelper. RB-3 fleet readiness probe runs
N1QLSELECT RAW MAX(...) FROM <bucket>.<scope>.<collection>against
the ledger keyspace; reads bucket/scope/collection from fleet manifest
topology overrides.--provider-option bucket-name=<name>required
for codegen (the snapshot scope is the bucket).
CouchbaseRestApiServiceis promoted from internal to public so the
Squash package can construct it without InternalsVisibleTo coupling. - CLI uses collectible
AssemblyLoadContextfor the migration assembly.
Previously usedAssembly.LoadFrom, which loads into the default ALC
and prevents unload. The collectible ALC (MigrationAssemblyLoader)
resolves transitively-referenced assemblies from the migration project's
output directory and unloads cleanly when the verb completes -- safe
for embedding the CLI in long-running hosts. CouchbaseRecordStore.IntersectWithAppliedAsyncrewritten to single N1QL
USE KEYSround-trip. Previously fanned out N parallelExistsAsync
KV probes -- a 500-migration squash auto-mark opened 500 concurrent
KV connections, risking throttle / retry storms on smaller clusters.
The new path issuesSELECT RAW META(d).id FROM <keyspace> d USE KEYS $ids
for a primary-key index hit; semantically identical, one round-trip,
no fan-out.- MongoDB + OpenSearch Squash provider packages ship as part of the
five-provider CLI extensibility cascade.MongoDBSquashProvider
spins ephemeralmongo:7containers viaTestcontainers.MongoDb;
OpenSearchSquashProviderspins ephemeral
opensearchproject/opensearch:2.18.0containers via the generic
Testcontainerspackage. Both route migration apply through the
discoveredIMigrationHostand emit.pqlscript form.
RB-3 per-provider readiness probes ship in both (Mongo:
N1QL-style aggregation over the migration ledger collection;
OpenSearch:_searchagainst the ledger index extracting the max
version from record_id). - CLI is a thin dispatch shell over
ISquashProvider. The CLI
assembly references zero provider packages; per-provider
CLI implementations are discovered via the migration assembly's reference
closure. NuGet package presence IS the registration: a migration project
addsHyperbee.Migrations.Providers.{Provider}.Squashto enable
hyperbee-migrations squash --provider {provider}codegen. v3.0 ships
PostgresSquashProviderandAerospikeSquashProvider(Week 2);
MongoDB / OpenSearch / Couchbase follow in Week 3-4. - RB-4 (apply-path reflection) closed. Provider CLI implementations
route migration apply through the discoveredIMigrationHost
-- no moreApplyToDataSourceAsyncstatic-method
reflection convention. The host class is the single supported
integration point. - R-5 (output file extension): emitted squash artifact filename uses
ISquashProvider.SquashFileExtensioninstead of a hardcoded.sql.
Postgres ->.sql; the four NoSQL providers ->.pql
(the recommended script form). - R-8 (per-provider source scanner dispatch): scanner dispatch routes
throughISquashProvider.ScanSourceinstead of hardcoding
PostgresMigrationSourceScanner.Scan. Each provider's package exposes
its own Roslyn scanner with provider-specific data-op heuristics. - R-4 (
--remove-originalsdefault to dry-run): the flag now LISTS
matched files without deleting; actual deletion requires
--confirm-delete. The version-delimited regex prevents false-positive
matches against names that contain the version as a substring
(Squash_1000.csdoes not match when squashing version 100). - RB-3 (fleet readiness probe per-provider):
FleetReadinessProbe
(replaces v1's Postgres-onlyFleetReadinessCheck) dispatches to
ISquashProvider.ProbeLastAppliedVersionAsync. Each provider's
implementation reads schema / table / namespace / set / index names
from the fleet manifest'stopology:overrides; no more hardcoded
public.migrations. recover from-mid-rangeroutes throughIMigrationHost(no longer
Postgres-coupled at the CLI tier). Reads--connection+--assembly,
activates the discovered host, persists the recovery row via the
host'sIMigrationRecordStore. Closes the Week 1 RB-2 "Postgres-only"
caveat; all 5 providers participate via the host contract.recover from-mid-rangepersists the acknowledgement to the ledger so
the runner picks it up on the next invocation, force-marks the mid-range
squash without running its body, and deletes the recovery row. Previously
the verb only validated the token and printed an audit summary -- the
operator had no automated path from "token validated" to "fleet member
unblocked"; the persisted recovery shipping in v3.0 closes that loop.
IntroducesMigrationRecordKind.Recovery(value 3);RecoveryRecordhelper
derives the deterministic row id from(env, squashVersion)and the
payload from(env, squashVersion, missing-versions); the runner
re-verifies the token before consuming the row, so a stale acknowledgement
from a previous incident with a different missing-set is rejected. v3.0
CLI persists via Postgres only; the remaining four providers wire through
the Week 2IMigrationHostdiscovery contract.- README quick-start uses the typed
PostgresMigrationRunnerinstead of the
baseMigrationRunner. The base type works in single-provider hosts but
throws in multi-provider hosts; the typed runner is the
documented entry point either way. The README also flags the multi-provider
pattern with a cross-link to the operator guide. squash --scan-sourceis required by default; explicit bypass requires
--no-scan="<reason>"(>= 20 chars). Source scanning is the
default-deny annotation gate; making it opt-in let operators ship squashes
that silently elided data ops. The bypass form preserves operator
autonomy (e.g. cluster-only scenarios with no source) while keeping the
choice auditable.squash --fleet-manifestis required by default; explicit bypass requires
--no-fleet-manifest="<reason>"(>= 20 chars). The fleet readiness gate degraded to a zero-phase no-op when the manifest was
omitted, hiding mid-range fleet members. The bypass is for solo-environment
squashes only.- CLI
ArgParserwhitelists flags per verb and rejects unknown long-options
with a did-you-mean suggestion (Damerau-Levenshtein-lite over the
per-verb known-flag set). A non-boolean flag missing its value
(e.g.--connection --range 1-2) now throws "flag --connection requires
a value" instead of being silently treated as the string"true".
Boolean flags (--remove-originals,--regenerate) retain value-less
semantics. - Fleet manifest YAML loader rejects unknown keys instead of silently
swallowing them. The previousIgnoreUnmatchedProperties()call let typos
through (squash-overidesparsed cleanly,expries: 2026-06-01produced
the default 30-day window) -- giving the operator the illusion that the
manifest was honored. v3.0 throwsMigrationExceptionwrapping the
YamlDotNet line/column on any unknown key. RegisterBaseAliasesremoves only helper-owned descriptors when a second
provider registers. Previously the second-provider flip called
RemoveAll<MigrationOptions>/RemoveAll<IMigrationRecordStore>/
RemoveAll<MigrationRunner>, which also wiped any user-supplied
registrations made before the firstAddXxxMigrationscall -- a
test-harness footgun where a bespoke fake store registered first vanished
as soon as a real provider was added. The marker now captures the
helper-installedServiceDescriptorinstances on first registration and
removes only those on the flip; user-supplied descriptors survive. In
multi-provider mode the throwing factory still poisons base-type
resolution by design (operators resolve typed runners) -- R-9's guarantee
is "your descriptor is not destroyed", not "your descriptor wins base-type
resolution".AddCouchbaseMigrationsvalidatesBucketNameat options-factory time.
Missing or whitespace-onlyBucketNamenow throws
InvalidOperationExceptionwith an operator-friendly message naming the
field plus the canonical fix (opts.BucketName = "..."). Previously the
failure surfaced as an obscureNullReferenceExceptioninside the Couchbase
SDK on the firstBucketAsync(null)call.MidRangeSquashExceptionprints the recovery acknowledgement token in
its message and exposes it asRecoveryTokenon the exception itself, so
operators have the token on hand during incident response without
recomputing it. NewMigrationOptions.EnvironmentNameproperty feeds the
token derivation; when unset, the token is computed against an<unset>
sentinel and the exception message includes a remediation note. Per- Runner snapshots the applied set once at startup instead of issuing a
per-migrationExistsAsyncround-trip. The loop now consults the in-memory
snapshot to decide skip-vs-run. On a 500-migration project the runner
formerly held the fleet lock for 500 sequential round-trips of nothing-but-
existence-probes; the snapshot collapses that to one bulk realtime read.
Up direction is correctness-stable (Up only adds records); Down direction
uses the start-of-run "exists?" answer, which is the correct semantic
(Down should revert what was present at the start, not chase concurrent
writers). The audit's PA-8 finding
(count-only optimization of the priorIsLedgerEmptyAsynchelper) is
dissolved by this change -- the full applied set is now consumed
pervasively, so sending all ids is fully justified. MigrationOptions.LockingEnableddefault flipped totrue. Production-grade safety:
the lazy path (callAddPostgresMigrations(...)and run) now acquires the provider's
native distributed lock. Operators who deliberately want lockless dev/test runs must
setopts.LockingEnabled = falseexplicitly. Existing consumers who never set the
property pick up locking automatically on upgrade -- if you have a CI deployment that
intentionally races (e.g., test fixtures that nuke + recreate the database between
runs), set the property explicitly.- All five provider record stores override
IntersectWithAppliedAsyncwith a
single-round-trip realtime read (PostgresWHERE = ANY, MongoDB
find _id $inwith majority+primary, Couchbase parallelExistsAsync,
AerospikeBatchGet, OpenSearch_mget realtime=true). - All five provider record stores override
IntersectWithSquashedAsyncfor
transitive squash satisfaction. Postgres uses
WHERE kind=1 AND replaces && ARRAY[...]; MongoDB usesfind { kind: 1, replaces: { $in: [...] } };
Couchbase uses N1QLWHERE kind = 1 AND ANY v IN replaces SATISFIES v IN [...] END;
OpenSearch uses_searchwith atermsfilter onreplaces; Aerospike uses a
filtered server-side scan onKind=Squashwith client-side replaces-array intersection
(R-15 -- previously the CHANGELOG incorrectly stated this override was deferred;
the code atAerospikeRecordStore.cs:309-361has shipped the implementation from
v3.0 day one). IMigrationRecordStore.IntersectWithSquashedAsyncDIM default is now fail-loud.
The DIM previously returned an empty set silently, which let mature
environments that auto-marked an inner squash misclassify asFreshagainst
an outer squash. v3.0 throwsNotSupportedExceptionwith a remediation
message naming the store type the first time the runner reconciles a
Kind=Squashdescriptor against a store that hasn't overridden the method.
v2 stores without any squash usage are untouched -- the runner reaches this
method only when a squash descriptor is being processed.- OpenSearch ledger index strict mapping extended with
kind(byte) and
replaces(long[]) fields. Existing v2-era indices receive an additive
PUT _mappingpatch on bootstrap, idempotent and IAM-aware. MigrationDescriptor(previously a private record onMigrationRunner)
is now a public core type so squash strategies can consume it.MigrationRunneracceptsILoggerFactoryin addition to
ILogger<MigrationRunner>. The new
primary constructor takesILoggerFactoryand creates a logger
categorized under the concrete runtime type so per-provider subclass
instances log under their own type names
(e.g.Hyperbee.Migrations.Providers.Postgres.PostgresMigrationRunner).
The originalILogger<MigrationRunner>constructor remains for back-
compat. Operators tailing logs by category may need to update filters.- Multi-provider hosts: calling
Add{Provider}Migrationsfor more
than one provider on the sameIServiceCollectionpreviously caused
silent shadowing — only the last-registered provider's runner ran.
The baseMigrationRunner/MigrationOptions/IMigrationRecordStore
resolutions now throwInvalidOperationExceptionwith a clear,
actionable message when multiple providers are registered; resolve
the typed{Provider}MigrationRunnerexplicitly. Single-provider
hosts are unaffected. (See the multi-provider hosts
operator guide atdocs/site/multi-provider-hosts.md.)
Pre-ship hardening
The v3.0 pre-ship audit closed the following before release. No behavior
change for correctly-configured consumers; these are robustness,
doc-accuracy, and dead-code items.
- Couchbase GSI rebalance flake fixed at the root. Index DDL during an
index-service rebalance ("rebalance in progress") was retried blindly,
leaving create-after-create and create-after-drop races. The provider now
exposes a singleCouchbaseIndexRetryhelper:
WithRebalanceRetryAsync(one 60x3s backstop -- single source of truth for
the retry budget),WaitForIndexReadyAsync(delegates to the SDK
WatchIndexesAsync; unnamed primary watched as#primary), and
WaitForIndexDroppedAsync(pollsGetAllIndexesAsyncuntil the index is
gone).CouchbaseRecordStoreandCouchbaseResourceRunnerboth route
through it; the prior triplicated rebalance-retry loop is consolidated.
A second axis was closed for the squash integration fixtures: cluster
idle (KVrebalancetask) is necessary but not sufficient for GSI DDL --
the index service runs its own initial topology placement on a fresh
cluster that the cluster-tasks endpoint does not surface. The isolated
Couchbase test container now performs a bounded indexer-DDL warmup
(sentinel create/ready/drop) before any test body, so the index service
is provably past initial placement before the first real CREATE INDEX.
The CI per-provider matrix is green 23/23 with the 6 previously-LocalOnly
Couchbase squash tests now running in CI via configured alternate addresses. - Aerospike lock-disabled readiness gate.
AerospikeRecordStore.InitializeAsync
previously only checked_client.Connected; on the lock-disabled path the
first ledger read could hit a not-yet-warm cluster. It now runs a sentinel
probe filtered by the existingIsTransientClusterErrorpredicate with a
60s bound, throwing a clearMigrationExceptionon timeout. - Documentation corrections.
docs/site/squashing-migrations.md: removed
the staleApplyToDataSourceAsyncapply-path (the CLI applies via the
discoveredIMigrationHost), corrected the CLI invocation example and the
recover from-mid-rangeflag list to matchRecoverVerb, and fixed the
AerospikeIntersectWithSquashedAsynctransitivity caveat.CHANGELOG.md
internal contradiction on the Aerospike override reconciled (R-15 shipped).
docs/site/supported-versions.md: non-ASCII em-dashes replaced (just-the-docs
ASCII constraint). Top-levelREADME.md: added a "What's new in v3.0" section. - Dead code removed.
ICouchbaseRestApiService.GetNodeStatusesAsync
(+impl, +RestApi.GetNodeStatuses),GetClusterInfoAsync(+impl; the
still-usedRestApi.GetClusterInfois retained), and the no-timeout
WaitUntilBucketReadyAsyncoverload had no callers and were deleted. - Two confirm-intent decisions recorded.
NullSquashStrategyis retained
as a public extension point (no first-party provider uses it); the
deploy-time fleet gate (SquashFleetGate.EnsureDeployable+
StaleFleetMemberException+UnregisteredEnvironmentException) was cut
as redundant before ship. - Accidental-drift cleanup. MongoDB / Postgres
appsettings.jsonSerilog
Overridekey corrected from the copy-pasted"Couchbase"to"MongoDB"/
"Npgsql". Couchbase runner DI helpers renamed to theAdd{Provider}Provider
/Add{Provider}Migrationsconvention used by the other four. Stale test
namespaceHyperbee.Migrations.Tests.Squash.Cli->.Squash.
Breaking changes (with safe back-compat paths)
-
IMigrationRecordStoregains three methods. Custom implementations
compile and run unchanged via the DIM defaults (WriteAsync(record, ...)
delegates to legacyWriteAsync(string);IntersectWithAppliedAsyncfalls
back to a per-idExistsAsyncloop;IntersectWithSquashedAsyncreturns an
empty set). Override these to opt into squash support. -
Provider record-store schemas gain
Checksum+Kindcolumns/fields.
Migration is automatic and idempotent on first v3 apply. Pre-existing rows
read asChecksum=null, Kind=Migrationand pass integrity validation.- Postgres:
ALTER TABLE ADD COLUMN IF NOT EXISTSforchecksumand
kind(withCHECK (kind IN (0,1,2))) andreplaces(bigint[]). - Aerospike, Couchbase, MongoDB: additive bins / fields; sparse on
pre-existing records. - OpenSearch: additive
PUT _mappingpatch (see Changed above).
- Postgres:
Operational notes
- Squash is operationally one-way. Once committed, original migration source
files are removed. Rollback to v2 against a squashed ledger is unsupported;
the documented recovery is backup-restore. - Mixed-version fleet hazard. Don't run v2 and v3 against the same ledger
simultaneously; deploy v3 to all environments before squashing. Safety nets:
the generation-time fleet readiness gate (MidRangeFleetException, refuses
to create a squash that would strand a listed fleet member) and the wired
apply-time refusal (MidRangeSquashException, refuses a mid-range
environment loudly withrecover from-mid-rangerecovery). The deploy-time
fleet-staleness gate was cut as redundant before ship.
Documentation
- Squashing migrations — the full operator guide
- Resource migrations — the
.pqlscript form +.down.pqlreversibility - Multi-provider hosts
- Upgrade guide v2 → v3