You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Clock unit change requires a full cluster wipe — would a migration tool be accepted?
#2201
I'd rather ask before building: would you accept a tool that migrates a cluster's on-disk state from one ClusterClock time unit to another?#1444 suggests migration tooling is in scope — I want to check this case is too.
Aeron 1.50.2, Java cluster. We moved an existing 3-member cluster from MillisecondClusterClock to NanosecondClusterClock. On restart every member exits with:
ClusterException: FATAL - incompatible time unit: NANOSECONDS snapshot=MILLISECONDS
at ConsensusModuleAgent.onLoadBeginSnapshot(ConsensusModuleAgent.java:638)
The failure is clean — the snapshot is refused, not consumed, so reverting the clock brings the cluster back intact. We stopped all members and cleared their cluster directories. That was acceptable only because we were not yet live. A cluster already serving users has no such option: never change the unit, or take an outage and reset state that may have no external source of truth.
A snapshot-only tool isn't enough, because three artifacts carry the unit:
ServiceSnapshotLoader reads the unit but doesn't reject on mismatch
Recovery is recoverFromSnapshotAndLog, so a migrated snapshot would still replay entries timestamped in the old unit. Two ways round that:
Quiesce first — drain and snapshot so there's no log left to replay, then migrate only the two snapshots. Safer, but constrains the procedure.
Rewrite the log too — more general, considerably more invasive.
Either needs to be in-tree: ConsensusModuleSnapshotTaker is package-private, and the shipped tools (ConsensusModuleSnapshotPrinter, StandbySnapshotReplicator) don't rewrite.
Either way — would a short docs note on the supported procedure be welcome? Happy to open that PR regardless; it's the part I'd most have valued before starting.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I'd rather ask before building: would you accept a tool that migrates a cluster's on-disk state from one
ClusterClocktime unit to another? #1444 suggests migration tooling is in scope — I want to check this case is too.Aeron 1.50.2, Java cluster. We moved an existing 3-member cluster from
MillisecondClusterClocktoNanosecondClusterClock. On restart every member exits with:The failure is clean — the snapshot is refused, not consumed, so reverting the clock brings the cluster back intact. We stopped all members and cleared their cluster directories. That was acceptable only because we were not yet live. A cluster already serving users has no such option: never change the unit, or take an outage and reset state that may have no external source of truth.
A snapshot-only tool isn't enough, because three artifacts carry the unit:
SnapshotMarker.timeUnit, sessiontimeOfLastActivity, timer deadlinesSessionMessageHeader.timestampon every entryServiceSnapshotLoaderreads the unit but doesn't reject on mismatchRecovery is
recoverFromSnapshotAndLog, so a migrated snapshot would still replay entries timestamped in the old unit. Two ways round that:Either needs to be in-tree:
ConsensusModuleSnapshotTakeris package-private, and the shipped tools (ConsensusModuleSnapshotPrinter,StandbySnapshotReplicator) don't rewrite.Questions:
All reactions