DAOS-19358 control: Fix dmg system erase for MD-on-SSD by tanabarr · Pull Request #18718 · daos-stack/daos

tanabarr · 2026-07-24T17:54:37Z

To completely wipe and reformat a DAOS system, dmg system erase can be used.

The system erase operation was failing in MD-on-SSD mode because the
 Management Service (MS) database was not being properly cleared during
 the erase process. This caused duplicate membership errors when ranks
 attempted to rejoin after erase, leading to operation timeouts.

 Root Cause:
 In MD-on-SSD mode, the control metadata (including raft DB) is stored
 on a mounted SSD device. The eraseAndRestart() function deleted raft DB
 files from the mounted device and immediately restarted via unix.Exec().
 When the control metadata device was remounted on restart, the raft DB
 files reappeared due to incomplete filesystem sync and mount caching,
 causing the membership database to retain old entries.

 Solution:
 1. Reordered SystemErase() operation sequence to erase leader DB first,
    preventing race conditions where replicas sync stale data from leader
 2. Added filesystem sync operations after raft DB deletions to ensure
    changes are committed to disk before process restart
 3. Implemented coordination logic to wait for MS replicas to restart and
    for raft leader election before allowing engines to join
 4. Increased exec restart delay from 50ms to 1s to ensure gRPC responses
    complete before process restart
 5. Enhanced error messages to guide users when UUID mismatches occur
 6. Added detailed trace logging throughout erase operation for debugging

 The fix ensures proper cleanup in both MD-on-SSD and traditional SCM
 configurations, with proper synchronization across all MS replicas.

Features: control

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Features: control Signed-off-by: Tom Nabarro <tom.nabarro@hpe.com>

github-actions · 2026-07-24T17:57:36Z

Ticket title is 'Fix dmg system erase for MD-on-SSD '
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-19358

daosbuild3 · 2026-07-24T17:59:33Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/1/execution/node/521/log

daosbuild3 · 2026-07-24T18:09:26Z

Test stage Build on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/1/execution/node/569/log

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2026-07-26T19:19:11Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/2/execution/node/521/log

daosbuild3 · 2026-07-26T19:24:35Z

Test stage Build on Leap 15 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/2/execution/node/569/log

daosbuild3 · 2026-07-26T21:02:14Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18718/3/testReport/

daosbuild3 · 2026-07-27T14:10:41Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18718/4/testReport/

daosbuild3 · 2026-07-27T21:53:46Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/4/execution/node/1675/log

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2026-07-29T20:04:51Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/6/execution/node/1463/log

kjacque

This is definitely a complicated problem... I find myself wondering about corner cases. What about:

There is only one MS replica.
A subset of the MS replicas are unresponsive, but there is quorum amongst those remaining (minus the original leader) to elect a new leader.
A subset of the MS replicas are unresponsive, and there is NOT quorum to elect a new leader once the current leader is subtracted from the set.

In general it would be nice to refactor the updated function to be more testable at the unit level. Erase is a pretty scary operation, especially since the system can wind up in a state requiring "manual recovery," which isn't really defined. If there's no way to close that window, we really need to document it.

kjacque · 2026-07-29T22:54:29Z

+	maxWait := 30 * time.Second
+	pollInterval := 500 * time.Millisecond


Would be better to find a way to inject these wait times - would make it easier to test.

kjacque · 2026-07-29T22:56:06Z

+	maxWait := 30 * time.Second
+	pollInterval := 500 * time.Millisecond


Same comment about the timing as above.

kjacque · 2026-07-29T22:58:23Z

+//  3. On the leader, raft DB is stopped early in the operation, creating a window
+//     where leadership changes cannot be safely handled
+//  4. If the leader crashes after stopping raft but before completing, manual
+//     recovery will be required (replicas/engines may be in inconsistent states)


How does a user recover in this situation?

kjacque · 2026-07-29T23:00:58Z

+		// Give the kernel time to complete pending I/O operations
+		time.Sleep(100 * time.Millisecond)


Does this wait time come from some specific documentation? Would be nice if there was a more deterministic way to tell it was done.

kjacque · 2026-07-29T23:04:45Z

 		if err := svc.eraseAndRestart(false); err != nil {
 			return nil, errors.Wrap(err, "erasing and restarting non-leader")
 		}
+		// Never reaches here - process is exec'd


I'd be inclined to panic or error here, so the issue can be discovered quickly if this ever changes.

kjacque · 2026-07-29T23:06:21Z

+	// superblock). They will remain in AwaitFormat until the leader calls
+	// ResetFormatRanks (which is idempotent for already-started engines) followed
+	// by StorageFormat to complete the format process.
+	if !isLeader {


Any reason not to break this block and the leader section out into their own helper functions?

kjacque · 2026-07-29T23:08:47Z

+		// Critical: DB is stopped but files not removed. Try to log and continue.
+		svc.log.Errorf("CRITICAL: failed to remove leader DB files, continuing: %s", err)
+	}
+	svc.log.Debug("SystemErase: LEADER - Raft database files removed")


Should maybe be in an else block, to avoid misleading messaging.

kjacque · 2026-07-29T23:09:21Z

+	unix.Sync()
+	time.Sleep(100 * time.Millisecond)


If the 100ms time is always needed after the sync, it'd be good to put it in a helper function.

kjacque · 2026-07-29T23:12:32Z

+	// Now tell MS replica peers to erase their databases and restart.
+	// When they restart and try to rejoin, they'll find the leader's DB is also clean.
+	for _, peer := range peers {


Hmm, so when the replicas come back up, they just form their own MS cluster, right? Which the old leader doesn't join yet, because it's still managing this erase process.

daosbuild3 · 2026-07-30T19:54:33Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/7/execution/node/1572/log

daosbuild3 · 2026-07-30T23:57:48Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18718/7/execution/node/1625/log

DAOS-19358 control: Fix dmg system erase for MD-on-SSD

95f6a78

Features: control Signed-off-by: Tom Nabarro <tom.nabarro@hpe.com>

tanabarr added 3 commits July 24, 2026 20:32

adjust sleep on exec

3a9e84d

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

fix build

26baabe

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

increased sleep on exec

7201ff7

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr added 2 commits July 28, 2026 02:09

fix harness_test

2616485

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

increased time for restart

284da1b

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr self-assigned this Jul 29, 2026

tanabarr requested review from kjacque, knard38 and mjmac July 29, 2026 13:48

tanabarr marked this pull request as ready for review July 29, 2026 13:49

tanabarr requested review from a team as code owners July 29, 2026 13:49

kjacque reviewed Jul 29, 2026

View reviewed changes

mjmac mentioned this pull request Jul 30, 2026

DAOS-623 test: instrument NLT fault-injection sweep hang #18756

Draft

		maxWait := 30 * time.Second
		pollInterval := 500 * time.Millisecond

		// Give the kernel time to complete pending I/O operations
		time.Sleep(100 * time.Millisecond)

Uh oh!

Conversation

tanabarr commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

daosbuild3 commented Jul 24, 2026

Uh oh!

daosbuild3 commented Jul 24, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 26, 2026

Uh oh!

daosbuild3 commented Jul 27, 2026

Uh oh!

daosbuild3 commented Jul 27, 2026

Uh oh!

daosbuild3 commented Jul 29, 2026

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jul 30, 2026

Uh oh!

daosbuild3 commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

tanabarr commented Jul 24, 2026 •

edited

Loading