Skip to content

Fault and Condition Reproduction

Jump to bottom
JaeSang Yoo edited this page Oct 30, 2026 · 1 revision

Fault and condition reproduction

Condition steps reproduce an operational situation the SUT must survive: a down OSD, held recovery, a full cluster, a scrub error, a fenced or frozen client. Each step returns a handle whose Restore or Resume undoes it, so the test can check the SUT both during and after the condition. Fault injection exists to reproduce conditions, not to stress the cluster.

Each scenario follows the same shape: the goal of the test, the fixture it builds independently of the system under test (SUT), the hand-off of connection details to the SUT, an optional condition step that reproduces an operational situation, the verification of the collaborator's state after the SUT ran, and cleanup.

Snippets run inside a test function with ctx := t.Context(). Fragments that do not show their own Run use the fixture named in the scenario (cluster, source, destination, link). Every snippet on this page compiles against the current API; replace the command-line writers with your own client in a real test.

The scenarios use this helper, which waits until a health check is reported:

func waitForHealthCheck(ctx context.Context, cluster *ceph.Container, code string) error {
	observer, err := ceph.Observe(cluster)
	if err != nil {
		return err
	}
	for {
		health, err := observer.HealthDetails(ctx)
		if err != nil {
			return err
		}
		if _, ok := health.Checks[code]; ok {
			return nil
		}
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(time.Second):
		}
	}
}

Keep an OSD down without data movement

hold, err := override.SetOSDFlag(ctx, cluster, ceph.OSDFlagNoOut, true)
// hold is never nil: after a refusal its Restore does nothing.
defer hold.Restore(context.Background())
if err != nil {
	t.Fatal(err)
}
osds, err := cluster.OSDs(ctx)
if err != nil {
	t.Fatal(err)
}
osd := osds[0]
timeout := 10 * time.Second
if err := osd.Stop(ctx, &timeout); err != nil {
	t.Fatal(err)
}
// Degraded reads and writes; the MON does not mark the OSD out, so no backfill
// starts. Expect OSD_DOWN, PG_DEGRADED and OSDMAP_FLAGS in HealthDetails.
if err := osd.Start(ctx); err != nil {
	t.Fatal(err)
}
if err := hold.Restore(ctx); err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}

To move data off a running OSD instead, use OSDContainer.SetIn(ctx, false) on its handle, wait with WaitForPGClean, and mark it in again.

Hold a pool relocation mid-flight

Create the state in which PGs have moved to new OSDs but objects are still being recovered, and test client I/O against it.

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithInitialOSDs(
		ceph.OSDConfig{DeviceClass: "fast"},
		ceph.OSDConfig{DeviceClass: "slow"},
	)),
)
// ... CleanupContainer and error check
app, err := cluster.CreatePool(ctx, ceph.PoolConfig{
	Name: "app", Application: "rados", Replicas: 1, DeviceClass: "fast",
})
if err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}
// ... write objects with the client
for _, flag := range []ceph.OSDFlag{ceph.OSDFlagNoBackfill, ceph.OSDFlagNoRecover} {
	hold, err := override.SetOSDFlag(ctx, cluster, flag, true)
	defer hold.Restore(context.Background())
	if err != nil {
		t.Fatal(err)
	}
}
if err := app.SetPlacement(ctx, ceph.PoolPlacement{DeviceClass: "slow"}); err != nil {
	t.Fatal(err)
}
// Pool.PGs now reports up/acting on the slow OSD with most PGs in
// active+recovery_wait+degraded. Reads of old objects trigger on-demand
// recovery and succeed; new writes succeed.
// Restore both flags, then poll Pool.PGs until every PG is active+clean on the
// slow OSD (WaitForClean alone can pass before the new mapping is reported).

Moving the pool back with app.SetPlacement(ctx, ceph.PoolPlacement{DeviceClass: "fast"}) reuses its original rule, which also makes it removable with Pool.Remove again.

Fill the cluster with low full ratios

change, err := override.SetFullRatios(ctx, cluster, ceph.FullRatios{
	NearFull: 0.0001, BackfillFull: 0.0002, Full: 0.0003,
})
// change is never nil: after a refusal its Restore does nothing.
defer change.Restore(context.Background())
if err != nil {
	t.Fatal(err)
}
if err := waitForHealthCheck(ctx, cluster, "OSD_FULL"); err != nil {
	t.Fatal(err)
}
// Ordinary RADOS writes now block. A librados write with the FULL_TRY
// operation flag fails with ENOSPC (errno 28) instead; reads still work.
if err := change.Restore(ctx); err != nil {
	t.Fatal(err)
}

The cluster needs a little existing data for the OSDs to cross the thresholds. To test only OSD_NEARFULL or OSD_BACKFILLFULL, lower only those ratios (for example NearFull: 0.0001, BackfillFull: 0.90, Full: 0.95); ordinary writes still succeed in those states. Setting FULL_TRY is the client's job; the module adds no client SDK.

Reproduce a scrub inconsistency and repair it

// A replicated pool with one PG and two replicas; the object "victim" exists.
var mapping struct {
	PGID          ceph.PGID `json:"pgid"`
	Acting        []int     `json:"acting"`
	ActingPrimary int       `json:"acting_primary"`
}
out, err := cluster.Ceph(ctx, "osd", "map", "app", "victim", "--format", "json")
if err != nil {
	t.Fatal(err)
}
if err := json.Unmarshal(out, &mapping); err != nil {
	t.Fatal(err)
}
replica := mapping.Acting[0]
if replica == mapping.ActingPrimary {
	replica = mapping.Acting[1]
}
if err := fault.InjectObjectDataError(ctx, cluster, "app", "victim", replica); err != nil {
	t.Fatal(err)
}
if err := cluster.DeepScrubPG(ctx, mapping.PGID); err != nil {
	t.Fatal(err)
}
observer, err := ceph.Observe(cluster)
if err != nil {
	t.Fatal(err)
}
objects, err := observer.PGInconsistencies(ctx, mapping.PGID)
if err != nil {
	t.Fatal(err)
}
// objects[0].Name == "victim", objects[0].UnionShardErrors == []string{"read_error"}
if len(objects) == 0 || objects[0].Name != "victim" {
	t.Fatalf("unexpected inconsistencies: %+v", objects)
}
if err := waitForHealthCheck(ctx, cluster, "PG_DAMAGED"); err != nil {
	t.Fatal(err)
}
if err := cluster.RepairPG(ctx, mapping.PGID); err != nil {
	t.Fatal(err)
}

After the repair the health checks clear, and a new DeepScrubPG reports no inconsistency.

Fence one client session

// address is the client's own session address, e.g. from rados_getaddrs or
// go-ceph Conn.GetAddrs: "172.19.0.6:0/3461995546".
fence, err := override.AddBlocklist(ctx, cluster, address, time.Minute)
// fence is never nil: after a refusal its Restore does nothing.
defer fence.Restore(context.Background())
if err != nil {
	t.Fatal(err)
}
// The fenced session's operations fail (ESHUTDOWN in librados); another session
// from the same host keeps working.
if err := fence.Restore(ctx); err != nil {
	t.Fatal(err)
}
// The client can resume or reconnect, depending on its library.

BlocklistEntries shows the entry and its expiry. For RBD exclusive-lock handover, the lock owner's address comes from rbd.ImageClients (see rbd).

Detect a stuck watcher

A client that holds a watch and then stops responding blocks notifies on that object until the OSD's osd_client_watch_timeout (30 seconds by default) expires.

// `watcher` is a WithClient container whose process watches "notify-object".
pause, err := fault.Pause(ctx, cluster, watcher)
if pause != nil {
	defer pause.Resume(context.Background())
}
if err != nil {
	t.Fatal(err)
}
observer, err := ceph.Observe(cluster)
if err != nil {
	t.Fatal(err)
}
watchers, err := observer.ObjectWatchers(ctx, "app-data", "", "notify-object")
if err != nil {
	t.Fatal(err)
}
// The frozen client is still listed, and a notify from another client fails
// with ETIMEDOUT (errno 110) after its own timeout. Poll ObjectWatchers: after
// the watch timeout the list is empty and notifies complete at once.
_ = watchers

An expired watch does not blocklist its client; BlocklistEntries stays unchanged.

Network interruptions between clusters, messenger delays and frozen clients in service scenarios are shown on RBD, CephFS, RBD Mirror and RGW Multisite. The fault injection guide lists every condition and how it is restored.

References

Home

Clone this wiki locally