Verification and Diagnostics

Verification and diagnostics

After the SUT ran, a test checks that the collaborator holds the expected state. Besides the test's own assertions, the fixture can fail a test whose client operations succeeded although a daemon crashed or logged an unexpected error, and it can keep a failure archive for later analysis.

Each scenario follows the same shape: the goal of the test, the fixture it builds independently of the system under test (SUT), the hand-off of connection details to the SUT, an optional condition step that reproduces an operational situation, the verification of the collaborator's state after the SUT ran, and cleanup.

Snippets run inside a test function with ctx := t.Context(). Fragments that do not show their own Run use the fixture named in the scenario (cluster, source, destination, link). Every snippet on this page compiles against the current API; replace the command-line writers with your own client in a real test.

Fail the test on unexpected daemon events and keep an archive

Register TerminateOnCleanup right after Run. It replaces the plain cleanup: when the test fails, a check fails or a finalization cleanup fails, it writes diagnostics, crash metadata and a manifest before terminating the cluster. Run go test -artifacts to keep the archive.

func TestWithVerification(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(
			ceph.PoolConfig{Name: "app-data", Application: "rados"})),
	)
	if cluster == nil {
		t.Fatal(err)
	}
	final, finalErr := cluster.TerminateOnCleanup(t, ceph.FinalizationConfig{Archive: true})
	if err != nil {
		t.Fatal(err)
	}
	if finalErr != nil {
		t.Fatal(finalErr)
	}

	// Expected events are listed; anything else fails the test after it ran.
	if _, err := final.Verify(ctx, ceph.VerificationConfig{
		// The MON, not the stopped OSD, reports health checks.
		Logs: []ceph.LogExpectation{{
			Name:    "planned OSD stop",
			Daemon:  "mon.a",
			Event:   ceph.LogClusterWarning,
			Pattern: `\(OSD_DOWN\)`,
		}},
	}); err != nil {
		t.Fatal(err)
	}

	// Caller-owned containers the archive should still see are removed
	// through the finalization, after the archive is written.
	client, err := testcontainers.Run(ctx, ceph.DefaultImage,
		cluster.WithClient(), ceph.WithIdleEntrypoint())
	final.Cleanup("client", func(ctx context.Context) error {
		if client == nil {
			return nil
		}
		return client.Terminate(ctx)
	})
	if err != nil {
		t.Fatal(err)
	}

	// Extra post-test checks run before the archive decision.
	final.Check("pool exists", func(ctx context.Context) error {
		_, err := cluster.Pool(ctx, "app-data")
		return err
	})

	// ... run the SUT ...
}

Add mirror daemons and other caller-owned daemons with DiagnosticsConfig.AdditionalContainers and VerificationConfig.AdditionalContainers, and mask application secrets with RedactValues. To collect by hand, use CollectDiagnostics and CollectCrashMetadata.

Read health, PG and pool state

ceph.Observe(cluster) returns read-only views: Status, HealthDetails, Pools, OSDStates, QuorumStatus, ObjectWatchers, BlocklistEntries, PGInconsistencies and more. cluster.WaitForClean and WaitForPGClean wait for a settled cluster; Pool.Status, Pool.Usage and Pool.PGs read one pool. Use them as the verification step, after the SUT ran.

Scrape the MGR Prometheus exporter

Use this when the SUT consumes Ceph metrics.

module, err := override.SetMGRModule(ctx, cluster, "prometheus", true)
// module is never nil: after a refusal its Restore does nothing.
defer module.Restore(context.Background())
if err != nil {
	t.Fatal(err)
}
var url string
observer, err := ceph.Observe(cluster)
if err != nil {
	t.Fatal(err)
}
for url == "" {
	services, err := observer.ManagerServices(ctx)
	if err != nil {
		t.Fatal(err)
	}
	url = services["prometheus"] // e.g. "http://172.19.0.3:9283/"
	if url == "" {
		time.Sleep(time.Second) // bound this loop with ctx in real tests
	}
}
metrics, err := observer.ManagerMetrics(ctx) // parsed samples; see ManagerMetrics
if err != nil {
	t.Fatal(err)
}
_ = metrics

In bridge mode each manager listens on its own container address on port 9283, and standbys answer with an empty 200 response. After a failover (cluster.Ceph(ctx, "mgr", "fail")), ManagerServices returns the new active manager's URL.

In host mode all managers share the host network, so give each manager its own port and bind the public address before enabling the module (using override.SetConfig). With the default settings and two managers, ManagerServices stayed empty and no manager served metrics:

facts, err := ceph.FactsOf(cluster)
if err != nil {
	t.Fatal(err)
}
settings := []ceph.ConfigSetting{
	{Section: "mgr", Name: "mgr/prometheus/server_addr", Value: facts.PublicAddress},
	{Section: "mgr", Name: "mgr/prometheus/a/server_port", Value: "19283"},
	{Section: "mgr", Name: "mgr/prometheus/b/server_port", Value: "19284"},
}
for _, setting := range settings {
	if _, err := override.SetConfig(ctx, cluster, setting); err != nil {
		t.Fatal(err)
	}
}

Managers added later with AddManager need their own port setting too. Pick ports that no other process or cluster uses.

References

• API reference: pkg.go.dev or go doc on each package
• Repository: README and guides
• Purpose and scope: API purpose and purpose tags