Skip to content

Ceph Cluster Basics

Jump to bottom
JaeSang Yoo edited this page Nov 1, 2026 · 2 revisions

Ceph cluster basics

A test starts its own cluster with ceph.Run(ctx, ceph.DefaultImage, opts...), registers it for cleanup before checking the error, prepares the state it needs, and hands connection details to the SUT. The service packages (rbd, cephfs, rgw and the mirror and multisite packages) build on such a cluster.

Each scenario follows the same shape: the goal of the test, the fixture it builds independently of the system under test (SUT), the hand-off of connection details to the SUT, an optional condition step that reproduces an operational situation, the verification of the collaborator's state after the SUT ran, and cleanup.

Snippets run inside a test function with ctx := t.Context(). Fragments that do not show their own Run use the fixture named in the scenario (cluster, source, destination, link). Every snippet on this page compiles against the current API; replace the command-line writers with your own client in a real test.

The basic test

func TestService(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(
			ceph.PoolConfig{Name: "app-data", Application: "rados"})),
	)
	testcontainers.CleanupContainer(t, cluster)
	if err != nil {
		t.Fatal(err)
	}
	if err := cluster.WaitForClean(ctx); err != nil {
		t.Fatal(err)
	}

	// Hand-off: a container attached with the admin identity, or the
	// configuration and keyring for a client you start yourself.
	cfg, err := cluster.ConnectionConfig(ctx)
	if err != nil {
		t.Fatal(err)
	}
	_ = cfg // cfg.Conf and cfg.Keyring for the SUT

	// ... run the SUT ...

	// Verification: read the cluster's state with native commands or the observer.
	out, err := cluster.Ceph(ctx, "df", "--format", "json")
	if err != nil {
		t.Fatal(err)
	}
	_ = out
}

Give each test its own cluster. Use ClusterConfig.WithPools for pools that exist from the start, or CreatePool for a pool the test creates explicitly.

Attach a client with a restricted identity

Create an identity limited to one pool and namespace, attach a container that only has that identity's keyring, and verify the limits with a fresh connection.

reader, err := cluster.CreateClient(ctx, "reader", ceph.ClientCaps{
	Mon: "allow r",
	OSD: "allow r pool=app-data namespace=tenant-a",
})
if err != nil {
	t.Fatal(err)
}
facts, err := ceph.FactsOf(cluster)
if err != nil {
	t.Fatal(err)
}
client, err := testcontainers.Run(ctx, facts.Images.Client,
	reader.WithIdentity(),
	ceph.WithIdleEntrypoint(),
)
testcontainers.CleanupContainer(t, client)
if err != nil {
	t.Fatal(err)
}
// Inside the container: rados --id reader -p app-data -N tenant-a ls
// Writes, other namespaces and other pools are denied with EPERM.

// Widen the identity; the key stays the same. Omitted services lose their caps.
if err := reader.UpdateCaps(ctx, ceph.ClientCaps{
	Mon: "allow r",
	OSD: "allow rw pool=app-data namespace=tenant-a",
}); err != nil {
	t.Fatal(err)
}
// Use a new client process to observe the change; existing sessions may keep
// their old tickets for a while.
if err := reader.Delete(ctx); err != nil {
	t.Fatal(err)
}

Create an erasure-coded data pool for RBD images

RBD keeps image metadata in a replicated pool and can store data in an EC pool that allows overwrites.

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(3)),
	rbd.WithPools(ceph.PoolConfig{Name: "rbd-meta"}),
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{
		Name: "rbd-data", Application: "rbd",
		ErasureCode: &ceph.ErasureCodeConfig{K: 2, M: 1, AllowOverwrites: true},
	})),
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}
// In a WithClient container:
// rbd create --size 64M --data-pool rbd-data rbd-meta/volume

K+M OSDs (three here) are needed because each shard goes to a distinct OSD.

Simulate failure domains with CRUSH hosts and racks

Logical hosts, racks, roots and device classes let you test placement rules and domain failures on one Docker engine.

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithInitialOSDs(
		ceph.OSDConfig{Host: "host-a", Rack: "rack-a", Root: "tc-storage", DeviceClass: "fast"},
		ceph.OSDConfig{Host: "host-b", Rack: "rack-b", Root: "tc-storage", DeviceClass: "fast"},
		ceph.OSDConfig{Host: "host-c", Rack: "rack-c", Root: "tc-storage", DeviceClass: "fast"},
	).WithPoolDefaults(3, 2).WithDefaultCRUSHRoot("tc-storage")),
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
pool, err := cluster.CreatePool(ctx, ceph.PoolConfig{
	Name: "by-host", Application: "rados",
	FailureDomain: "host", CRUSHRoot: "tc-storage", DeviceClass: "fast",
})
if err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}
// Each PG now has one replica per host. Stopping the only OSD of host-a takes
// that whole failure domain down; with min_size 2 the pool stays writable.
_ = pool

Verify the hierarchy with cluster.Ceph(ctx, "osd", "tree", "--format", "json") or "osd", "find", "<id>".

Start without OSDs, then add storage

cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithNoInitialOSDs()))
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
// MON and MGR are up; the application can be tested before storage exists.
// Pool defaults stay size 2, min_size 1.
for range 2 {
	if _, err := cluster.AddOSD(ctx); err != nil {
		t.Fatal(err) // a non-nil partial OSD is still owned by the cluster
	}
}
if _, err := cluster.CreatePool(ctx, ceph.PoolConfig{Name: "application", Application: "rados"}); err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}

CreatePool before any OSD exists is refused by the failure-domain check.

Start without a manager

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithNoInitialManagers().WithPools(ceph.PoolConfig{Name: "application", Application: "rados"})),
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
// MON quorum and both OSDs are up and in; the pool exists. Direct RADOS I/O
// works, MGR-backed commands and PG statistics do not.
if _, err := cluster.AddManager(ctx, "a"); err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}

For a MON-only phase, drop WithPools and use ceph.ClusterConfig{}.WithNoInitialManagers().WithNoInitialOSDs(), because WithNoInitialOSDs refuses initial pools; create the pool with CreatePool after AddOSD. You can add the first manager before or after the first OSD.

Reproduce a minimal legacy placement setup

A client that implements only the oldest CRUSH and placement features can be tested against one MemStore OSD, a legacy CRUSH map and a pool with one PG, one replica and no hashpspool flag on any pool. The fixture has no option for a hand-written CRUSH map; install it with the native tools, as below. crushtool is in the official image and the published client and essential images.

// Ceph creates the MGR's .mgr pool with the MON's default flags, so turn
// the hashpspool default off before any pool exists.
legacyPools := filepath.Join(t.TempDir(), "legacy-pools.conf")
if err := os.WriteFile(legacyPools, []byte("[global]\nosd_pool_default_flag_hashpspool = false\n"), 0o644); err != nil {
	t.Fatal(err)
}
cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1).WithPoolDefaults(1, 1)),
	ceph.WithOSDStorage(ceph.OSDStorage{}.WithObjectStore(ceph.OSDMemStore)),
	ceph.WithOSDStorage(ceph.OSDStorage{}.WithBlockSize(256<<20)),
	ceph.WithConfigFile(legacyPools),
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}

// Keep OSDs from moving themselves into host buckets or setting a device
// class when they start, so the installed map stays as written.
for _, name := range []string{"osd_crush_update_on_start", "osd_class_update_on_start"} {
	if _, err := override.SetConfig(ctx, cluster, ceph.ConfigSetting{Section: "osd", Name: name, Value: "false"}); err != nil {
		t.Fatal(err)
	}
}
const legacyMap = `tunable choose_local_tries 2
tunable choose_local_fallback_tries 5
tunable choose_total_tries 19
tunable chooseleaf_descend_once 0
tunable chooseleaf_vary_r 0
tunable chooseleaf_stable 0
tunable straw_calc_version 0
device 0 osd.0
type 0 osd
type 10 root
root default {
    id -1
    alg straw
    hash 0
    item osd.0 weight 1.00000
}
rule replicated_rule {
    id 0
    type replicated
    step take default
    step choose firstn 0 type osd
    step emit
}
`
control, err := cluster.ControlContainer(ctx)
if err != nil {
	t.Fatal(err)
}
if err := control.CopyToContainer(ctx, []byte(legacyMap), "/tmp/crush.txt", 0o644); err != nil {
	t.Fatal(err)
}
if code, _, err := control.Exec(ctx, []string{"crushtool", "-c", "/tmp/crush.txt", "-o", "/tmp/crush.bin"}); err != nil || code != 0 {
	t.Fatalf("crushtool exited %d: %v", code, err)
}
if _, err := cluster.Ceph(ctx, "osd", "setcrushmap", "-i", "/tmp/crush.bin"); err != nil {
	t.Fatal(err)
}

pool, err := cluster.CreatePool(ctx, ceph.PoolConfig{
	Name: "legacy", PGNum: 1, Replicas: 1, MinSize: 1, NoHashPSPool: true, Application: "rados",
})
if err != nil {
	t.Fatal(err)
}
_ = pool // hand the pool name to the client under test
  • While any pool has hashpspool, the MON requires the OSDHASHPSPOOL feature of every client and refuses a client without it when it opens its MON session. NoHashPSPool clears the flag only on the pool it creates; the configuration file keeps it off .mgr and every later pool. Once .mgr exists, list every pool with Observer.Pools and check that Pool.Status reports no hashpspool in Flags.
  • CreatePool adds its own rule, tc-<pool>-replicated, which with the default osd failure domain also chooses OSDs directly. To use the map's replicated_rule, create the pool with cluster.Ceph(ctx, "osd", "pool", "create", ...) instead; such a pool is not one that Pool.Remove removes.
  • The legacy tunables lower the features a client needs to read the CRUSH map. Switching to modern tunables later (ceph osd crush tunables optimal) raises them again, which tests a client's refusal of unsupported features.
  • Ceph reports POOL_NO_REDUNDANCY for a pool with one replica; WaitForClean still passes.

Keep OSD storage small

Small logical files keep disk and tmpfs usage low. Skip mClock's startup benchmark before the first OSD starts and keep the setting for the cluster's whole lifetime, because restored defaults would run the benchmark at the next OSD start.

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithClusterConfig(ceph.ClusterConfig{}.WithNoInitialOSDs()),
	ceph.WithOSDStorage(ceph.OSDStorage{}.WithBlockSize(64<<20)),
	ceph.WithOSDStorage(ceph.OSDStorage{}.WithInMemory(256<<20)), // optional; omit for container-local files
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
// No Restore: Terminate disposes of the configuration database.
if _, err := override.SetConfig(ctx, cluster, ceph.ConfigSetting{
	Section: "osd", Name: "osd_mclock_skip_benchmark", Value: "true",
}); err != nil {
	t.Fatal(err)
}
for range 2 {
	if _, err := cluster.AddOSD(ctx); err != nil {
		t.Fatal(err)
	}
}

Known-good pairs for a workload of a few MiB, including OSD restarts, backfill to a third OSD and the removal of an original OSD, are 64 MiB OSDs with a 256 MiB shared cap, 128 MiB with 512 MiB, 256 MiB with 1 GiB, and 512 MiB with 2 GiB. Size both bounds for your own workload with headroom for replicas, metadata and recovery. The three limits are independent:

Resource Bounded by Includes
Logical storage per OSD OSDStorage.WithBlockSize BlueStore data, metadata and temporary writes inside each file
Shared RAM backing OSDStorage.WithInMemory Allocated tmpfs pages of all OSD files
Process memory Docker host or VM limits and daemon settings MON, MGR, OSD and client memory

If you customize daemon arguments or config files, confirm the effective value with cluster.Ceph(ctx, "config", "show", "osd.0", "osd_mclock_skip_benchmark").

Replace every monitor and refresh clients

Rolling replacement adds a new monitor before removing an old one, so quorum is never at risk. Owned daemons are updated automatically; your client containers are not.

cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithMonitorCount(3)))
// ... CleanupContainer, error check, client container `client` with WithClient()
original, err := cluster.Monitors(ctx) // a, b, c
if err != nil {
	t.Fatal(err)
}
for i, name := range []string{"d", "e", "f"} {
	if _, err := cluster.AddMonitor(ctx, name); err != nil {
		t.Fatal(err)
	}
	if err := original[i].Remove(ctx); err != nil {
		t.Fatal(err)
	}
}
if err := cluster.WaitForQuorum(ctx); err != nil {
	t.Fatal(err)
}
// Running sessions follow the monmap; a restarted client reads ceph.conf.
if err := cluster.RefreshClientMonitorConfig(ctx, client); err != nil {
	t.Fatal(err)
}
// For a client you configure yourself, write the new mon_host value.
observer, err := ceph.Observe(cluster)
if err != nil {
	t.Fatal(err)
}
addresses, err := observer.MonitorBootstrapAddresses(ctx)
if err != nil {
	t.Fatal(err)
}
t.Log("mon_host =", addresses)

For RBD and CephFS mirror links between clusters, call the link's RefreshMonitorConfig(ctx) (packages rbd/mirror and cephfs/mirror). For an RBD link, call Rebootstrap afterwards so that the destination peer gets the new source mon_host. See RBD Mirror and CephFS Mirror.

Connect a native client in host mode

On a Linux Docker host, host mode lets a test process use go-ceph or librados directly:

cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithHostNetwork())
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
cfg, err := cluster.ConnectionConfig(ctx)
if err != nil {
	t.Fatal(err)
}
// Write cfg.Conf and cfg.Keyring to files and point librados at them.
_ = cfg

See Host networking for remote engines, Docker Desktop and port selection.

Run a cluster that accepts only encrypted connections

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	ceph.WithNetworkConfig(ceph.NetworkConfig{}.WithMessengerMode(ceph.MessengerV2Secure)),
)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
cfg, err := cluster.ConnectionConfig(ctx)
if err != nil {
	t.Fatal(err)
}
// Hand cfg.Conf and cfg.Keyring to the client under test (for example an
// independent Messenger v2 implementation), or attach a container with
// cluster.WithClient() / client.WithIdentity(...).
_ = cfg

The client must reach the advertised addresses: run it in a WithClient container, or use host mode.

References

Home

Clone this wiki locally