Skip to content

RBD

Jump to bottom
JaeSang Yoo edited this page Nov 1, 2026 · 2 revisions

RBD

rbd.WithPools creates initialized RBD pools at startup, rbd.CreateNamespace adds namespaces, and Cephx identities from CreateClient give the SUT exactly the capabilities it needs. Observation helpers such as rbd.ImageClients report watchers and lock owners for the verification step.

Each scenario follows the same shape: the goal of the test, the fixture it builds independently of the system under test (SUT), the hand-off of connection details to the SUT, an optional condition step that reproduces an operational situation, the verification of the collaborator's state after the SUT ran, and cleanup.

Snippets run inside a test function with ctx := t.Context(). Fragments that do not show their own Run use the fixture named in the scenario (cluster, source, destination, link). Every snippet on this page compiles against the current API; replace the command-line writers with your own client in a real test.

Helpers

The examples share these helpers. run executes a command in any container and returns its exit code and combined output. poll repeats a check every two seconds until it holds or the context ends. Mirroring examples are on RBD Mirror, which has its own helpers; the package reference is rbd/mirror.

import (
	"bytes"
	"context"
	"io"
	"slices"
	"testing"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/fault"
	"github.com/JSYoo5B/ceph-testcontainers-go/override"
	"github.com/JSYoo5B/ceph-testcontainers-go/rbd"
	"github.com/testcontainers/testcontainers-go"
	tcexec "github.com/testcontainers/testcontainers-go/exec"
)

func run(ctx context.Context, ctr testcontainers.Container, cmd ...string) (int, string, error) {
	code, reader, err := ctr.Exec(ctx, cmd, tcexec.Multiplexed())
	if err != nil {
		return 0, "", err
	}
	out, err := io.ReadAll(reader)
	return code, string(out), err
}

func poll(ctx context.Context, t *testing.T, what string, done func() bool) {
	t.Helper()
	for !done() {
		select {
		case <-ctx.Done():
			t.Fatalf("%s: %v", what, ctx.Err())
		case <-time.After(2 * time.Second):
		}
	}
}

The examples write test data with rbd bench --io-type write, which goes through librbd like any application writer. Use your own client in a real test.

Test a librbd client in a namespace with read-write and read-only identities

Give the application its own namespace and two least-privilege identities: one that can create and write images in the namespace, and one that can only read.

func TestTenantNamespace(t *testing.T) {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
	)
	testcontainers.CleanupContainer(t, cluster)
	if err != nil {
		t.Fatal(err)
	}
	tenant, err := rbd.CreateNamespace(ctx, cluster, "volumes", "tenant-a")
	if err != nil {
		t.Fatal(err)
	}
	if err := cluster.WaitForClean(ctx); err != nil {
		t.Fatal(err)
	}

	scope := " pool=" + tenant.PoolName() + " namespace=" + tenant.Name()
	writer, err := cluster.CreateClient(ctx, "tenant-a-rw", ceph.ClientCaps{
		Mon: "profile rbd",
		MGR: "profile rbd" + scope,
		OSD: "profile rbd" + scope,
	})
	if err != nil {
		t.Fatal(err)
	}
	reader, err := cluster.CreateClient(ctx, "tenant-a-ro", ceph.ClientCaps{
		Mon: "profile rbd",
		MGR: "profile rbd-read-only" + scope,
		OSD: "profile rbd-read-only" + scope,
	})
	if err != nil {
		t.Fatal(err)
	}

	// Each container gets the cluster network, /etc/ceph/ceph.conf and only
	// its identity's keyring at KeyringPath(). Use your application image.
	start := func(id *ceph.ClientConfig) testcontainers.Container {
		ctr, err := testcontainers.Run(ctx, ceph.DefaultImage,
			id.WithIdentity(),
			ceph.WithIdleEntrypoint(),
		)
		testcontainers.CleanupContainer(t, ctr)
		if err != nil {
			t.Fatal(err)
		}
		return ctr
	}
	rw, ro := start(writer), start(reader)

	// An image in the default namespace, created with admin credentials in
	// the cluster's client toolkit (the control container may lack rbd).
	client, err := cluster.ClientContainer(ctx)
	if err != nil {
		t.Fatal(err)
	}
	if code, out, err := run(ctx, client,
		"rbd", "create", "--size", "16M", "volumes/shared"); err != nil || code != 0 {
		t.Fatalf("create volumes/shared: %d %s %v", code, out, err)
	}

	rbdAs := func(ctr testcontainers.Container, id *ceph.ClientConfig, args ...string) int {
		cmd := append([]string{"rbd", "--id", id.User(), "--keyring", id.KeyringPath()}, args...)
		code, _, err := run(ctx, ctr, cmd...)
		if err != nil {
			t.Fatal(err)
		}
		return code
	}
	if rbdAs(rw, writer, "create", "--size", "16M", "volumes/tenant-a/disk") != 0 {
		t.Fatal("writer could not create an image in its namespace")
	}
	if rbdAs(ro, reader, "info", "volumes/tenant-a/disk") != 0 {
		t.Fatal("reader could not read image metadata")
	}
	if rbdAs(ro, reader, "create", "--size", "16M", "volumes/tenant-a/other") == 0 {
		t.Fatal("read-only identity created an image")
	}
	if rbdAs(rw, writer, "info", "volumes/shared") == 0 {
		t.Fatal("namespace-scoped identity read an image in the default namespace")
	}
}

In the application, select the namespace on the I/O context and open images read-only with the read-only identity. A normal (writable) open registers a header watch, which needs write permission, so it fails with that identity. With go-ceph inside the reader container (illustration; go-ceph needs cgo and the Ceph libraries, so this snippet is not compiled here):

import (
	"github.com/ceph/go-ceph/rados"
	librbd "github.com/ceph/go-ceph/rbd"
)

conn, _ := rados.NewConnWithUser("tenant-a-ro")
_ = conn.ReadDefaultConfigFile() // /etc/ceph/ceph.conf written by ClientConfig.WithIdentity
_ = conn.SetConfigOption("keyring", "/etc/ceph/ceph.client.tenant-a-ro.keyring") // reader.KeyringPath()
_ = conn.Connect()
ioctx, _ := conn.OpenIOContext("volumes")
ioctx.SetNamespace("tenant-a")
img, err := librbd.OpenImageReadOnly(ioctx, "disk", librbd.NoSnapshot)

A failed write through a read-only open handle (EROFS) is a client-side refusal. To prove the server-side permission, test operations the server must deny: image creation, a direct RADOS write, or access to another namespace, which fail with EPERM/EACCES.

Cephx changes apply to new connections; an existing session can keep tickets with the old capabilities. Change capabilities with ClientConfig.UpdateCaps and revoke identities with ClientConfig.Delete (see ceph).

To remove the namespace before the cluster ends, delete its images (and empty its trash) through the client, then call rbd.RemoveNamespace(ctx, cluster, tenant).

Prepare the server for librbd client features

The client feature under test decides what the fixture must provide:

Client feature Server-side preparation
Clone, flatten Format 2 images with layering. Clone v1 needs a protected parent snapshot.
Trash move, restore, purge Initialized pool. Use a dedicated namespace so purges do not touch other tests' trash. The Python binding converts trash deferment times with local time, so run its process with TZ=UTC.
Live migration (prepare, execute, commit, abort) Replicated initialized pool, layering on source and target, source writers closed.
Group snapshots Format 2 images; flush and close writers before the snapshot. It is a checkpoint of those images, not an application transaction.
Exclusive lock exclusive-lock feature and two independent RADOS sessions. See the next example.
Encryption format/load (LUKS1, LUKS2) librbd built with libcryptsetup (included in the client role and all images). No journaling on the image.
Passphrase change librbd has no rekey API. Run the external cryptsetup luksChangeKey on the exported LUKS header (the client role and the all image ship cryptsetup) and write the changed header back.
Erasure-coded data Create the EC pool with ceph.ClusterConfig.WithPools and ErasureCode: &ceph.ErasureCodeConfig{..., AllowOverwrites: true}, keep the image metadata in a replicated rbd.WithPools pool, and pass the EC pool as the image's data pool.

The images, the Python bindings and the native libraries live in Linux containers; build and run go-ceph applications on Linux.

Hand over an exclusive lock from an unresponsive client

A client that freezes while it holds an image's exclusive lock keeps its header watch until osd_client_watch_timeout (30 seconds by default) expires. A second writer blocks until then, breaks the lock and blocklists the old owner. Fencing the frozen owner with a blocklist entry hands the lock over without waiting.

func TestLockHandover(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
	defer cancel()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
	)
	testcontainers.CleanupContainer(t, cluster)
	if err != nil {
		t.Fatal(err)
	}
	if err := cluster.WaitForClean(ctx); err != nil {
		t.Fatal(err)
	}
	client, err := cluster.ClientContainer(ctx)
	if err != nil {
		t.Fatal(err)
	}
	if code, out, err := run(ctx, client,
		"rbd", "create", "--size", "16M", "volumes/disk"); err != nil || code != 0 {
		t.Fatalf("create image: %d %s %v", code, out, err)
	}

	// waitOwner polls until a client other than `previous` holds the lock
	// and also watches the image.
	waitOwner := func(previous string) *rbd.ImageLock {
		var owner *rbd.ImageLock
		poll(ctx, t, "exclusive lock owner", func() bool {
			status, err := rbd.ImageClients(ctx, cluster, "volumes", "", "disk")
			if err != nil {
				t.Fatal(err)
			}
			owner = status.ExclusiveOwner()
			return owner != nil && owner.Address != previous &&
				slices.ContainsFunc(status.Watchers, func(w rbd.ImageWatcher) bool {
					return w.Address == owner.Address
				})
		})
		return owner
	}

	// startWriter is your helper: it runs an application container attached
	// with cluster.WithClient() whose librbd process opens volumes/disk and writes.
	first := startWriter(ctx, t, cluster)
	owner := waitOwner("")

	pause, err := fault.Pause(ctx, cluster, first)
	if pause != nil {
		// Registered after the container's cleanup, so it runs first.
		t.Cleanup(func() { _ = pause.Resume(context.Background()) })
	}
	if err != nil {
		t.Fatal(err)
	}

	// Fence the frozen owner instead of waiting for its watch to expire.
	fence, err := override.AddBlocklist(ctx, cluster, owner.Address, 10*time.Minute)
	if err != nil {
		t.Fatal(err)
	}
	startWriter(ctx, t, cluster) // the second writer
	waitOwner(owner.Address)     // takes over within seconds

	if err := pause.Resume(ctx); err != nil {
		t.Fatal(err)
	}
	// Check in your application that the old owner's next write fails with
	// ESHUTDOWN (errno 108). Only then lift the fence.
	if err := fence.Restore(ctx); err != nil {
		t.Fatal(err)
	}
}

Behaviour to rely on:

  • Without the fence, the second writer's first write blocks for roughly the watch timeout, then succeeds. Afterwards the new client owns the lock, the old owner no longer watches the image, and librbd has blocklisted the old owner's address.
  • With the fence, the frozen owner's watch disappears at once and the handover takes about a second.
  • After Resume, the old owner's writes fail with errno 108 (ESHUTDOWN).
  • Keep the fence until the old owner has seen the blocklist. The exclusive lock is cooperative: an old owner that is unfrozen before it sees the blocklist asks the current owner for the lock, receives it, and its writes succeed.
  • The blocklist entry that librbd adds when it breaks a lock is not owned by the fixture. It stays until Ceph expires it (one hour by default).
  • The cluster resumes paused containers during Terminate, but resume a caller-owned container explicitly before it is removed.

Fault injection primitives (fault.Pause, override.AddBlocklist, network interruptions, messenger delays) are described in the fault injection guide.

Seed an image and read it back on another cluster

src, err := source.ClientContainer(ctx)
if err != nil {
	t.Fatal(err)
}
dst, err := destination.ClientContainer(ctx)
if err != nil {
	t.Fatal(err)
}
// destination has an initialized pool "volumes" (rbd.WithPools or InitPool).

// Full backup with a baseline snapshot.
_, _, _ = run(ctx, src, "rbd", "snap", "create", "volumes/disk@baseline")
var full bytes.Buffer
if err := rbd.ExportBackup(ctx, src, "volumes/disk", &full); err != nil {
	t.Fatal(err)
}

// Incremental backup of the changes since baseline, ending at snapshot "next".
// ... write more data on the source ...
_, _, _ = run(ctx, src, "rbd", "snap", "create", "volumes/disk@next")
var delta bytes.Buffer
if err := rbd.ExportIncremental(ctx, src, "volumes/disk@next", "baseline", &delta); err != nil {
	t.Fatal(err)
}

// Restore on the other cluster: full first (it carries "baseline"), then the diff.
if err := rbd.RestoreBackup(ctx, dst, "volumes/disk", bytes.NewReader(full.Bytes())); err != nil {
	t.Fatal(err)
}
if err := rbd.RestoreIncremental(ctx, dst, "volumes/disk", bytes.NewReader(delta.Bytes())); err != nil {
	t.Fatal(err)
}
// volumes/disk now has the head, @baseline and @next of the source image.

The restored image is independent of the source: it stays readable and writable when the source cluster is stopped, and later writes do not change its restored snapshots. Applying the diff to an image without the baseline snapshot fails. For large images, write the archives to files (for example under t.TempDir()) instead of memory.

References

Home

Clone this wiki locally