Skip to content

feat(api): add spec.policy.ec for cold-data erasure coding - #19

Draft
tdakkota wants to merge 2 commits into
claude/16-private-backendfrom
claude/policy-ec
Draft

feat(api): add spec.policy.ec for cold-data erasure coding#19
tdakkota wants to merge 2 commits into
claude/16-private-backendfrom
claude/policy-ec

Conversation

@tdakkota

Copy link
Copy Markdown
Contributor

Adds spec.policy.ec — the cold-data erasure-coding tier oteldb merged in
oteldb#1279 — mirroring storagebackend.ECConfig's
shape ({data, parity, after}) onto tenant.ECScheme.

Erasure coding stores (data+parity)/data of the logical bytes and survives parity node losses:
{4,2} is 1.5x for two tolerated losses, against 3x for RF=3.

Why this belongs in the operator specifically

The engine gates EC on a shared-nothing cluster — cluster mode and private_backend. That is
precisely and only what an OtelDBCluster deploys: spec.etcd is required, and the default file
backend is one PVC per pod. A single-node oteldb cannot reach this policy, and neither can a cluster
on a shared bucket. So this is the one deployment shape where erasure coding is usable at all — and
until now it was not configurable here.

Validation, and the RF interaction

Every precondition fails silently upstream, so the operator (which owns the topology) turns each
into an InvalidSpec condition:

  • Shared backendspec.storage.backend: s3 without spec.cluster.privateBackend: true.
    oteldb only logs a warning and carries on with full-copy parts.
  • More shards than nodesdata+parity > spec.replicas. The ring clamps the owner set to the
    members it has instead of failing.
  • after past retention.maxAge — merge work whose output is dropped unread, the same rule the
    other cold tiers already get.
  • data+parity > 256, negative after — the bounds ec.Scheme.Validate enforces.

The interesting one: under EC the owner count is data+parity and the tenant's replication factor
is ignored
Storage.rfFor returns Scheme.Shards() before it ever looks at RF, so this covers
the unflushed head as well as the converted parts. I chose to validate rather than document:
spec.policy.ec together with spec.cluster.replicationFactor is rejected outright. Honouring one
and dropping the other would leave the CR stating a replication factor the cluster does not use, and
it is the same class of explicit-but-ignored setting the repo already rejects (maxSeriesSoft
without maxSeries, dead merge tiers). Dropping replicationFactor when adopting EC is a one-line
spec edit. It is documented as well, on ECSpec.Parity and in the README.

Rack safety (ceil((data+parity)/parity) distinct zones) is documented, not validated — the
operator cannot see the cluster's zone topology from the CR, and the engine converts and warns
rather than failing.

Shape

  • storage.policy.ec joins the reserved extraConfig paths, so the block stays modelled in full.
  • Absent spec.policy.ec renders byte-identically to today: no ec key, and no policy block at
    all if nothing else is set.
  • An unset after omits the key, which oteldb reads as zero — erasure-code every part.

Stack

Based on #17 (spec.cluster.privateBackend), which is based on #15 (spec.query), which is based on
#14 (spec.ingest). The privateBackend dependency is load-bearing, not just ordering: the EC
validation reads privateBackendOf(cr).

Merge #14#15#17 → this.

oteldb models storage.policy.ec ({data, parity, after}) but the operator had no way to
reach it. Erasure coding only applies to a shared-nothing cluster — cluster mode plus a
private per-node backend — which is exactly and only what an OtelDBCluster deploys, so
this is the one topology where the policy is reachable, and it was not configurable.

The preconditions all fail silently upstream, so the operator validates them: a shared
backend makes the policy inert, a scheme wider than spec.replicas gets clamped by the
ring, and data+parity replaces the tenant's replication factor outright — for the
unflushed head too. That last one is refused rather than documented: setting it next to
spec.cluster.replicationFactor would leave the CR stating a factor the cluster ignores.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant