Runtime rule hot-update for MAL and LAL

[wu-sheng]

Runtime rule hot-update for MAL and LAL

Operators can now hot-update OTEL MAL, log MAL, telegraf MAL, and LAL rule files
without restarting OAP. A new admin REST surface on port 17128 (off by default)
persists rule changes to management storage; every node in an OAP cluster converges
on the new content within ~30 s. Engine state, cluster Suspend/Resume, BanyanDB
schema-watch fence, and per-backend dropTable lifecycle make the apply end-to-end
safe across structural cutovers.

⚠️ The endpoint has no built-in authentication and is disabled by default.
Enable only behind a gateway with IP allow-list + auth. See
docs/en/security/README.md for the full operator duty.

---

Catalogs

Catalog	What it holds
otel-rules	OTEL MAL rule YAML files
log-mal-rules	Log-derived MAL rule YAML files
telegraf-rules	Telegraf MAL rule YAML files
lal	LAL rule YAML files

name mirrors the static filesystem layout — relative path under the catalog root
without extension. Segments [A-Za-z0-9._-]+ joined by /. No leading slash,
no .., no empty segments. Examples: nginx, aws-gateway/gateway-service,
k8s/node.

---

API surface

Write endpoints

POST /runtime/rule/addOrUpdate

Creates or replaces a rule. Body is the raw YAML (same shape you'd ship as a
static file under oap-server/server-starter/src/main/resources/<catalog>/).

curl -X POST --data-binary @vm.yaml \
  -H 'Content-Type: text/plain' \
  'http://OAP:17128/runtime/rule/addOrUpdate?catalog=otel-rules&name=vm'

Optional flags:

• allowStorageChange=true — required for edits that move storage identity
  (scope change, downsampling change, single/labeled/histogram switch on MAL;
  outputType change or rule-key add/remove on LAL). Drops the existing
  measure's data on BanyanDB; orphans old rows on JDBC/ES. Off by default.
• force=true — recovery flag. Bypasses the byte-identical no_change HTTP
  shortcut so a re-post of known-good content is treated as a fresh apply
  request: the persisted row is re-written and any peers stuck mid-Suspend
  are re-Resumed. Engine state is content-keyed, so a true no-op against a
  healthy node remains a no-op even with this flag.

# Recovery: re-post a known-good payload to unstick a peer + bypass no_change
curl -X POST --data-binary @vm-previous-known-good.yaml \
  'http://OAP:17128/runtime/rule/addOrUpdate?catalog=otel-rules&name=vm&allowStorageChange=true&force=true'

POST /runtime/rule/inactivate

Soft-pause. OAP stops emitting metrics for the rule; backend measure +
historical data preserved for a later reactivation. The "off" intent is
durable across reboots — bundled rules on disk are not auto-resurrected
when an inactivate removes the runtime override (use delete if the
operator wants bundled to take over).

curl -X POST 'http://OAP:17128/runtime/rule/inactivate?catalog=otel-rules&name=vm'

POST /runtime/rule/delete

Removes an INACTIVE row (active rules return 409 requires_inactivate_first).
The OAP-side teardown is uniform; the storage-side effect splits on whether a
bundled YAML exists on disk for (catalog, name):

• No bundled twin → destructive: backend resource is dropped and the rule
  is fully gone.
• Bundled twin exists → non-destructive revert: backend resources runtime
  claimed that bundled does NOT claim (or claims at a different shape) are
  dropped; bundled-shared at matching shape is preserved (no data loss for
  measures bundled will reuse). The runtime row is removed; bundled is
  reinstalled into a static: loader on the local node. Peers converge via
  the periodic reconcile.

?mode=revertToBundled is an explicit operator hint that fails with
400 no_bundled_twin when no bundled YAML exists — useful for scripts that
want to fail loudly on assumption mismatch. To kill a bundled rule entirely,
the bundled YAML on disk must be edited or removed (it's part of the OAP
image).

Backend	After destructive /delete (no twin)	Old data still queryable?
BanyanDB	Measure / stream group + schema dropped	No
Elasticsearch	dropTable no-op (merging index stays)	Yes (until TTL)
JDBC (H2/MySQL/PostgreSQL/TiDB/OceanBase)	dropTable no-op (table stays)	Yes (until TTL)

# Destructive removal of a runtime-only rule:
curl -X POST 'http://OAP:17128/runtime/rule/inactivate?catalog=otel-rules&name=my-custom'
curl -X POST 'http://OAP:17128/runtime/rule/delete?catalog=otel-rules&name=my-custom'

# Revert a bundled rule's runtime override back to the bundled YAML:
curl -X POST 'http://OAP:17128/runtime/rule/inactivate?catalog=otel-rules&name=vm'
curl -X POST 'http://OAP:17128/runtime/rule/delete?catalog=otel-rules&name=vm&mode=revertToBundled'

Read endpoints

GET /runtime/rule

Fetch one rule. Default order: runtime row first, bundled YAML second, 404
otherwise. Raw YAML by default; JSON envelope on Accept: application/json.
Supports ETag and If-None-Match.

?source=bundled reads the on-disk bundled YAML even when a runtime row
exists — useful for "compare runtime vs bundled" / "fetch bundled body for a
revert via addOrUpdate" workflows. Returns 404 not_found when the rule
has no bundled twin.

# raw YAML — runtime first, bundled fallback
curl 'http://OAP:17128/runtime/rule?catalog=otel-rules&name=vm'

# JSON envelope (carries contentHash, source, updateTime, content)
curl -H 'Accept: application/json' \
  'http://OAP:17128/runtime/rule?catalog=otel-rules&name=vm'

# Read bundled body even though a runtime override exists
curl 'http://OAP:17128/runtime/rule?catalog=otel-rules&name=vm&source=bundled'

GET /runtime/rule/bundled

Bundled rules for one catalog as JSON. withContent=true (default) ships the
full YAML body; withContent=false omits it. Each item flags whether an
operator override exists.

# every bundled rule, with body
curl 'http://OAP:17128/runtime/rule/bundled?catalog=otel-rules' | jq

# manifest only (lighter; useful for diff vs filesystem)
curl 'http://OAP:17128/runtime/rule/bundled?catalog=otel-rules&withContent=false' | jq

GET /runtime/rule/list

Single JSON envelope {generatedAt, loaderStats, rules} merging stored rules
with this node's local state. Each row carries status, localState,
suspendOrigin, loaderGc, loaderKind (RUNTIME / STATIC / NONE),
loaderName, contentHash, bundled (whether a YAML exists on disk),
bundledContentHash (when bundled=true), updateTime, and
lastApplyError. loaderStats exposes process-wide manager counters
(active and pending). Optional catalog= filters; unknown values return
400 invalid_catalog.

# all rules, all catalogs
curl 'http://OAP:17128/runtime/rule/list' | jq

# only one catalog
curl 'http://OAP:17128/runtime/rule/list?catalog=otel-rules' | jq

# find rules with apply errors
curl 'http://OAP:17128/runtime/rule/list' | jq -c '.rules[] | select(.lastApplyError != "")'

# rules whose runtime override has drifted from the bundled YAML
curl 'http://OAP:17128/runtime/rule/list' \
  | jq -c '.rules[] | select(.bundled == true and .contentHash != .bundledContentHash)'

GET /runtime/rule/dump

Downloads a tar.gz of stored runtime rules + manifest.yaml. Useful for DR
backup or for replaying through addOrUpdate to restore state.

# all catalogs
curl -o rules.tar.gz 'http://OAP:17128/runtime/rule/dump'

# one catalog
curl -o otel.tar.gz 'http://OAP:17128/runtime/rule/dump/otel-rules'

# extract one rule and re-apply
tar -xzf rules.tar.gz runtime-rule-dump/otel-rules/vm.yaml
curl -X POST --data-binary @runtime-rule-dump/otel-rules/vm.yaml \
  'http://OAP:17128/runtime/rule/addOrUpdate?catalog=otel-rules&name=vm&force=true'

Catalog shortcut routes

Implicit catalog — useful when scripting against a single catalog:

Shortcut path	Resolves to
/runtime/mal/otel/{addOrUpdate,inactivate,delete}	catalog=otel-rules
/runtime/mal/log/{addOrUpdate,inactivate,delete}	catalog=log-mal-rules
/runtime/lal/{addOrUpdate,inactivate,delete}	catalog=lal

# equivalent to /runtime/rule/addOrUpdate?catalog=otel-rules&name=vm
curl -X POST --data-binary @vm.yaml \
  'http://OAP:17128/runtime/mal/otel/addOrUpdate?name=vm'

telegraf-rules does not have a shortcut; use the canonical /runtime/rule/...
routes.

---

Response shape

Write endpoints return JSON: {applyStatus, catalog, name, message}. Successful
applies use applyStatus of applied, filter_only_applied, structural_applied,
reactivated, inactivated, static_tombstoned, deleted,
reverted_to_bundled, reverted_to_bundled_partial, or no_change. Failures
map to specific codes (e.g. requires_inactivate_first, no_bundled_twin,
invalid_catalog, invalid_mode, delete_refused, compile_failed,
storage_change_requires_explicit_approval, cluster_not_ready,
forward_unknown_operation, etc.) — full table in
backend-runtime-rule-api.md.

---

Notices

Security

• No built-in authentication. Module disabled by default; enable only when
  the network around port 17128 is operator-controlled.
• Never expose port 17128 to the public internet. Bind to localhost or a
  private interface; reach it through an operator-controlled gateway with
  IP allow-list + authentication.
• Audit every request. Rule content is arbitrary YAML that compiles into
  the OAP JVM — a malicious rule could exfiltrate data, spike resource use,
  or create metric-name collisions. Treat POST /runtime/rule/* as
  equivalent to shell access on the OAP host.
• Keep the port off the cluster-external interface even in cluster mode.
  The cluster-internal Suspend / Forward RPCs ride on the cluster bus
  gRPC server (shared with RemoteService / HealthCheck) — that is a
  separate transport from 17128 and follows the same security posture
  as the rest of the cluster bus.

Full security notice: docs/en/security/README.md.

Data-loss guardrail

allowStorageChange=true is an explicit "I accept data loss" affirmation, not
a routine flag — it drops the existing measure's data on BanyanDB and orphans
old rows on JDBC / ES. Prefer a rename (new metric name, new rule name) so the
old data keeps accumulating until TTL and new data starts fresh under a clean
identity.

Consistency model

• Persist is commit. Once /addOrUpdate returns 200, the cluster will
  converge on that content.
• Last write wins. Concurrent writes to different nodes serialize on the
  cluster main; the second write wins.
• Bounded convergence. Healthy structural commits land cluster-wide
  within 30 s (one periodic scan). Aborted commits self-heal within 60 s.
  Filter-only edits land locally in milliseconds and on every other node
  within 30 s.
• No quorum, no leader election, no two-phase commit. The runtime-rule
  entry in storage is the single source of truth.
• Samples for an affected metric are dropped during a structural cutover.
  This is by design — the schema is moving and in-flight samples have no
  valid landing.

BanyanDB schema-cutover fence

After firing a schema install or drop, OAP waits up to a bounded window
(default 2 s) for every BanyanDB data node to apply the change before
resuming dispatch. Best-effort by design: when all nodes confirm in
time, the 200 OK marks the moment the cluster's data boundary moves
(samples after 200 use the new shape, samples before use the old). On
laggard timeout, OAP logs a warning naming the laggards and proceeds
anyway so a single slow node doesn't wedge the apply.

Persistence + boot-time merge

Runtime rules persist in management storage and survive OAP restart. At boot,
OAP merges bundled rule files (under
oap-server/server-starter/src/main/resources/<catalog>/) with persisted
runtime rules, so the cluster never silently regresses to the bundled
defaults after a restart.

For BanyanDB specifically, schema mismatches at boot are visible, not
silent: if BanyanDB already holds a resource whose shape doesn't match
the declared model (e.g., a rule was edited on disk while OAP was offline),
OAP skips the resource, logs an ERROR with the declared-vs-backend diff,
and continues booting. Re-shape the metric via
POST /runtime/rule/addOrUpdate.

/delete storage semantics differ per backend

For the destructive path (no bundled twin), the OAP-side teardown is uniform;
the on-disk effect is not:

• BanyanDB drops the measure / stream group + schema → old rows gone.
• ES / JDBC keep the index / table (append-only TTL contract) → old
  rows queryable until TTL expires.

If you need the data gone immediately on ES / JDBC, drop the table
out-of-band with the storage backend's own tools after /delete returns.

When a bundled twin exists, /delete is non-destructive by default:
runtime-only / shape-mismatched measures are dropped, but bundled-shared
measures at matching shape are preserved (bundled re-registers against
them). To force destructive removal of a bundled rule, edit the bundled
YAML on disk and restart — bundled rules are part of the OAP image, not
data.

---

Documentation

• Concept doc: docs/en/concepts-and-designs/runtime-rule-hot-update.md
• REST API reference: docs/en/setup/backend/backend-runtime-rule-api.md
• Security notice: docs/en/security/README.md

Tests

• UTs across the new receiver plugin (engine / cluster / state / apply / rest).
• E2E under test/e2e-v2/cases/runtime-rule/:
  • mal-storage/{banyandb,elasticsearch,postgresql} — full 10-phase lifecycle
    (CREATE → FILTER_ONLY → STRUCTURAL → DUMP → 4× ILLEGAL → SHAPE-BREAK →
    INACTIVATE → ACTIVATE → DELETE → DUMP) with a per-phase step label so
    verification queries attribute data back to the phase that wrote it.
  • lal — LAL hot-swap with a log-mal aggregation rule; swctl asserts the
    extracted metric carries the swap-flipped step label.
  • cluster — 2-OAP convergence over ZooKeeper.

Dependency bumps

Driven by the new BanyanDB schema-consistency RPCs whose generated validation
code requires the protobuf-java 4.x runtime:

• gRPC 1.70.0 → 1.80.0
• protobuf-java 3.25.5 → 4.33.1
• pgv (protoc-gen-validate) 1.2.1 → 1.3.0
• Netty 4.2.10.Final → 4.2.12.Final
• Netty-tcnative 2.0.75 → 2.0.77

---

Checklist

• Update the documentation to include this new feature.
• Tests (UT, IT, E2E) added to verify the new feature.
• Update the CHANGES log.
• If it's UI related, attach screenshots below. (no UI changes)