[Feature] Cross-node RPC primitive

[pathosDev]

Size / Priority

• Size: S
• Reference: Erlang :rpc.call/4.

Rationale

Cross-node "call this actor at path X on node Y" is doable today via:

const remoteRef = new RemoteActorRef(targetNode, targetPath, cluster);
const reply = await ask(remoteRef, msg, 5000);

That's three concepts (RemoteActorRef construction, ask, timeout) for a common pattern. Erlang offers :rpc.call(Node, Module, Fn, Args) as a single primitive — same use case, single concept.

A cluster.rpc(node, path, msg) helper:

• Reduces boilerplate.
• Has a clear error type (RpcError) distinct from local errors.
• Centralises metric instrumentation (cluster-wide RPC counter, histogram).
• Documents the pattern as first-class instead of "you can compose it from primitives".

Reference: what Erlang does

%% Synchronous RPC
{ok, Result} = rpc:call(NodeName, Module, FunName, [Arg1, Arg2], 5000).
%% {badrpc, Reason} on failure

%% Async variant
rpc:cast(NodeName, Module, FunName, [Args]).

%% Multi-node
rpc:multi_call([Node1, Node2, Node3], Module, Fn, Args, 5000).
%% Returns {GoodResults, BadNodes}

Erlang's RPC is strictly to-an-actor: there's no general-purpose function call across nodes (functions are part of code modules, loaded per-node). actor-ts's natural equivalent is "tell-an-actor" — already what RemoteActorRef does. RPC is sugar.

Design sketch — actor-ts equivalent

// src/cluster/Rpc.ts (new) — or extension on Cluster

export class RpcError extends Error {
  constructor(
    message: string,
    public readonly node: NodeAddress,
    public readonly path: string,
    public readonly cause?: unknown,
  ) {
    super(message);
    this.name = 'RpcError';
  }
}

export interface RpcOptions {
  /** Per-call timeout.  Default: 5000 ms. */
  readonly timeoutMs?: number;
}

/**
 * Synchronous cross-node call.  Resolves with the reply or rejects with
 * RpcError on timeout / network failure / actor failure.
 *
 *   const reply = await cluster.rpc<Reply>(
 *     targetNode, '/user/services/orders', new GetOrder('42'), { timeoutMs: 3000 });
 */
async function rpc<R = unknown>(
  cluster: Cluster,
  node: NodeAddress,
  path: string,
  msg: unknown,
  options?: RpcOptions,
): Promise<R>;

// Multi-node variant
export interface MultiRpcResult<R> {
  /** Per-node successful results. */
  readonly succeeded: ReadonlyMap<string, R>;    // address-string → result
  /** Per-node errors. */
  readonly failed: ReadonlyMap<string, RpcError>;
}

async function rpcMulti<R = unknown>(
  cluster: Cluster,
  nodes: ReadonlyArray<NodeAddress>,
  path: string,
  msg: unknown,
  options?: RpcOptions,
): Promise<MultiRpcResult<R>>;

// Cast (fire-and-forget)
function cast(cluster: Cluster, node: NodeAddress, path: string, msg: unknown): void;

Method-on-Cluster ergonomic:

// extension methods on Cluster:
cluster.rpc(node, path, msg);              // sync
cluster.rpcMulti(nodes, path, msg);        // multi
cluster.cast(node, path, msg);             // fire-and-forget

Integration with existing actor-ts subsystems

• RemoteActorRef: existing primitive; rpc builds one internally.
• ask: existing primitive; rpc wraps it with explicit RpcError translation.
• Cluster.transport: unchanged.
• Metrics: cluster_rpc_total{node, outcome=ok|timeout|error} Counter + cluster_rpc_duration_seconds{node} Histogram. Subject to [Security] Prometheus cardinality attack via user-controlled label values #131's cardinality cap (node-label can grow with cluster size; bucketise via truncateAddress).
• MDC propagation: existing ask already propagates MDC; rpc inherits.
• Tracing: existing tracing wraps ask; rpc inherits.

Out of scope / non-goals

• Function-call-across-nodes (Erlang's rpc:call(Node, Module, Fn, Args)) — not idiomatic in TS. Use actor-message pattern.
• pmap-style fan-out compute — out of scope; rpcMulti is the closest. For real parallel compute, use Streams ([Feature] Akka-Streams DSL subset (SourceQueue, MergeHub, BroadcastHub) #147) or a worker pool.
• Authenticated RPC — relies on existing cluster TLS / mTLS.

Open design questions

1. Sync vs async error semantics: ask-based RPC always rejects on timeout. Should cluster.rpc re-package as RpcError always, or pass through the original? Recommend: always wrap in RpcError for type consistency; original error in cause.
2. Path resolution: do we accept relative paths ('services/orders') or only absolute ('/user/services/orders')? Recommend: absolute only — explicit + matches ClusterClient.send.
3. rpcMulti parallelism: send to all nodes in parallel and aggregate? Or sequentially? Parallel is more useful. Recommend: parallel, with Promise.allSettled.
4. Local-node shortcut: cluster.rpc(selfAddress, path, msg) — does this short-circuit to local ask, or still go over the loopback transport? Recommend: short-circuit (faster + tracks the local-tell metric).

Test plan

1. rpc happy path — 2-node cluster; rpc from A to actor on B; reply received.
2. rpc timeout — actor on B never replies; rpc rejects with RpcError (cause: timeout) after timeoutMs.
3. rpc actor not found — path doesn't resolve on B; rpc rejects with RpcError.
4. rpc to self-node — local short-circuit; bypasses cluster transport.
5. rpc network partition — B unreachable; rpc rejects with RpcError (cause: network).
6. rpcMulti mixed success/failure — 3 nodes; 2 reply, 1 times out; result has succeeded.size === 2 and failed.size === 1.
7. cast fire-and-forget — no return value; doesn't await; metric increments.
8. MDC propagation — set MDC; call rpc; remote actor's log.info includes MDC.
9. Tracing propagation — active span; rpc; remote actor.receive span linked.
10. Metrics correctness — counters + histogram emit.

Acceptance criteria

• cluster.rpc<R>(node, path, msg, opts?) exported.
• cluster.rpcMulti<R>(nodes, path, msg, opts?) exported.
• cluster.cast(node, path, msg) exported.
• RpcError class with node, path, cause fields.
• Local-node short-circuit.
• Metrics emitted with cardinality cap.
• Documentation: "Cross-node RPC" with single-call + multi-node examples.
• Test suite covers all 10 cases.
• CHANGELOG entry under "Cluster: RPC primitive".