Skip to content

io-connectors: REST Output sink — write records to HTTP APIs #224

Description

@rustpunk

Context. Clinker can already read from REST APIs as a finite-pull Source. This adds the write direction: an Output sink that pushes records to an HTTP API.

Scope.

  • Add an Output sink backend in clinker-net that writes records to a REST endpoint (POST/PUT/PATCH/DELETE), per-record or batched, over the existing synchronous ureq + rustls transport.
  • Support auth (header / token / basic), configurable retry with backoff, and rate limiting.
  • Map per-record failures to the correlated dead-letter path rather than aborting the run; respect the memory budget with bounded batching / back-pressure.
  • Define the YAML surface for declaring a REST sink as an Output node.

Acceptance.

  • A pipeline writes records to a mock HTTP endpoint; failed records are dead-lettered with context; retry / backoff / rate-limit are covered by tests.

Depends on: none hard (uses the existing clinker-net REST / ureq path).

Success criteria

  • Records are written to an HTTP API in configurable batches with retry/backoff, timeout, and auth behavior consistent with the REST source's semantics.
  • The batch buffer is bounded and charged to the memory budget; batch size and in-flight limits are configuration with validated bounds.
  • A failed batch routes its records to the DLQ with correlation intact (or aborts, per the pipeline's error strategy); partial-success exit codes reflect sink failures.
  • End-to-end tests run against a local mock HTTP server covering success, retry-then-success, retry-exhaustion, and mid-batch failure.
  • Docs and a runnable example pipeline land with the feature.

Performance contract

Bounded memory is the binding constraint: every allocation introduced by this work must be charged to the memory arbitrator, spillable, or fail loud with the standard memory diagnostic — nothing may grow with input cardinality outside the budget. Within that constraint, maximize records/second. When memory boundedness and throughput conflict, memory wins.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-mode:implementImplement, verify, and closeagent-plan-firstAgent must inspect and plan before editingagent-size:LComplex one-outcome issue; plan firstarchitectureArchitectural design or structural changecrate:clinker-netNetwork finite-pull Source/Output connectors (REST, SQL)enhancementNew feature or requestrisk:highHigh-risk correctness, security, memory, architecture, or dependency work

    Projects

    Status
    Intake

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions