feat(pipeline-runner): execute pipelineplan.Plan with Iceberg providers (ADR-0045 Phase C.5)

[DioCrafts]

Summary

Phase C.5 of ADR-0045. The runner side of the new contract: this PR replaces the Scala/Spark pipeline-runner and the `spark-submit` shell-out with a distroless Go binary that decodes a `pipelineplan.Plan` from the `PIPELINE_PLAN_B64` env var (the dispatcher from #73 populates it) and executes it via `libs/pipeline-runtime` (#69) against new Iceberg providers introduced here.

Branch base: stacked on #69 (C.2) because the runner imports both `libs/pipeline-plan` and `libs/pipeline-runtime`. Merge order: C.1 → C.2 → C.5.

New: `internal/providers/`

File	Implements	Source pattern
`iceberg_reader.go` (`IcebergReader`)	`pipelineruntime.Reader`	Phase A indexer (`services/iceberg-object-indexer/internal/source/iceberg.go`)
`http_writer.go` (`HTTPWriter`)	`pipelineruntime.Writer`	Phase B sink (`services/audit-sink/internal/writer/iceberg.go`)

`IcebergReader` carries every Phase A discovery: blank import `_ "github.com/apache/iceberg-go/io/gocloud"` (S3/GCS/Azure scheme handlers), `s3.remote-signing-enabled=false` override (Lakekeeper advertises `s3.signer-uri` but iceberg-go v0.5.0's gocloud S3 adapter does not implement remote signing), and the same REST options (`WithCredential` / `WithAuthURI` / `WithScope` / `WithWarehouseLocation`).

`HTTPWriter` posts to `iceberg-catalog-service`'s `/openfoundry/iceberg/v1/append` (the Phase B pattern). Typed errors `ErrTableNotFound` (404) / `ErrSchemaMismatch` (409, 422) / `ErrCommitFailed` (5xx, transport). `X-Internal-Token` forwarded when set.

Rewritten: `internal/runner/`

File	Change
`args.go`	Drops `--pipeline-build-url` and the SparkApplication-shaped flags (`--inline-sql`, `--inline-format`). Adds the Iceberg/OAuth knobs the providers need (`--catalog-warehouse`, `--catalog-credential`, `--oauth-token-uri`, `--oauth-scope`, `--table-writer-url`, `--internal-token`, `--log-format`) plus env-var fallbacks. `--smoke` skips field validation so integration CI can hit it without any catalog config.
`run.go`	Replaces `ResolveSpec + submitToSpark` with: `loadPlanFromEnv` → `pipelineplan.Plan.Validate` → `pipelineruntime.Executor{Reader, Writer}.Run(ctx, plan)`. Switches to `slog` (text or json via `--log-format`) so the dispatcher's structured logs flow through.
`spec.go`	deleted — no more pipeline-build-service HTTP spec fetch. Plan is the wire contract.

Other touches

• `cmd/pipeline-runner/main.go` documents the new CLI surface; one less import (`runner.Usage` removed).
• `Dockerfile` collapses from a Spark-image multi-stage (Spark base + Iceberg JAR downloads + sbt-built Scala JAR + Go orchestrator) to a distroless static build mirroring `services/iceberg-object-indexer/Dockerfile` (Phase A). Image size drops from ~700 MB → ~25 MB.
• `go.mod` adds `apache/iceberg-go@v0.5.0` and pins `substrait-protobuf/go@v0.81.0` — the same Phase A workaround (v0.85.0 removed `extensions.SimpleExtensionURI` which `substrait-go/v7` still references). `arrow-go/v18` trails along at v18.5.2.

Test plan

• `go vet ./...` clean across the full repo
• `go build ./...` clean across the full repo
• `go test -count=1 ./services/pipeline-runner/...` — every package green:
  • `runner`: 10 sub-tests (happy-path arg parsing, env-var fallbacks, smoke-validation gate, unknown-flag rejection, mandatory-field matrix, dangling-flag, bad log-format, Plan env decode happy path, empty env, bad base64, bad JSON, `firstCatalogNameInPlan`)
  • `providers`: 7 sub-tests (HTTPWriter POST shape, token forwarding, 404 → `ErrTableNotFound`, 409/422 → `ErrSchemaMismatch`, 5xx → `ErrCommitFailed`, URL validation matrix)
  • `server`: unchanged

`IcebergReader` has no unit tests in this PR — its surface is too tightly coupled to `apache/iceberg-go` to fake cleanly without a real REST catalog. Smoke against a live Lakekeeper catalog is the validation gate per Phase A discovery (and per ADR-0045 § Phase C exit criterion).

Follow-ups

• C.6 migrates `infra/dev/poc-pipeline-nodes.yaml` from the SQL-shaped node configs to the structured `planner.NodeConfig` shape (#74 defined it) so the four PoC pipelines run on the new runner end-to-end. That PR also exercises the smoke gate in the dev k3s cluster.

🤖 Generated with Claude Code