feat(infra/dev): migrate PoC pipelines to pipeline-runner Job + Confi…

[DioCrafts]

…gMap (ADR-0045 Phase C.6)

Phase C.6 of ADR-0045 — the closing PR of Phase C. Migrates the dev manifests in infra/dev/ from SparkApplication CRs that ran the Scala pipeline-runner-spark JAR to batch/v1 Job + ConfigMap pairs that run the Go pipeline-runner (Phase C.5) over a structured pipelineplan.Plan (Phase C.1, executed by Phase C.2 against Iceberg).

Runner extension:

• internal/runner/args.go: new --plan-file flag. internal/runner/run.go resolves the Plan from --plan-file > PIPELINE_PLAN_FILE env var > PIPELINE_PLAN_B64 env var (the dispatcher's path). Dev YAMLs use the file path so the Plan JSON stays readable in kubectl describe configmap; the dispatcher (Phase C.4.a) keeps using the env var.
• Tests added for loadPlanFromFile happy path + missing file; one existing test updated for the new "no Plan source" error message.

infra/dev manifests:

• poc-pipeline-nodes.yaml fully rewritten. Three pipelines ship as Job + ConfigMap pairs:
  • online-retail-clean (read → filter quantity>0 AND price>0 → project
    • derived revenue → write_table create_or_replace)
  • online-retail-returns (same shape with WHERE quantity<0)
  • online-retail-cust (read transactions_clean → aggregate group_by customer_id with sum(revenue), count_distinct(invoice), count_distinct(country) → write_table create_or_replace)
• spark-smoke.yaml → pipeline-runner-smoke.yaml: trivial one-row read_table → limit 1 → write_table(append) for catalog connectivity validation.
• spark-ingest-online-retail.yaml deleted. CSV DataSource API is the Phase 0 inventory's flagged deferred case (re-route via connector-management-service); no runner-side replacement.
• spark-rbac.yaml deleted. Spark Operator goes away in Phase D; the pipeline-runner Jobs use the standard pipeline-runner service account.

online-retail-anomalies is intentionally NOT migrated: the CTE + CROSS JOIN shape needs the v2 join operator (libs/pipeline-plan only ships union in v1, per ADR-0045 § Migration plan / Phase C). When join lands the pipeline becomes a two-stage decomposition; the inventory doc tracks it.

Documentation:

• services/pipeline-runner/README.md fully rewritten for the Phase C surface (Plan-source priority order, CLI flags, env fallbacks, Phase A discoveries bundled with IcebergReader, distroless image size delta, dev manifest pointers).
• services/pipeline-runner-spark/README.md gains a SUPERSEDED header pointing at the new flow + migration map for the three Scala mains (PipelineRunner → pipeline-runner; IcebergToObjectStoreIndexer → iceberg-object-indexer (Phase A); ActionLogStreamSink → action-log-sink (Phase B)).
• docs/migration/pipeline-runner-spark-to-go-inventory.md gains a §G with the sub-PR map and the pre-requisite checklist showing what's done (Enhance frontend testing, localization, and UI improvements #1, [Epic] Service consolidation program: 96 → ≤30 (ADR-0030 / S8.1) #3) vs what's still pending (Enhance project and files flow with new features and fixes #2 CSV re-route, S8.1 absorb → agent-runtime-service #4 live smoke).

Test plan:

• go vet ./... clean across the full repo.
• go test -count=1 ./services/pipeline-runner/... green (all three packages; new loadPlanFromFile tests + the updated empty-env assertion).
• The dev YAMLs themselves are validated by kubectl apply against the dev k3s cluster — that smoke is the Phase D entry gate per ADR-0045 § Migration plan.