Provider Tool

ProviderTool (kind: provider)

Executes typed cloud-provider operation specs from a playbook step so that infrastructure provisioning and day-2 operations run as NoETL playbooks instead of shelling out to gcloud / Terraform. The first backend is Google Cloud over REST; the kind: provider surface keeps AWS / Azure as explicit, not-yet-implemented arms.

Source: src/tools/provider.rs. Registered in create_default_registry; the worker and the CLI local runtime (noetl exec --runtime local) both dispatch it — same tool, no per-runtime reimplementation.

Step shape

- step: ensure_project
  tool:
    kind: provider
    provider: google              # google (aws / azure are reserved seams)
    runtime: rest                 # rest (backend) | rust-sdk (deferred → rest)
    action: google.cloudresourcemanager.projects.ensure
    dry_run: "{{ workload.action != 'apply' }}"
    input:
      project_id: shastaratech-youtube-prod
      parent: folders/20-media
    auth:                         # REQUIRED for a real (apply-mode) call
      type: gcp_adc
      credential: gcp_org_admin
      scopes: ["https://www.googleapis.com/auth/cloud-platform"]
    poll:                         # optional — bounds the LRO wait
      max_attempts: 30
      interval_ms: 2000
      max_wait_secs: 120

Action grammar

google.<service>.<resource>.<verb>. The short <resource>.<verb> form is also accepted when a service: domain is supplied (the shape the gcp-org-playbooks emit) — e.g. projects.ensure + service: cloudresourcemanager.googleapis.com.

Action	Kind	Apply-mode converge
cloudresourcemanager.folders.list	read	single GET
cloudresourcemanager.folders.ensure	mutate	list under parent → create (LRO) if no ACTIVE folder with display_name
cloudresourcemanager.organizations.iam.get_policy	read	single getIamPolicy
cloudresourcemanager.organizations.iam.ensure_binding	mutate	getIamPolicy → add {role,member} if absent → setIamPolicy
cloudresourcemanager.projects.describe	read	single GET
cloudresourcemanager.projects.ensure	mutate	GET → create (LRO) under parent on 404
cloudbilling.projects.link	mutate	GET billingInfo → PUT if the account differs
cloudbilling.billing_accounts.iam.get_policy	read	single getIamPolicy
cloudbilling.billing_accounts.iam.ensure_binding	mutate	getIamPolicy → add binding if absent → setIamPolicy
serviceusage.services.list_enabled	read	single GET
serviceusage.services.enable	mutate	GET service state → enable (LRO) if not ENABLED
cloudresourcemanager.folders.delete	destroy	resolve ACTIVE folder by display_name → DELETE (LRO); confirm-gated
cloudresourcemanager.projects.delete	destroy	GET project → DELETE (LRO); confirm-gated
serviceusage.services.disable	destroy	GET service state → disable (LRO) if ENABLED; confirm-gated
cloudresourcemanager.organizations.iam.remove_binding	destroy	getIamPolicy → drop {role,member} → setIamPolicy; confirm-gated
cloudbilling.billing_accounts.iam.remove_binding	destroy	getIamPolicy → drop {role,member} → setIamPolicy; confirm-gated

Plan vs apply

• dry_run: true (default — never silently mutates). Returns the request the tool would send (result.data.would_call) without touching the network or minting a credential. result.data.converge names the strategy that apply mode would run (ensure_folder, ensure_iam_binding, …).
• dry_run: false (apply). Requires an explicit auth: alias — there is no ambient-ADC fallback. Omitting auth: on a real call yields a Configuration error and makes no network request.

Idempotent GET-first converge

Every mutating action reads actual state first and only writes when the desired state is absent, so re-running any ensure is a no-op (changed: false) once converged. The converge is stateless — desired-vs-actual is derived from the live cloud API, not a local state file — so it holds regardless of the platform's eventual state/ownership model.

IAM ensure_binding reads the whole policy, adds the member to the role binding (or appends a new binding), and writes the full policy back with the etag preserved for optimistic concurrency — it never clobbers unrelated bindings.

Long-running Operations (LRO)

Folder/project create return a long-running Operation. The tool polls GET {operation.name} until done or the configured bound (poll.max_attempts / poll.interval_ms / poll.max_wait_secs, whichever trips first). On timeout it returns a descriptive ExecutionFailed that names the operation and the resume path — re-running the ensure is safe (the GET-first converge makes the retry a no-op once the operation completes).

Overflow (documented, not built): an operation that legitimately exceeds any sane inline bound should hand off via the callback/hook pattern — the create call captures operation.name + execution_id, returns and frees the worker slot, and a watcher emits a resume event on completion. This keeps a worker slot from being held for a multi-minute cloud operation (the execution-model callback rule). See the module docs for the full shape.

Secrets

The resolved bearer token lives only on the outbound request — never in the result, the would_call echo, spans, or errors. A field-allowlist redaction scrubs any sensitive-looking key (token, secret, credential, api_key, …) from echoed input / request bodies as defence in depth.

Endpoint override (testing / emulators only)

endpoint: is a config-level override that points the tool at a mock server (wiremock) or a local Google API emulator so a playbook can be validated offline in noetl exec --runtime local without a single live cloud call. Never set it against a real workload.

tool:
  kind: provider
  provider: google
  action: google.serviceusage.services.enable
  dry_run: false
  endpoint: http://127.0.0.1:8089      # base URL → crm=/v3, billing=/v1, serviceusage=/v1
  # endpoint: { crm: http://mock/crm } # or per-service; unset services use the real host
  auth: { type: bearer, token: fake-for-mock }
  input: { project_id: p, service_name: youtube.googleapis.com }

A base-URL string derives the three service URLs by appending the same version segments the real hosts carry (/v3 for CRM, /v1 for billing + service usage) — so an offline run exercises the identical URL construction as production. The override reroutes a real ProviderTool (defaulted to Google hosts) onto the mock; every plan/apply/LRO code path is exercised, only the destination changes.

Confirm-gated destroy (Fork 2)

Destroy verbs (*.delete, services.disable, *.iam.remove_binding) are permitted but safety-gated so a deletion can never happen by accident or against a plan the human did not review:

1. Dry-run by default. A destroy dry_run: true resolves the target against live state (folder id, etag, lifecycle state, presence) and emits an explicit plan plus a plan_digest (SHA-256 over the live-resolved plan). It performs no deletion — only the read needed to resolve the plan. Destroy therefore needs auth: even to plan (a plan that isn't resolved against live state can't detect drift).
2. Apply requires a matching confirm digest. A destroy dry_run: false requires confirm: <plan_digest> — the digest the human reviewed. A blind apply (no confirm) is refused.
3. Stale confirmations are refused. At apply time the tool re-resolves the live plan and recomputes the digest. If the cloud drifted since review (etag changed, resource now absent, lifecycle state changed), the live digest no longer matches confirm → the apply is refused and nothing is deleted. This is the core safety property.
4. No wildcard / bulk / glob deletes. Targets are explicit literal identifiers; a display_name matching more than one ACTIVE folder is refused as ambiguous.
5. Explicit auth: still required for any real call.

# 1) plan — review the emitted plan_digest
- step: plan_destroy_folder
  tool:
    kind: provider
    provider: google
    action: google.cloudresourcemanager.folders.delete
    dry_run: true
    auth: { type: gcp_adc, credential: gcp_org_admin }
    input: { parent: organizations/1, display_name: 20-media }
# result.data.plan_digest → e.g. "830b4b0b..."; result.data.plan lists the exact target

# 2) apply — echo the reviewed digest
- step: apply_destroy_folder
  tool:
    kind: provider
    provider: google
    action: google.cloudresourcemanager.folders.delete
    dry_run: false
    confirm: "830b4b0b..."     # must equal the reviewed plan_digest
    auth: { type: gcp_adc, credential: gcp_org_admin }
    input: { parent: organizations/1, display_name: 20-media }

Executing real destructive cloud operations is a human-run step. The tool builds + tests the destroy path against mocks; agents do not run live destroys.

Ownership / drift / orphan projection (Fork 1)

Ownership, last-known-desired-state, and drift are modelled as an EHDB projection over the event log — not a Terraform-style state file. The stateless GET-first converge stays the read-actual path; the projection answers what do we own, what did we create, what drifted from desired, what is now orphaned.

Event contract. Every mutating converge (ensure + destroy, plan + apply) embeds a secret-free provider_fact in its result:

{
  "urn": "google:cloudresourcemanager:project:st-prod",
  "provider": "google", "service": "cloudresourcemanager",
  "resource_type": "project", "verb": "ensure", "stack": "shastaratech-prod",
  "desired": { "project_id": "st-prod", "parent": "folders/20" },
  "execution_id": 123, "outcome": "changed"
}

The urn is the declared desired identity (stable across an ensure and its matching delete). verb is ensure / delete / adopt (round 4); outcome is planned (dry-run intent) / changed / noop / deleted / absent / adopted (round 4 — actual accepted as desired). Ownership is scoped by an explicit stack: label (defaults to <unscoped>); set it to make orphan detection meaningful across runs.

Projection. The fact rides inside a step result payload. The #178 projection read-model (/api/ehdb/events) is secret-free by construction and deliberately excludes result payload bodies, so the ownership fold consumes the raw eventlog tier (/api/ehdb/tiers/eventlog, worker-served, carries full records) instead — a #178 tier, not a parallel store, and the control-plane read-model stays secret-free. The pure fold lives in noetl_tools::provider_state:

• fold_facts(&[ProviderFact]) -> OwnershipModel — last-known-desired per URN; applied ensure asserts ownership, applied delete tombstones it (a planned dry-run fact is intent, not ownership).
• compute_drift(model, urn, live_actual) -> Drift — InSync / Missing (owned but gone) / Modified{fields} / Untracked (live but unowned) / NotManaged. live_actual is the live GET normalized into the desired key space by the query adapter.
• detect_orphans(model, declared_urns) -> Vec<Owned> — URNs owned but no longer declared by the current stack run.

The pure halves of the I/O adapter shipped in round 4 (normalize_live maps a live GCP GET into the desired key space). For folding real event data, use provider_state::extract_facts_from_tier_records (round 5, 3.25.0), not the older extract_facts_from_eventlog — the latter probes fixed paths (result.data.provider_fact) that do not match the real event envelope (context.result.context.data.provider_fact, served as a payload string), so it silently folds nothing against real tier data. See the round-5 section below and noetl/ai-meta#191. extract_facts_from_eventlog remains only for callers that already hold a decoded event body shaped to those paths.

Reconcile policy: report / enforce / adopt (round 4 — drift adoption)

When infra is changed out-of-band (the GCP console, another tool), the tool must be able to accept the change rather than revert it or destroy-and-recreate. A reconcile: policy on the step selects the response for the mutating ensure actions (reads and the destroy verbs are unaffected):

Policy	Cloud effect	State effect	Gate
report (default)	none	none	none — a plain run is always safe
enforce	push desired → actual (the round-2 GET-first converge)	records the converge fact	explicit opt-in; dry_run gates plan/apply
adopt	none (read/GET only)	accept actual as the new desired → provider_fact (verb: "adopt")	opt-in + confirm-gated

report is the default, so an unqualified run changes nothing (the Terraform-plan shape). Converging now requires an explicit reconcile: enforce — a behavior change for any playbook that relied on dry_run: false alone.

adopt covers both cases the umbrella requires:

• Update-in-place — a resource we already own was tweaked out-of-band → adopt the new field values as desired (refreshes last-known-desired).
• Import / take-ownership — a resource created entirely outside NoETL (Untracked, no owning fact) → adopt transitions it Untracked → Owned (import: true in the result; the fold's ("adopt","adopted") arm inserts the URN).

adopt makes no cloud mutation — read/GET only. This is enforced structurally: the adopt handler only reaches resolve_actual (which issues GET / getIamPolicy), with no path to a create / update / delete / setIamPolicy. On apply it writes only the provider_fact, never the cloud.

adopt is confirm-gated, identical to the destroy gate:

1. dry-run emits the field-by-field drift diff (diff.<field> = {desired, actual}) + a plan_digest (SHA-256 over the live-resolved actual: urn, presence, normalized actual, etag, lifecycle state).
2. apply requires confirm: <plan_digest> — a blind adopt is refused.
3. a stale digest is refused — at apply the tool re-resolves live and recomputes the digest; if the resource drifted again since review, the live digest differs → refused, no state written.

Ungated adoption would launder an accidental or malicious out-of-band change into "desired" permanently; the confirm gate makes human review — "is this change legit?" — the whole point of adopt.

# 1) review the drift
- step: adopt_plan
  tool:
    kind: provider
    provider: google
    action: cloudresourcemanager.projects.ensure
    service: cloudresourcemanager.googleapis.com
    reconcile: adopt
    dry_run: true
    known_desired: { project_id: st-prod, parent: folders/1 }   # from the fold
    input: { project_id: st-prod, parent: folders/1 }
    auth: { type: gcp_adc, scopes: ["https://www.googleapis.com/auth/cloud-platform"] }
# → emits diff {parent: {desired: folders/1, actual: folders/2}} + plan_digest

# 2) accept the reviewed actual as the new desired (no cloud mutation)
- step: adopt_apply
  tool:
    kind: provider
    provider: google
    action: cloudresourcemanager.projects.ensure
    service: cloudresourcemanager.googleapis.com
    reconcile: adopt
    dry_run: false
    confirm: "<plan_digest from step 1>"   # stale digest → refused
    known_desired: { project_id: st-prod, parent: folders/1 }
    input: { project_id: st-prod, parent: folders/1 }
    auth: { type: gcp_adc, scopes: [...] }

report/adopt read last-known-desired from known_desired (supplied by the caller's EHDB raw-eventlog-tier fold) and diff it against the live actual (normalize_live'd from a GET) using the shared provider_state::drift_between primitive. fold_facts treats an applied adopt the same as an applied ensure — it asserts/refreshes ownership — so a subsequent report/orphans sees the adopted resource as owned with its new desired.

Query I/O adapter + stack-scoped ownership (round 5 — noetl-tools 3.25.0)

The provider_state fold gained the server-facing half of the query adapter and a multi-stack scoping fix. These are what the noetl provider {plan,drift,orphans,adopt} CLI verbs (noetl/cli) build on.

Supported mechanism (decided — not a stopgap). Provider ownership / drift / orphan state is folded from the EHDB raw eventlog tier (GET /api/ehdb/tiers/eventlog). A provider_fact rides inside the step result payload, so the raw tier — which serves the full event body — is where it is reachable; the secret-free projection read-model (/api/ehdb/events) deliberately excludes payload bodies and never carries the fact. The raw tier is a legitimate #178 tier, not a parallel store, so this reuses #178. This is the supported path, settled by an ADR on noetl/ai-meta#189 (2026-07-14): a first-class provider.resource.converged event type + curated projection column was rejected — carrying the payload-derived desired spec in the secret-free projection would be a carve-out of #178's secret-free-by-construction invariant (rewriting its two invariant tests), which we chose not to weaken. The fold adapter is extract_facts_from_tier_records (≥ 3.25.0); the older extract_facts_from_eventlog shipped in 3.23.0/3.24.0 is broken against real data (see below and noetl/ai-meta#191).

• extract_facts_from_tier_records / fold_tier_records — consume the wire-shape EventLogRecordView rows served by GET /api/ehdb/tiers/eventlog. Two things differ from the round-4 extract_facts_from_eventlog, and both bite against real data:

  1. the event body arrives as a JSON string in payload — unwrapped + re-parsed here;
  2. the real call.done envelope nests the fact at context.result.context.data.provider_fact (worker emit_call_done → build_call_done_result → ToolResult.data) — deeper than the round-4 fixed paths reach. A bounded recursive search (guarded by a strict ProviderFact deserialize) finds it regardless of the exact nesting.

  ⚠ Bug fixed here. The fixed-path extract_facts_from_eventlog shipped in 3.23.0 / 3.24.0 never matched the real envelope, so a fold over real event-log data on those versions silently returned nothing. It was only ever exercised against synthetic records shaped to the fixed paths. Do not trust an ownership/drift/orphan fold from 3.23.0 or 3.24.0. extract_facts_from_eventlog stays for decoded-body callers; extract_facts_from_tier_records is the path for real tier data.

• Stack-scoped ownership — detect_orphans folded all stacks into one model, so a multi-stack fold reported another stack's resources as orphans of the current run. Added:

  • OwnershipModel::for_stack(stack) / ::stacks() — narrow / enumerate;
  • detect_orphans_scoped(model, stack, declared) — the correct multi-stack shape;
  • detect_stack_conflicts(facts) — flags a URN owned by >1 stack without an intervening delete (a legitimate transfer A-deletes-B-adopts is not flagged). Warning only — it doesn't prevent two stacks managing one resource (nor does Terraform), it makes the contention visible.

  <unscoped> (the default when a step sets no stack:) behaves as a single global stack — no change for single-stack users.

The CLI surface itself is documented on the noetl/cli wiki — noetl provider page.

Decided against (not deferred)

• First-class provider.resource.converged event type + curated projection column — rejected by ADR on noetl/ai-meta#189 (2026-07-14), not merely deferred. The event type alone is cheap but useless (the fact body still isn't in the secret-free projection); making it useful requires putting the payload-derived desired spec into the curated projection column, which is a carve-out of #178's secret-free-by-construction invariant (rewriting its two invariant tests) plus server + worker + EHDB changes. We chose not to weaken that invariant. The raw-tier fold is the supported mechanism. Revisit only if the raw-tier fold's cost measurably bites — fold latency at scale, eventlog-tier retention/GC reclaiming facts the fold still needs, or the raw tier becoming unavailable to the query path — and then as its own umbrella with an explicit invariant-carve-out decision, not as a follow-up of #189.

Still deferred

• runtime: rust-sdk — accepted and mapped to the REST backend; the gRPC SDK backend is a later swap behind the same YAML surface.

Related

• Tool kinds overview
• noetl/cli wiki — executor crate architecture (the tools_bridge arm that dispatches this tool in local mode)
• Umbrella: noetl/ai-meta#189