Umbrella Transfer Tool Credentials

Umbrella: Transfer Tool — Resolve Credential Aliases for Source/Target

Issue: noetl/ai-meta#99 Opened: 2026-06-14 Status: CLOSED 2026-06-14

---

Closing summary

Opened and closed same day (2026-06-14). The issue surfaced from the #98 Snowflake key-pair JWT session when the transfer_sf_to_pg step was the last red step in the data_transfer/snowflake_postgres fixture.

How it was solved:

• Option A (worker injects) was chosen: the worker pre-resolves the keychain alias on each transfer endpoint (source.auth / target.auth) before dispatch, exactly mirroring the task_sequence pre-resolution already in place.
• noetl/tools v3.10.0 (tools#65) implemented both transfer arms:
  • (Snowflake, Postgres): new SnowflakeTool::query_rows returns rows as Vec<HashMap<String,String>> (Snowflake returns every cell as a string); target column types fetched from information_schema.columns; each cell coerced via $n::text::<udt>; Snowflake's internal TIMESTAMP_TZ format (<epoch>.<nanos> <tzmin>) reformatted to RFC3339 before the PG cast.
  • (Postgres, Snowflake): generates SQL-escaped INSERT statements.
  • SourceConfig/TargetConfig add #[serde(flatten)] extra so the worker can inject resolved credential fields (account, warehouse, private_key, connection string, etc.) at dispatch time.
• noetl/worker v5.22.0 (worker#87) added transfer endpoint alias resolution and bumped noetl-tools to 3.10.0.
• noetl/e2e#58 migrated the fixture off the Python-era nested-auth map to string-alias auth + table-based auto-INSERT.

Validation: Full bidirectional data_transfer/snowflake_postgres fixture green on kind against the live sf_test account (NDCFGPC-MI21697). Every step COMPLETED: create_sf_database, setup_sf_table, create_pg_table, transfer_sf_to_pg, transfer_pg_to_sf. Real data moved with correct types: id(int) / name(text) / value(numeric 100.50) / created_at(timestamptz 2026-06-15 05:55:01.262+00) / metadata(jsonb) in the SF→PG direction; all 5 rows back in the PG→SF direction.

---

Goal

The transfer tool (noetl_tools::tools::transfer) inlines credentials in the tool config and cannot resolve credential aliases. When a data_transfer/snowflake_postgres fixture uses an aliased Snowflake source with key-pair fields (private_key, public_key), the transfer step fails — the tool has no path to look up the alias and has no Snowflake key-pair auth fields.

Acceptance:

• TransferConfig.source and TransferConfig.target support credential aliases (resolved via the worker's credential-alias mechanism, the same way other tools resolve auth: "{{ alias }}" through SNOWFLAKE_FIELD_MAP / POSTGRES_FIELD_MAP etc.).
• The Snowflake source path gains private_key / public_key fields (matching SnowflakeConfig) so the transfer tool can authenticate via key-pair JWT when the aliased credential carries those fields.
• The worked example fixture (repos/e2e/fixtures/data_transfer/snowflake_postgres) runs to playbook.completed on kind against the live sf_test account.

Context

Surfaced 2026-06-14 during the #98 Snowflake validation session. After the two tool: snowflake steps (create_sf_database + setup_sf_table) passed via key-pair JWT, the transfer_sf_to_pg step failed: the transfer tool reads inlined credentials and the Snowflake source connector has no private_key / public_key fields at all.

The immediate question is where credential injection belongs:

• Option A — worker injects at dispatch time, the same way apply_source_credential / SNOWFLAKE_FIELD_MAP already works for tool: snowflake steps. The transfer tool would declare its own field map so the worker merges the resolved credential into the source / target config before dispatch.
• Option B — transfer tool calls the credential API at run time, similar to how the server's CredentialService resolves aliases per data-access-boundary.md. This puts credential resolution inside the tool itself rather than in the worker dispatch layer.

Per agents/rules/data-access-boundary.md, business-logic credentials live in the NoETL keychain and are referenced by alias inside playbook steps. Either option must honor that boundary.

Recent activity

Date	What
2026-06-14	Umbrella closed. tools v3.10.0 + worker v5.22.0 + e2e#58 landed. Both transfer arms implemented; credential-alias resolution wired at the worker endpoint level (Option A). Full bidirectional fixture green on kind.
2026-06-14	Umbrella filed from the #98 Snowflake validation. transfer_sf_to_pg step fails: inline creds only, no alias support, no key-pair fields on the Snowflake source.

Pointers

• Transfer tool: repos/tools/src/tools/transfer.rs
• Snowflake config: repos/tools/src/tools/snowflake.rs (SnowflakeConfig)
• Worker field maps: repos/worker/src/auth_alias.rs (SNOWFLAKE_FIELD_MAP)
• Fixture: repos/e2e/fixtures/data_transfer/snowflake_postgres/
• Related: #98 (regression baseline)
• Related: Snowflake-Tool wiki page
• Rule: agents/rules/data-access-boundary.md