Skip to content

feat(git_repository): add GitRepository resource + grant support#17

Merged
noel merged 4 commits into
datacoves:mainfrom
usbrandon:feat/git-repository
May 11, 2026
Merged

feat(git_repository): add GitRepository resource + grant support#17
noel merged 4 commits into
datacoves:mainfrom
usbrandon:feat/git-repository

Conversation

@usbrandon
Copy link
Copy Markdown
Contributor

@usbrandon usbrandon commented May 10, 2026

Summary

Adds Snowflake GIT REPOSITORY support and fills two related gaps in integration handling that surface together when managing Git repos.

Three commits, three problems

1) feat(git_repository) — first-class GitRepository resource

GIT REPOSITORY was listed in ResourceType but had no resource class, no priv enum, and no data_provider hooks (privs.py:392 mapped it to None). Users couldn't declare grants like priv: READ on git repository <fqn>, forcing raw-SQL bridges.

  • snowcap/resources/git_repository.pyGitRepository class (SchemaScope) with required origin + api_integration and optional git_credentials (secret) + comment + owner, mirroring CREATE GIT REPOSITORY syntax.
  • snowcap/privs.pyGitRepositoryPriv enum (READ, WRITE, OWNERSHIP); wires into both PRIVS_FOR_RESOURCE_TYPE and CREATE_PRIV_FOR_RESOURCE_TYPE.
  • snowcap/data_provider.pyfetch_git_repository (uses SHOW GIT REPOSITORIES) and list_git_repositories (via list_schema_scoped_resource). Auto-discovered by the existing fetch/list dispatch.
  • snowcap/resources/__init__.py — exports GitRepository.
  • Docs + unit tests + integration fetch test + JSON/SQL fixtures.

2) fix(grant)on: list parser for plain-object grants + multi-word types

The on: string parser only recognized single-word resource types (schema, database, warehouse), so on: git repository <fqn> — and incidentally on: api integration <fqn>, on: network rule <fqn>, on: image repository <fqn>, on: external access integration <fqn> — failed with Grant type not recognized. Fixed with a longest-match look-ahead across consecutive words.

Separately, the list form of on: was treated as multiple grants ONLY if every list item contained FUTURE or ALL. Plain-object lists like on: [warehouse FOO, warehouse BAR] or on: [git repository D.S.A, git repository D.S.B] were misinterpreted as a single multi-element grant spec, again hitting "Grant type not recognized." Fixed by inspecting the first element: form (1) keeps its 4-element semantics if on[0] is FUTURE/ALL; otherwise each item is its own complete on spec and gets queued via rest_of_ons.

3) feat: generic INTEGRATION grants + APIIntegration non-AWS subtypes

Surfaced while migrating GitHub-related grants:

  • ResourceType.INTEGRATION (umbrella) had no RESOURCE_SCOPES entry, no fetcher, no lister. on: integration <fqn> crashed with KeyError: ResourceType.INTEGRATION. Fixed: register AccountScope(); add fetch_integration (walks SHOW INTEGRATIONS) and list_integrations. No concrete Integration resource class — declarative management still goes through the specific subtypes (APIIntegration, CatalogIntegration, etc.). The umbrella exists solely to let on: integration <fqn> parse and resolve.
  • fetch_api_integration crashed with KeyError: 'api_aws_role_arn' on non-AWS-Gateway API integrations (GIT_HTTPS_API, AZURE_API_MANAGEMENT, GOOGLE_API_GATEWAY). Fixed: ApiProvider enum expanded, _APIIntegration has optional azure_tenant_id/azure_ad_application_id/google_audience, and api_aws_role_arn is now optional. fetch_api_integration uses .get() so absent fields fall back to None instead of crashing.

Example usage

api_integrations:
  - name: github_api_integration
    api_provider: GIT_HTTPS_API
    api_allowed_prefixes: ["https://github.com/some-org/"]
    enabled: true

git_repositories:
  - name: dbt_platform_repo
    database: db_dev
    schema: public
    origin: https://github.com/some-org/some-repo.git
    api_integration: github_api_integration
    git_credentials: github_secret
    comment: dbt platform repo

grants:
  - priv: READ
    on:
      - git repository db_dev.public.dbt_platform_repo
      - git repository db_prod.public.dbt_platform_repo
    to: dbt_runner
  - priv: USAGE
    on: integration github_api_integration   # umbrella also works
    to: dbt_developer

Tests

pytest tests/ --ignore=tests/integration1492 passed.

Integration tests (real Snowflake) require credentials; covered behind pytest.mark.requires_snowflake.

Notes for review

  • Out of scope intentionally: ALTER GIT REPOSITORY ... FETCH orchestration. Snowcap is state-shape; FETCH is a runtime op.
  • The git_credentials prop uses IdentifierProp (matches DESC GIT REPOSITORY returning a fully-qualified secret identifier).
  • Both (2) and (3) are independent improvements that I'd happily split out into separate PRs if you prefer; folding them here because each gap surfaced while exercising the GitRepository feature in real YAML.

🤖 Generated with Claude Code

usbrandon added 2 commits May 10, 2026 15:52
Adds first-class support for Snowflake GIT REPOSITORY objects, which model
externally hosted Git repositories (GitHub, GitLab, Bitbucket, etc.) registered
for use with Snowflake's Git integration.

Why: GIT REPOSITORY was previously listed in ResourceType but had no resource
class, no priv enum, and no data_provider hooks (privs.py:392 mapped it to None).
This left users unable to declare grants like `priv: READ on git repository <fqn>`,
forcing raw SQL bridges for any account that uses Snowflake's Git integration.

Added:
- snowcap/resources/git_repository.py — GitRepository class (SchemaScope) with
  required `origin` + `api_integration` and optional `git_credentials` (secret)
  + `comment` + `owner`, mirroring CREATE GIT REPOSITORY syntax.
- snowcap/privs.py — GitRepositoryPriv enum (READ, WRITE, OWNERSHIP); wires
  GIT_REPOSITORY into PRIVS_FOR_RESOURCE_TYPE and CREATE_PRIV_FOR_RESOURCE_TYPE.
- snowcap/data_provider.py — fetch_git_repository (SHOW GIT REPOSITORIES) and
  list_git_repositories (uses list_schema_scoped_resource).
- snowcap/resources/__init__.py — exports GitRepository.
- snowcap/resources/grant.py — fixes a pre-existing limitation in the on=
  string parser: multi-word resource types (GIT REPOSITORY, API INTEGRATION,
  NETWORK RULE, IMAGE REPOSITORY, EXTERNAL ACCESS INTEGRATION, …) were not
  recognized in `on: <type> <fqn>` form, only via `on_<type>:` kwarg. The parser
  now does a longest-match look-ahead across consecutive words.
- docs/resources/git_repository.md + mkdocs.yml nav entry.
- tests/test_resource_types.py — TestGitRepository (minimal + with credentials).
- tests/integration/data_provider/test_fetch_resource.py — end-to-end fetch test
  with an APIIntegration dependency.
- tests/fixtures/{json,sql}/git_repository.{json,sql} — fixture pair.

All 1482 non-integration tests pass.
Previously the `Grant.__init__` list-handling treated `on:` as multiple
grants ONLY if every list item contained the keyword FUTURE or ALL. Plain
object lists like `on: [warehouse FOO, warehouse BAR]` or
`on: [git repository D.S.A, git repository D.S.B]` were misinterpreted as
a single multi-element grant spec, hitting "Grant type not recognized".

Distinguish the two list forms by inspecting the FIRST element:

  (1) Single grant whose target is a multi-element spec, like
      ["FUTURE", "TABLES", "SCHEMA", "db.schema"] — first element is the
      keyword FUTURE or ALL.
  (2) Multiple grants where each list item is its own complete `on` spec —
      anything else.

Form (2) now correctly populates `rest_of_ons` and is expanded by
`process_shortcuts` (same mechanism already used for `priv:` lists).

Tests added in `tests/test_grant.py::TestGrantOnList` covering:
- list of plain warehouses
- mixed resource types (warehouse + database) in one list
- list of multi-word types (git repository)
- existing 4-element ["FUTURE", "TABLES", "SCHEMA", fqn] form unchanged
- existing all-FUTURE/all-ALL list expansion preserved

All 1482 non-integration tests pass.
usbrandon added a commit to usbrandon/snowcap that referenced this pull request May 10, 2026
…port

Two related fixes folded into the GitRepository PR since both surface when
managing Snowflake Git integrations (and they're how I hit these gaps).

## (1) Generic `ResourceType.INTEGRATION` grants

Snowflake's `GRANT USAGE ON INTEGRATION <name>` SQL is valid for any subtype.
Users naturally write `on: integration <fqn>` in YAML, but the umbrella
`ResourceType.INTEGRATION` had no entry in `RESOURCE_SCOPES`, no fetcher, and
no lister — so any such grant crashed with:

    KeyError: <ResourceType.INTEGRATION: 'INTEGRATION'>

Fix: register `RESOURCE_SCOPES[ResourceType.INTEGRATION] = AccountScope()`,
add `fetch_integration` (walks `SHOW INTEGRATIONS`) and `list_integrations`.
We do NOT expose a concrete `Integration` resource class — declarative
management still goes through the specific subtypes (APIIntegration,
CatalogIntegration, etc.). The umbrella exists solely to let
`on: integration <fqn>` parse and resolve.

## (2) APIIntegration: non-AWS api_provider subtypes

`fetch_api_integration` crashed with `KeyError: 'api_aws_role_arn'` when
DESC'ing API integrations whose `api_provider` is anything other than the
AWS gateway flavors — e.g., `GIT_HTTPS_API` (used by Snowflake git repos),
`AZURE_API_MANAGEMENT`, `GOOGLE_API_GATEWAY`.

Fix:
- Expanded `ApiProvider` enum to include `AZURE_API_MANAGEMENT`,
  `GOOGLE_API_GATEWAY`, `GIT_HTTPS_API` alongside the existing AWS values.
- `_APIIntegration.api_aws_role_arn` is now optional (defaults to None) and
  paired with optional `azure_tenant_id`, `azure_ad_application_id`,
  `google_audience` for the other providers.
- `fetch_api_integration` uses `properties.get()` so missing fields fall
  back to None instead of crashing.
- Added `Props` entries for the new fields so they round-trip through
  `CREATE/ALTER API INTEGRATION` SQL.

## Tests

- 4 new APIIntegration tests covering AWS / GIT_HTTPS_API / AZURE / GCP
  provider shapes
- 2 new TestGenericIntegrationGrants tests covering single + list-form
  `on: integration <fqn>` grants
- Updated `tests/fixtures/json/api_integration.json` to include the new
  optional fields (null) so identity tests still pass

All 1492 non-integration tests pass.

## Why fold into this PR (datacoves#17)

These gaps both surface when YAML references the Git-related integrations
that the GitRepository feature targets (every git repo depends on a GIT_HTTPS_API
api_integration). Reviewing them together avoids three round-trip PRs against
the same code paths.
@usbrandon usbrandon changed the title feat: add GitRepository resource + grant support feat(git_repository): add GitRepository resource + grant support May 10, 2026
usbrandon added 2 commits May 10, 2026 18:17
…rceType)

Snowflake's `GRANT USAGE ON INTEGRATION <name>` SQL is valid for any subtype
(API/CATALOG/EXTERNAL_ACCESS/NOTIFICATION/SECURITY/STORAGE). Users naturally
write `on: integration <fqn>` in YAML, but the umbrella `ResourceType.INTEGRATION`
had no `RESOURCE_SCOPES` entry, no fetcher, and no lister — so any such grant
crashed with:

    KeyError: <ResourceType.INTEGRATION: 'INTEGRATION'>

Fix:
- Register `RESOURCE_SCOPES[ResourceType.INTEGRATION] = AccountScope()`
  (`snowcap/resources/resource.py`).
- Add `fetch_integration` (walks `SHOW INTEGRATIONS`) and `list_integrations`
  to `snowcap/data_provider.py` so the existing fetch/list dispatch resolves.

No concrete `Integration` resource class. Declarative management still goes
through the specific subtypes (`APIIntegration`, `CatalogIntegration`, etc.) —
the umbrella exists solely to let `on: integration <fqn>` parse and resolve.

Tests: 2 new `TestGenericIntegrationGrants` cases covering single- and
list-form generic integration grants. All non-integration tests pass.
…e/GitHTTPS)

`fetch_api_integration` crashed with `KeyError: 'api_aws_role_arn'` when DESC'ing
API integrations whose `api_provider` is anything other than AWS Gateway flavors
— e.g., `GIT_HTTPS_API` (used by Snowflake's git repos), `AZURE_API_MANAGEMENT`,
`GOOGLE_API_GATEWAY`. The `_APIIntegration` dataclass also required
`api_aws_role_arn`, blocking instantiation for non-AWS providers.

Changes:
- `ApiProvider` enum expanded with `AZURE_API_MANAGEMENT`, `GOOGLE_API_GATEWAY`,
  and `GIT_HTTPS_API` alongside the existing AWS variants.
- `_APIIntegration` now has optional `azure_tenant_id` / `azure_ad_application_id`
  (Azure), `google_audience` (GCP), and `api_aws_role_arn` is no longer required.
- `APIIntegration.__init__` keeps `api_aws_role_arn` in its original positional
  slot (just optional with default None) so existing positional callers
  continue to work; new fields appended after the original positional set.
- New `Props` entries for the Azure/Google fields so they round-trip through
  `CREATE/ALTER API INTEGRATION` SQL.
- `fetch_api_integration` uses `properties.get()` for the provider-specific
  fields so missing keys return None instead of crashing.
- `tests/fixtures/json/api_integration.json` extended with null entries for the
  new optional fields so the identity test still passes.
- 3 new `TestAPIIntegration` cases covering GIT_HTTPS_API, AZURE_API_MANAGEMENT,
  and GOOGLE_API_GATEWAY shapes.
- Docs updated with a provider-vs-required-fields table + a YAML example for
  the GitHub case used by `GitRepository`.

All 1492 non-integration tests pass.
@usbrandon usbrandon force-pushed the feat/git-repository branch from eb2c1ae to 71984bf Compare May 10, 2026 23:18
@noel
Copy link
Copy Markdown
Contributor

noel commented May 11, 2026

Thanks for this contribution! This is a well-structured PR with comprehensive test coverage and documentation.

All tests pass (1492) and linting is clean. Merging now.

@noel noel merged commit 5142c99 into datacoves:main May 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants