Skip to content

feat: generate terraform-plugin-framework providers from a blueprint - #7

Merged
ShocOne merged 1 commit into
mainfrom
feat/provider-codegen-walking-skeleton
Jul 27, 2026
Merged

feat: generate terraform-plugin-framework providers from a blueprint#7
ShocOne merged 1 commit into
mainfrom
feat/provider-codegen-walking-skeleton

Conversation

@ShocOne

@ShocOne ShocOne commented Jul 27, 2026

Copy link
Copy Markdown
Member

Pull Request Description

Summary

Bootstraps this repository from the template scaffold and lands a walking skeleton: a blueprint goes in, a compiling and terraform plan-able provider comes out.

$ tfprovidergen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes
$ cd pilot/thousandeyes && go build ./... && go test ./... && terraform plan
  + resource "thousandeyes_tag" "example" { ... }
Plan: 1 to add, 0 to change, 0 to destroy.

Issue Reference

No issue; this is the first implementation phase of the agreed plan. The charter line is in share/engineering_culture/Forge.md: "Evaluate code generation tools … for schema-first development and reducing boilerplate."

Motivation and Context

An OpenAPI document records what an API's fields are called. It does not record which are writable, which are immutable, what the server normalises on the way in, or what it defaults when a field is omitted. Those are the facts that decide whether a Terraform provider actually works.

Providers built on the specification alone therefore accumulate those facts as hand-maintained special cases, discovered one production bug at a time. terraform-provider-thousandeyes does exactly this, in a 1,165-line runtime reflection engine (thousandeyes/util.go) whose special cases — sensitiveFields, emptyStringToNilTypes, preserveNestedSensitiveFields, resourceFixups — are a catalogue of precisely that.

terraform-provider-microsoft365 solved it properly, with ~400 resources on a rigid six-file archetype and zero runtime reflection. But all 400 were hand-written by copy-pasting _resource_template/.

Meanwhile go-sdk-thousandeyes already proves this org can do industrial codegen: 97 service packages and ~90k lines from an OpenAPI snapshot, with a CI drift gate. The SDK layer is generated and the provider layer is hand-written. This closes that gap.

Later phases derive the missing behaviour empirically by probing a live API and committing the HTTP transcripts as evidence.

Why not HashiCorp's toolchain: terraform-plugin-codegen-spec v0.2.0, -framework v0.4.1 and -openapi v0.3.0 have had no functional commits since September 2024, generate no CRUD logic at all, and cannot express dynamic, int32/float32, blocks, resource identity or write-only attributes. Every renderer in -framework lives under internal/, so there is nothing to import. Their Provider Code Specification is adopted as an interop format (phase 3), not as the model.

Dependencies

  • Root module: mvdan.cc/gofumpt v0.9.1 only. It deliberately does not depend on terraform-plugin-framework — the toolkit emits text, and depending on the framework would couple every consumer to one framework version.
  • pilot/thousandeyes is a nested module pinning framework v1.19.0, plugin-testing v1.16.0 and httpmock v1.4.1, matching the ms365 provider exactly so nobody reconciles two framework versions across the org. It pins go-sdk-thousandeyes v0.1.0, which resolves from the proxy — no replace needed.
  • No go.work; it breaks dependabot's per-directory resolution. CI enters the nested module explicitly, which is also the honest simulation of a downstream consumer.
  • go 1.25.0, matching the sibling repos rather than the newer local toolchain.

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)
  • 🔧 Configuration change

Nothing here is breaking: the repository previously contained no Go code.

Testing

  • I have added unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this code in the following environments: macOS arm64, Go 1.25/1.26, Terraform v1.14.3

267 tests across both modules. The ones worth reviewing:

Test Why it exists
TestUnit_Emit_IsDeterministic Runs 25 times, not twice. Map iteration order varies per iteration, so two runs can agree by luck. Any non-determinism makes verify fail on a run that changed nothing, which destroys the whole drift design.
TestUnit_Emit_CarriesNoTimestampOrVersion Guards the two values somebody will later add trying to be helpful.
TestUnit_Naming_TerraformName Acronym runs are where every camel-to-snake implementation breaks. Seeded from real ThousandEyes property names plus the hard shapes (HTTPProxy, iOSVersion, ipv6Address).
TestUnit_Naming_TerraformNameAndGoFieldName_AgreeOnWordBoundaries A disagreement here produces a model field whose tfsdk tag silently does not match its schema attribute.
TestUnit_Convert_NilPointerFlattensToNull Flattening an absent field to the zero value is the single most common cause of a permanent diff.
TestUnit_Convert_EnumUsesRawWireValue The SDK's String() renders an unlisted value as AccessType(raw) for logging; putting that in state would corrupt it.
TestUnit_Emit_RefusesToOverwriteHandWrittenFiles Stops a mistyped -out destroying work with no recovery.
TestUnit_CLI_Dispatch_ImplementationClaimsAreTrue Bidirectional: an unbuilt subcommand must not silently succeed, and a built one must not still report "not implemented".

Manually verified, not assumed:

  • Both modules gofumpt-clean, so emitted code is a formatter fixed point — this matters because the target repos autofix with gofumpt/gci/golines, and merely gofmt-clean output gets rewritten on first open and then reads as drift with no source change.
  • Regenerating reports 0 written, 7 unchanged.
  • The drift gate has teeth, proven by breaking it: editing a generated file → verify exits 1 with a $GITHUB_STEP_SUMMARY diffstat; deleting one → reported as missing rather than drifted; re-emitting restores both.
  • terraform plan against a dev-override build renders the resource with zero errors. It needs no credentials: a plan for a new resource makes no API calls.

Quality Checklist

  • I have reviewed my own code before requesting review
  • I have verified there are no other open Pull Requests for the same update/change
  • All CI/CD pipelines pass without errors or warnings — first run on this PR; nothing has exercised these workflows before
  • My code follows the established style guidelines of this project
  • My comments are used only when necessary
  • I have added necessary documentation
  • I have made corresponding changes to the README and other relevant documentation
  • My changes generate no new warnings

Additional Notes

What to actually review

The emitted output, not the emitter. pilot/thousandeyes/internal/services/resources/tags/v7/tag/state.go is the file that replaces runtime reflection — 15 direct assignments, each a compile error if wrong. Then internal/templates/*.tmpl, which is the emitted shape as ordinary reviewable text.

Design choices worth disagreeing with

  • cmd/ rather than the house scripts/<Area>/<Command>/main.go. Mechanical reason: .golangci.yml excludes scripts/.* and only rescues internal/.*\.go$, so under scripts/ the CLI's flag validation and exit codes would be permanently unlinted. Also one import path for downstream repos to pin instead of six. What is kept is the substantive part — stdlib flag, no cobra.
  • Registration files are generated whole, not patched. The ms365 equivalent is 168 hand-maintained aliased imports. Owning the whole file also lets the drift check police it, which a partially-generated file makes impossible.
  • Enums generate documented values but no stringvalidator.OneOf. The SDK's enums are open by design; a validator would turn a routine upstream addition into a plan failure — reintroducing exactly the fragility the SDK avoided.
  • updateStyle is mandatory when a resource has an update operation. UpdateTag is PUT, so an omitted field is cleared. Guessing this wrong silently erases attributes the practitioner never mentioned, so the blueprint refuses to be ambiguous about it.

Two judgement calls in the tag blueprint that need probing

Both are recorded in the blueprint's own descriptions rather than hidden, and phase 4 settles them:

  • color, access_type and match_type are computed_optional on the assumption the API assigns defaults. If it does not, those attributes show (known after apply) unnecessarily.
  • legacy_id is float64 because the specification says number, though observed values are integral. A wrong type in a published schema is breaking to fix.

A real bug this caught, worth acting on

I wrote a binding against the local go-sdk-thousandeyes working tree, which is ahead of the tag. At the pinned v0.1.0 the client is Client{Transport, API *API} with services on API, so r.client.Tags does not exist and the accessor is r.client.API.Tags. Only the compile gate caught it, as four identical errors.

This is plan risk #10 ("the pilot's SDK is a moving target") arriving on day one, and it argues for pulling sdkbind forward — an AST scan of the pinned module would have said "CreateTag not found on *thousandeyes.Client" at blueprint time instead.

Known gaps in this PR

  • No codegen-verify.yml yet. verify works and is proven to fail correctly, but nothing runs it in CI, so drift is currently only caught locally. It is the SDK repo's workflow with the remediation command swapped and an added cd pilot/thousandeyes.
  • dependabot.yml has no /pilot/thousandeyes entry, so the pilot's framework pins will rot unwatched.
  • SECURITY.md still points at GitHub's own bug bounty (inherited from the template, shared across your repos) — left alone deliberately, since changing a house-wide file is not mine to decide.

Deferred, with reasons

Phase 2 OpenAPI ingestion · 3 codegen-spec interop · 4 the prober · 5 generated tests and mocks · 6 breadth and docs. The anchor region-patcher is unnecessary while the pilot is greenfield; it is only needed to adopt the toolkit into ms365 or jamfpro, which is phase 7.

🤖 Generated with Claude Code

Bootstraps the toolkit and lands a walking skeleton that takes a blueprint
through to a compiling, plan-able provider.

The problem: an OpenAPI document records what an API's fields are called, not
which are writable, which are immutable, what the server normalises, or what it
defaults. Providers built on the specification alone therefore carry those facts
as hand-maintained special cases discovered one production bug at a time --
terraform-provider-thousandeyes does exactly this, in 1,165 lines of runtime
reflection. This toolkit generates that mapping instead, and later phases derive
the missing behaviour by probing a live API and committing the transcripts.

What works end to end:

    tfprovidergen emit -blueprint blueprints/thousandeyes -out pilot/thousandeyes
    cd pilot/thousandeyes && go build ./... && go test ./... && terraform plan
    #  + resource "thousandeyes_tag" "example"
    #  Plan: 1 to add, 0 to change, 0 to destroy.

Structure:

- internal/blueprint  the IR. A superset of HashiCorp's Provider Code
  Specification, which cannot express CRUD wiring, SDK binding, observed
  behaviour or test scaffolding -- most of what a working provider is.
- internal/render     all the logic. Every value a template consumes is a
  finished string, so templates branch on presence and never on meaning.
- internal/templates  the emitted shape as reviewable text, per house
  convention, embedded in its own package.
- internal/emit       plan, format with gofumpt, refuse to overwrite files the
  tool does not own.
- pilot/thousandeyes  a nested module holding a real generated provider, built
  and tested in CI. Deliberately a separate module: the toolkit emits text and
  must never depend on terraform-plugin-framework itself.

Deliberate choices worth knowing:

- Generated files carry no timestamp and no tool version. Either would make
  every regeneration a diff and destroy the drift check, which is the only
  thing keeping committed output honest.
- Enums generate documented values but no OneOf validator. The SDK's enums are
  open by design; a validator would turn a routine upstream addition into a
  plan failure.
- Absent fields flatten to null rather than the zero value, which is the usual
  cause of a provider with a permanent diff.
- gofumpt rather than go/format, because the target repositories autofix with
  gofumpt, gci and golines; merely gofmt-clean output gets rewritten on first
  open and then reads as drift with no source change.

Verified: 267 tests across both modules; both gofumpt-clean, so emitted code is
a formatter fixed point. Regenerating reports 0 written, 7 unchanged. Editing a
generated file makes verify exit 1 with a step-summary diffstat; deleting one is
reported as missing rather than drifted.

Deferred with reasons recorded in the task list: OpenAPI ingestion (phase 2),
codegen-spec interop (3), the prober (4), generated tests and mocks (5). The
anchor region-patcher is unnecessary while the pilot is greenfield, since
registration files are generated whole. sdkbind should come next: a binding was
written against the SDK working tree rather than the pinned v0.1.0, where the
services hang off Client.API rather than Client, and only the compile gate
caught it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ShocOne
ShocOne merged commit cd2f437 into main Jul 27, 2026
5 of 6 checks passed
ShocOne added a commit that referenced this pull request Aug 5, 2026
…oards-wave

fix+feat(kiota): schema-default stripping (run #7's five 400s) + the final wave — full 23-resource parity
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant