Rebuild templating using Go text templates #446
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
iwahbe
reviewed
Jun 6, 2023
There was a problem hiding this comment.
I like moving from custom generation code to go templates. What's the plan for rolling this out? I'm assuming we keep the old behavior (and all the old code) for team providers?
Will we flip a switch somewhere to make sure that we only build our team's actions with the new generator?
iwahbe
reviewed
Jun 6, 2023
There was a problem hiding this comment.
I like moving from custom generation code to go templates. What's the plan for rolling this out? I'm assuming we keep the old behavior (and all the old code) for team providers?
Will we flip a switch somewhere to make sure that we only build our team's actions with the new generator?
package-ci/pkg/templates/bridged-provider/.github/workflows/update-bridge.yml
Outdated
Show resolved
Hide resolved
t0yv0
reviewed
Jun 6, 2023
|
I'm in favor, the devil though is in ensuring uneventful rollout. If the new system still materializes all the workflow files and there's a zero diff (it's byte-for-byte the same) then it'd be excellent we can just swap the generator out without projects noticing. |
AaronFriel
force-pushed
the
friel/package-ci
branch
from
50b84cc
to
1cb869d
Compare
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
This commit deletes the provider CI folder and replaces it with package-ci, applying #446, updating the Makefile to conform to the same interface. Two shared TypeScript files are moved to native-provider-ci: * provider-ci/src/shared-workflows.ts * provider-ci/src/action-versions.ts
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
This commit deletes the provider CI folder and replaces it with package-ci, applying #446, updating the Makefile to conform to the same interface. Two shared TypeScript files are moved to native-provider-ci: * provider-ci/src/shared-workflows.ts * provider-ci/src/action-versions.ts
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
This commit deletes the provider CI folder and replaces it with package-ci, implementing #446, updating the Makefile to conform to the same interface. Two shared TypeScript files are moved to native-provider-ci: * provider-ci/src/shared-workflows.ts * provider-ci/src/action-versions.ts
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
This commit deletes the provider CI folder and replaces it with package-ci, implementing #446, updating the Makefile to conform to the same interface. Two shared TypeScript files are moved to native-provider-ci: * provider-ci/src/shared-workflows.ts * provider-ci/src/action-versions.ts
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
Builds on #446 and #450 by adding linting. The Makefile is updated to make for a fast inner dev loop on changing templates. Using a tool such as `watchexec`, we can run: ```sh watchexec -i providers -- NAME=databricks make lint-provider ``` When running in a loop like this, `watchexec` should trigger on changes to templates, rebuild `bin/provider-ci`, and regenerate the output. When changing config files, the `provider-ci` binary should remain unchanged: ``` $ make bin/provider-ci make: 'bin/provider-ci' is up to date. ``` The `provider` and `providers` targets now run linting by default.
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
Builds on #446 and #450 by adding linting. The Makefile is updated to make for a fast inner dev loop on changing templates. Using a tool such as `watchexec`, we can run: ```sh watchexec -i providers -- NAME=databricks make lint-provider ``` When running in a loop like this, `watchexec` should trigger on changes to templates, rebuild `bin/provider-ci`, and regenerate the output. When changing config files, the `provider-ci` binary should remain unchanged: ``` $ make bin/provider-ci make: 'bin/provider-ci' is up to date. ``` The `provider` and `providers` targets now run linting by default.
AaronFriel
added a commit
that referenced
this pull request
Jun 8, 2023
Builds on #446 and #450 by adding linting. The Makefile is updated to make for a fast inner dev loop on changing templates. Using a tool such as `watchexec`, we can run: ```sh watchexec -i providers -- NAME=databricks make lint-provider ``` When running in a loop like this, `watchexec` should trigger on changes to templates, rebuild `bin/provider-ci`, and regenerate the output. When changing config files, the `provider-ci` binary should remain unchanged: ``` $ make bin/provider-ci make: 'bin/provider-ci' is up to date. ``` The `provider` and `providers` targets now run linting by default.
guineveresaenger
approved these changes
Jun 9, 2023
There was a problem hiding this comment.
Since @kpitzen and team cleared this change, let's do it.
Do we want to include instructions for running the /scripts/test.sh as part of development? Or should we remove it, as it was part of the PoC?
Seeking to resolve the CI disparity between ecosystem managed repositories and pulumi/pulumi-std#29, it quickly became apparent that the style of GitHub Actions - using reusable workflows - in that repository and in pulumi-command would require a major rewrite of the templating in this repository. Making large changes to the templating in this repository is more difficult as it requires modifying the Node scripts and the type-safe "generator" to reverse engineer CI workflows into a known form. Directly generating text templates, we skip the intermediate steps and can more naturally write the intended GitHub Actions YAML. The Go utility in this repository can be run and a diff with the current repository scripts generated via: ``` cd ./package-ci ./scripts/test.sh ``` The committed template, "bridged-provider", is intended to be bit-for-bit identical in output to the current scripts to the extent possible. As a result, there is no semantic diff with the current output. The two cases where the output was not identical are in the New Relic provider, and the diff is inconsequential wrapping behavior in the current Node scripts that is not present in the Go text template, both appearing in `.goreleaser` files. In this case, the wrapping has no effect on Goreleaser: ```diff diff --git a/../provider-ci/providers/newrelic/repo/.goreleaser.prerelease.yml b/./test-output/providers/newrelic/repo/.goreleaser.prerelease.yml index 439adbf7a..a5b009761 100644 --- a/../provider-ci/providers/newrelic/repo/.goreleaser.prerelease.yml +++ b/./test-output/providers/newrelic/repo/.goreleaser.prerelease.yml @@ -29,8 +29,7 @@ builds: ignore: [] ldflags: - -X github.com/pulumi/pulumi-newrelic/provider/v5/pkg/version.Version={{.Tag}} - - -X - github.com/newrelic/terraform-provider-newrelic/v2/main.UserAgentServiceName=pulumi + - -X github.com/newrelic/terraform-provider-newrelic/v2/main.UserAgentServiceName=pulumi main: ./cmd/pulumi-resource-newrelic/ ```
AaronFriel
force-pushed
the
friel/package-ci
branch
from
100e331
to
2604875
Compare
AaronFriel
added a commit
that referenced
this pull request
Jun 9, 2023
This commit deletes the provider CI folder and replaces it with package-ci, implementing #446, updating the Makefile to conform to the same interface. Two shared TypeScript files are moved to native-provider-ci: * provider-ci/src/shared-workflows.ts * provider-ci/src/action-versions.ts
AaronFriel
added a commit
that referenced
this pull request
Jun 9, 2023
Builds on #446 and #450 by adding linting. The Makefile is updated to make for a fast inner dev loop on changing templates. Using a tool such as `watchexec`, we can run: ```sh watchexec -i providers -- NAME=databricks make lint-provider ``` When running in a loop like this, `watchexec` should trigger on changes to templates, rebuild `bin/provider-ci`, and regenerate the output. When changing config files, the `provider-ci` binary should remain unchanged: ``` $ make bin/provider-ci make: 'bin/provider-ci' is up to date. ``` The `provider` and `providers` targets now run linting by default.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebuild templating using Go text templates
Seeking to resolve the CI disparity between ecosystem managed repositories and pulumi/pulumi-std#29, it quickly became apparent that the style of GitHub Actions - using reusable workflows - in that repository and in pulumi-command would require a major rewrite of the templating in this repository.
Making large changes to the templating in this repository is more difficult as it requires modifying the Node scripts and the type-safe "generator" to reverse engineer CI workflows into a known form.
Directly generating text templates, we skip the intermediate steps and can more naturally write the intended GitHub Actions YAML.
The Go utility in this repository can be run and a diff with the current repository scripts generated via:
The committed template, "bridged-provider", is intended to be bit-for-bit identical in output to the current scripts to the extent possible. As a result, there is no semantic diff with the current output.
The two cases where the output was not identical are in the New Relic provider, and the diff is inconsequential wrapping behavior in the current Node scripts that is not present in the Go text template, both appearing in
.goreleaserfiles. In this case, the wrapping has no effect on Goreleaser: