Add support for GitHub private releases as plugin source #8944

patriziobruno · 2022-02-08T13:36:30Z

Description

This change adds support for downloading plugins from private GitHub releases. I added the change under the feature flag PULUMI_EXPERIMENTAL. It works as follows:
If the plugin can't be downloaded from Pulumi's public GitHub releases, an attempt is made to download from private GitHub releases using GitHub info and credentials passed as environment variables. To be compatible with GitHub Actions, env variable names come from official documentation.

GITHUB_REPO_OWNER
GITHUB_TOKEN
GITHUB_ACTOR
GITHUB_PERSONAL_ACCESS_TOKEN (this is not an official pipeline context var)
The implementation uses either GITHUB_TOKEN for service2service authentication or GITHUB_ACTOR + GITHUB_PERSONAL_ACCESS_TOKEN for HTTP basic authentication. It gets the requested releaseId from GitHub's Releases API and, if successful, downloads the release from the URL of the asset matching the expected file-name format pulumi-<type>-<name>-v<version>-<operatingSystem>-<arch>.tar.gz.

If this method fails, pulumi plugin download falls back on get.pulumi.com.

Before adding testing, I wanted to gather feedback from the maintainers on the implementation.

Fixes #8943

Checklist

I have added tests that prove my fix is effective or that my feature works

I have updated the CHANGELOG-PENDING file with my change

Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

github-actions · 2022-02-08T13:37:08Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

github-actions · 2022-02-08T16:29:08Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

CHANGELOG_PENDING.md

sdk/go/common/workspace/plugins.go

github-actions · 2022-02-09T15:55:12Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

patriziobruno · 2022-02-09T16:00:37Z

@Frassle I was wondering if it made sense to put this code under a different feature flag from PULUMI_EXPERIMENTAL. Maybe, PULUMI_EXPERIMENTAL_GH_PRIVATE_REL? To let users decide which experiments they want to use and which not?

github-actions · 2022-02-09T18:14:26Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

github-actions · 2022-02-09T18:16:40Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

…inResponse

Co-authored-by: Fraser Waters <frassle@gmail.com>

github-actions · 2022-02-10T11:02:08Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

Frassle · 2022-02-10T16:32:44Z

@Frassle I was wondering if it made sense to put this code under a different feature flag from PULUMI_EXPERIMENTAL. Maybe, PULUMI_EXPERIMENTAL_GH_PRIVATE_REL? To let users decide which experiments they want to use and which not?

We've discussed this internally before and decided to just stick to the simplicity of everything under one experimental flag for now. We don't expect to have things live as experimental for long.

github-actions · 2022-02-10T16:53:41Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

github-actions · 2022-02-10T17:07:39Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

github-actions · 2022-02-10T17:34:30Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

Frassle

This looks reasonable to me.

Frassle · 2022-02-11T09:54:13Z

/run-acceptance-tests

github-actions · 2022-02-11T09:54:45Z

Please view the results of the PR Build + Acceptance Tests Run Here

iwahbe

This looks good to me. LGTM

patriziobruno · 2022-02-11T15:40:41Z

Are there any additional steps before merging this?

Frassle · 2022-02-11T16:16:43Z

Spoken to @stack72 about this. We do want to make sure we can support your workflow @patriziobruno, but further thinking about this we're not sure the proposed download flow is one we should support.

Would it be acceptable to use the pluginDownloadURL to point to your private github release instead of the download flow trying to work that out from env[GITHUB_REPOSITORY_OWNER]. We'd then change the pluginDownloadURL using code to add github auth tokens (if and only if the url was for something at github.com) so it could auth to that custom url.

I'm happy to do the required code changes here if that would still work for your usecase. If that wouldn't work do share why, we'll think some more about what we can do.

patriziobruno · 2022-02-11T16:32:03Z

@Frassle @stack72 the problem about the pluginDownloadURL approach is that endpoints pointed by a browser_download_url don't accept GitHub Actions tokens, nor Basic authentication. Clients need to authenticate using a browser and the cookies that GitHub.com sends, which is not doable in a pipeline. Furthermore, once the user has authenticated, how would pulumi CLI access the authentication cookies? Unless GitHub.com adds OAuth tokens as a means to authenticate access to their browser_download_url, there's no way to download a private release without using the release API.

Frassle · 2022-02-11T16:57:52Z

Ah right I've followed through the REST calls this makes and I see what you mean there. pluginDownloadURL makes assumptions about the format of the url, github releases don't match that format even if you pre-looked up the api style url to use.

@stack72 I can't see a simple way to fit this into our pluginDownloadURL flow given the above. I think we need to give that feature a major rethink and for now accept the PR as is, with the expectation that we'll have to break all custom plugin flows at some point to tidy this up (4.0 or even 5.0 change).

mikhailshilkov · 2022-02-14T12:26:05Z

I think we need to give that feature a major rethink and for now accept the PR as is, with the expectation that we'll have to break all custom plugin flows at some point to tidy this up (4.0 or even 5.0 change).

This sounds somewhat scary to me... Will you leave this PR's flow behind the flag until 4.0/5.0? Otherwise that breaking change may be quite hard to pull in. It would be nice to understand what that future tidy-up may look like.

patriziobruno · 2022-02-14T12:43:30Z

@mikhailshilkov This workflow doesn't make any use of pluginDownloadURL, so it would be minimally affected by any breaking change. Should the new workflow introduce configuration to replace the environment variables used by the current code, backward compatibility may be kept with little effort by keep using the all of the env vars that are set by GitHub Actions and progressively deprecating the one that is not, GITHUB_PERSONAL_ACCESS_TOKEN.
example schema.yaml:

pluginDownload:
  workflow: privateGitHubRelease
  with:
    repoOwner: owner # defaults to env[GITHUB_REPO_OWNER]
    actor: github_username # defaults to env[GITHUB_ACTOR]
    personalAccessToken: /path/to/file/containing/token or env var name containing token or any other means that is not putting a plain text secret into schema.yaml # initially defaults to env[GITHUB_PERSONAL_ACCESS_TOKEN] for backward compatibility

I don't mean to give suggestions on how to change the plugin download workflow, just making an example that wouldn't break this code.

Frassle · 2022-02-14T13:02:00Z

Will you leave this PR's flow behind the flag until 4.0/5.0?

Yes I think we'd want to do that.

This workflow doesn't make any use of pluginDownloadURL, so it would be minimally affected by any breaking change

I should have been more clear here. I think pluginDownloadURL is not descriptive enough to support all the cases we want to support custom download from, it's not even descriptive enough to support github releases let alone anything more esoteric. So I think potential breaking change 1 is redesigning pluginDownloadURL so it can describe these cases, which might not be doable in a way that also maintains the current behaviour (it might be possible to do this change compatibly, needs more thinking).
Breaking change 2 is that once we can support github releases via some sort of pluginDownloadURL feature we would not want to have it run implicitly for every plugin download that this PR is adding, and so we would break this flow at somepoint.

It would be nice to understand what that future tidy-up may look like.

The example schema.yaml in @patriziobruno post looks something like what I expect we'd change to.

github-actions · 2022-02-16T10:54:18Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

github-actions · 2022-02-17T18:49:39Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project
/run-codegen - used to test the Pull Request against downstream codegen
/run-docs-gen - used to test the Pull Request against documentation generation
/run-containers - used to test the Pull Request against the containers tests

Frassle · 2022-02-17T21:20:13Z

I'm going to merge this in with the understanding that it will stay under EXPERIMENTAL until we can spend the engineering time to rework custom plugin downloads to support this properly. Some time after adding proper support for this via the custom plugin work we will remove this code flow, and your workflow will be broken if you haven't updated to use the supported custom plugin settings.

I have raised #9007 to track the re-design work.

mikhailshilkov requested review from stack72, Frassle and iwahbe February 9, 2022 14:36

Frassle reviewed Feb 9, 2022

View reviewed changes

CHANGELOG_PENDING.md Outdated Show resolved Hide resolved

sdk/go/common/workspace/plugins.go Show resolved Hide resolved

sdk/go/common/workspace/plugins.go Outdated Show resolved Hide resolved

patriziobruno and others added 4 commits February 10, 2022 11:01

Add support for GitHub private releases as plugin source

0c0c4bb

update CHANGELOG_PENDING.md

6e62ddc

fix: invert token and tokentype parms in call to authenticatedGetPlug…

7c39400

…inResponse

Update CHANGELOG_PENDING.md

7e42255

Co-authored-by: Fraser Waters <frassle@gmail.com>

patriziobruno force-pushed the patriziobruno/8943/private-github-rel branch from 179ccca to 7e42255 Compare February 10, 2022 11:01

refactor: more reusable/testable private github private release download

1297e34

lint: fix errors

2a6ae61

syntax: add missing newline

a3b05c8

patriziobruno requested a review from Frassle February 11, 2022 09:22

Frassle approved these changes Feb 11, 2022

View reviewed changes

iwahbe approved these changes Feb 11, 2022

View reviewed changes

Merge branch 'master' into patriziobruno/8943/private-github-rel

4640f4c

Merge branch 'master' into patriziobruno/8943/private-github-rel

443f6f4

Frassle mentioned this pull request Feb 17, 2022

Support custom plugins from other sources #9007

Closed

Frassle merged commit fd25e56 into pulumi:master Feb 17, 2022

patriziobruno deleted the patriziobruno/8943/private-github-rel branch February 17, 2022 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for GitHub private releases as plugin source #8944

Add support for GitHub private releases as plugin source #8944

patriziobruno commented Feb 8, 2022 •

edited

Loading

github-actions bot commented Feb 8, 2022

github-actions bot commented Feb 8, 2022

github-actions bot commented Feb 9, 2022

patriziobruno commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

github-actions bot commented Feb 10, 2022

Frassle commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

Frassle left a comment

Frassle commented Feb 11, 2022

github-actions bot commented Feb 11, 2022

iwahbe left a comment

patriziobruno commented Feb 11, 2022

Frassle commented Feb 11, 2022

patriziobruno commented Feb 11, 2022

Frassle commented Feb 11, 2022

mikhailshilkov commented Feb 14, 2022

patriziobruno commented Feb 14, 2022 •

edited

Loading

Frassle commented Feb 14, 2022 •

edited

Loading

github-actions bot commented Feb 16, 2022

github-actions bot commented Feb 17, 2022

Frassle commented Feb 17, 2022

Add support for GitHub private releases as plugin source #8944

Add support for GitHub private releases as plugin source #8944

Conversation

patriziobruno commented Feb 8, 2022 • edited Loading

Description

Checklist

github-actions bot commented Feb 8, 2022

github-actions bot commented Feb 8, 2022

github-actions bot commented Feb 9, 2022

patriziobruno commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

github-actions bot commented Feb 10, 2022

Frassle commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

github-actions bot commented Feb 10, 2022

Frassle left a comment

Choose a reason for hiding this comment

Frassle commented Feb 11, 2022

github-actions bot commented Feb 11, 2022

iwahbe left a comment

Choose a reason for hiding this comment

patriziobruno commented Feb 11, 2022

Frassle commented Feb 11, 2022

patriziobruno commented Feb 11, 2022

Frassle commented Feb 11, 2022

mikhailshilkov commented Feb 14, 2022

patriziobruno commented Feb 14, 2022 • edited Loading

Frassle commented Feb 14, 2022 • edited Loading

github-actions bot commented Feb 16, 2022

github-actions bot commented Feb 17, 2022

Frassle commented Feb 17, 2022

patriziobruno commented Feb 8, 2022 •

edited

Loading

patriziobruno commented Feb 14, 2022 •

edited

Loading

Frassle commented Feb 14, 2022 •

edited

Loading