Seeing sporadic 403s when action tries to install plugins

[jbrunton]

What happened?

I'm using this action to automate deployments in a few projects. In recent days I've started seeing inconsistent failures when pulumi tries to install plugins (especially the synced-folder plugin, but I'm unsure if it's specific to that).

The error looks like this, apparently caused by a 403 when trying to download the plugin:

  error: could not load plugin for synced-folder provider 'urn:pulumi:staging::auth0-test::pulumi:providers:synced-folder::default_0_0_9': Could not automatically download and install resource plugin 'pulumi-resource-synced-folder' at version v0.0.9, install the plugin using `pulumi plugin install resource synced-folder v0.0.9`.
  Underlying error: error downloading plugin synced-folder to file: failed to download plugin: synced-folder-0.0.9: 403 HTTP error fetching plugin from https://get.pulumi.com/releases/plugins/pulumi-resource-synced-folder-v0.0.9-linux-amd64.tar.gz

However, the behaviour is inconsistent, and a re-run of the workflow usually fixes it.

Steps to reproduce

It doesn't happen all the time so it may be difficult to provide a reproducible example – though I'm happy to try to create a simplified app if it's helpful – but it is happening pretty regularly. You can see an example here:

https://github.com/jbrunton/auth0-test/actions/runs/3802920177/jobs/6468891986

You can also see that attempt #2 was successful:

https://github.com/jbrunton/auth0-test/actions/runs/3802920177/jobs/6468979717

That is, the code works as is – it just occasional fails.

Given the nondeterminism, I'm wondering if I might be encountering rate limiting? That's just a guess, however.

Expected Behavior

For the app to be provisioned.

Actual Behavior

The provisioning fails and I have to re-run the workflow.

Output of pulumi about

CLI
Version      3.50.2
Go Version   go1.19.4
Go Compiler  gc

Plugins
NAME           VERSION
aws            5.24.0
docker         3.6.1
nodejs         unknown
random         4.8.2
synced-folder  0.0.9

Host
OS       darwin
Version  13.1
Arch     arm64

This project is written in nodejs: executable='/Users/john/.nvm/versions/node/v16.19.0/bin/node' version='v16.19.0'

Current Stack: jbrunton/auth0-test/dev

TYPE                                               URN
pulumi:pulumi:Stack                                urn:pulumi:dev::auth0-test::pulumi:pulumi:Stack::auth0-test-dev
pulumi:providers:pulumi                            urn:pulumi:dev::auth0-test::pulumi:providers:pulumi::default
pulumi:providers:aws                               urn:pulumi:dev::auth0-test::pulumi:providers:aws::aws
pulumi:pulumi:StackReference                       urn:pulumi:dev::auth0-test::pulumi:pulumi:StackReference::jbrunton/jbrunton-aws.com-infra/prod
pulumi:providers:aws                               urn:pulumi:dev::auth0-test::pulumi:providers:aws::default_5_24_0
aws:ecs/cluster:Cluster                            urn:pulumi:dev::auth0-test::aws:ecs/cluster:Cluster::auth0-test-dev
aws:cloudwatch/logGroup:LogGroup                   urn:pulumi:dev::auth0-test::aws:cloudwatch/logGroup:LogGroup::/ecs/auth0-test-dev
aws:ec2/securityGroup:SecurityGroup                urn:pulumi:dev::auth0-test::aws:ec2/securityGroup:SecurityGroup::auth0-test-dev
aws:dynamodb/table:Table                           urn:pulumi:dev::auth0-test::aws:dynamodb/table:Table::auth0-test-dev
aws:lb/targetGroup:TargetGroup                     urn:pulumi:dev::auth0-test::aws:lb/targetGroup:TargetGroup::auth0-test-dev
aws:lb/listenerRule:ListenerRule                   urn:pulumi:dev::auth0-test::aws:lb/listenerRule:ListenerRule::auth0-test-dev
aws:route53/record:Record                          urn:pulumi:dev::auth0-test::aws:route53/record:Record::auth0-test-dev
aws:iam/role:Role                                  urn:pulumi:dev::auth0-test::aws:iam/role:Role::auth0-test-dev-exec-role
aws:iam/role:Role                                  urn:pulumi:dev::auth0-test::aws:iam/role:Role::auth0-test-dev-role
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::auth0-test::aws:iam/rolePolicyAttachment:RolePolicyAttachment::auth0-test-dev-exec-role-attachment
aws:iam/rolePolicy:RolePolicy                      urn:pulumi:dev::auth0-test::aws:iam/rolePolicy:RolePolicy::auth0-test-dev-get-params
aws:s3/bucket:Bucket                               urn:pulumi:dev::auth0-test::aws:s3/bucket:Bucket::auth0-test-dev-bucket
aws:iam/rolePolicy:RolePolicy                      urn:pulumi:dev::auth0-test::aws:iam/rolePolicy:RolePolicy::auth0-test-dev-dynamo-db
aws:ecs/taskDefinition:TaskDefinition              urn:pulumi:dev::auth0-test::aws:ecs/taskDefinition:TaskDefinition::auth0-test-dev
aws:iam/rolePolicyAttachment:RolePolicyAttachment  urn:pulumi:dev::auth0-test::aws:iam/rolePolicyAttachment:RolePolicyAttachment::auth0-test-dev-role-attachment
pulumi:providers:synced-folder                     urn:pulumi:dev::auth0-test::pulumi:providers:synced-folder::default_0_0_9
synced-folder:index:S3BucketFolder                 urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder::auth0-test-dev-folder
pulumi:providers:aws                               urn:pulumi:dev::auth0-test::pulumi:providers:aws::default_5_23_0
aws:ecs/service:Service                            urn:pulumi:dev::auth0-test::aws:ecs/service:Service::auth0-test-dev
aws:s3/bucketObject:BucketObject                   urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder$aws:s3/bucketObject:BucketObject::assets/index-3fce1f81.css
aws:s3/bucketObject:BucketObject                   urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder$aws:s3/bucketObject:BucketObject::assets/react-35ef61ed.svg
aws:s3/bucketObject:BucketObject                   urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder$aws:s3/bucketObject:BucketObject::index.html
aws:s3/bucketObject:BucketObject                   urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder$aws:s3/bucketObject:BucketObject::vite.svg
aws:s3/bucketObject:BucketObject                   urn:pulumi:dev::auth0-test::synced-folder:index:S3BucketFolder$aws:s3/bucketObject:BucketObject::assets/index-32c211af.js
aws:cloudfront/distribution:Distribution           urn:pulumi:dev::auth0-test::aws:cloudfront/distribution:Distribution::auth0-test-dev-cdn
aws:route53/record:Record                          urn:pulumi:dev::auth0-test::aws:route53/record:Record::auth0-test-dev.dev.jbrunton-aws.com


Found no pending operations associated with dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/jbrunton
User           jbrunton
Organizations  jbrunton

Dependencies:
NAME                              VERSION
eslint                            8.30.0
prettier                          2.8.1
ts-node                           10.9.1
@pulumi/aws                       5.24.0
@pulumi/pulumi                    3.49.0
@pulumi/random                    4.8.2
@typescript-eslint/parser         5.46.1
eslint-config-prettier            8.5.0
date-fns                          2.29.3
jest-mock-extended                3.0.1
@pulumi/awsx                      0.40.1
@types/jest                       29.2.4
@types/node                       16.18.10
jest                              29.3.1
tsconfig-paths                    4.1.1
@pulumi/synced-folder             0.0.9
@typescript-eslint/eslint-plugin  5.46.1
eslint-plugin-prettier            4.2.1
ts-jest                           29.0.3

Additional context

I noticed a couple of open issues related to plugins, but I think the error I'm seeing is different?

EDIT: See also my comment below regarding the open PRs related to this issue.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[jbrunton]

Oh, I checked the open issues but not the PRs. Looks like #810 and #812 were created to address this.

[jbrunton]

I note per this comment that the problem seems to be GitHub's rate limiter? If so, I'm not a fan of the proposed solutions, both of which require maintaining separate lists of the plugins used (or even hosting the artifacts independently if I'm reading them right?) This doesn't seem ideal: since I use the Pulumi action in a few different workflows, I would have to maintain the list of plugins in multiple places.

I'm guessing from the described behaviour and proposed solutions that Pulumi is trying to download artifacts from GitHub without authenticating, and thus all users of this action are sharing the same rate limits? If so, a better fix would be for Pulumi to do one of the following:

1. Detect the presence of a GITHUB_TOKEN secret and use that to authenticate if present. This action could then ensure that is correctly set.
2. Accept a GitHub token as a parameter, so that this action can provide one if set.

Unless I misunderstood the problem, I think doing something like this would provide a more robust experience for all users of this action.

[philprime]

I agree with using authentication if possible, but AFAIK this would require a change in the pulumi core as it would require to allow headers set in the plugin installation command (please correct me if I am wrong).

[jbrunton]

AFAIK this would require a change in the pulumi core

@philprime: I assume so. In fact, I see there's a related open issue in the core Pulumi repo: pulumi/pulumi#11743

I don't think I have the time or Go experience to contribute to that, so I hope the Pulumi maintainers are able to address this. I really like what Pulumi offers but it's becoming too unreliable when running in GitHub to make it a viable product for me atm.

[jbrunton]

Actually, I see that the githubSource plugin class does appear to use the GITHUB_TOKEN (link).

However, I suspect this action isn't correctly setting a GITHUB_TOKEN secret. I'm not 100% certain, but:

1. This plugin has a github-token parameter for authoring PR comments, but doesn't have the option to specify a GITHUB_TOKEN secret.
2. AFAIK actions don't inherit the GITHUB_TOKEN from the workflow.

The fix for this action may, therefore, be as simple as adding a GITHUB_TOKEN secret. I'd be happy to contribute a fix for that, but may not have time until next weekend.

EDIT: Actually, the fix may be as simple as doing this when using the action, since one can set any environment variables for an action:

      - uses: pulumi/actions@v3
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        ...

I'm testing this now, but if this works then it would be helpful to update the README about this.

EDIT: It did work. Opened a PR to update the readme.

[1oglop1]

I partially fixed this by cashing the plugins directory, but as @jbrunton says GITHUB_TOKEN would be better together with the cache combination.

      - uses: actions/cache@v3
        id: pulumi-cache
        with:
          path: "~/.pulumi/plugins"
          key: ${{ runner.os }}-pulumi-plugins-${{ hashFiles('yarn.lock') }}

[thomas11]

Likely the same root cause as pulumi/pulumi-metabase#3.

[RobbieMcKinstry]

I'm convinced @JoseAlban 's suggestion of caching plugins is the way to go, in addition to providing guidance around GITHUB_TOKEN. I'm planning to take some time next week to spike on adding caching.

[jbrunton]

@RobbieMcKinstry: Thanks.

Btw, a note from a related issue: apparently setting a default GitHub token doesn't work with GitHub Enterprise – see the comment from @Moon1706: pulumi/pulumi#11743 (comment)

I assume that's because GitHub Enterprise instances don't share any tokens with vanilla GitHub. If so, this might slightly complicate things, since you might need two tokens (one for downloading plugins, and another for commenting on internal PRs). There are also the discussions in other PRs in this repo about using self-hosted plugins (e.g. #812).