Release of jenkins-contribution-* GO applications fail: homebrew token not found

[jmMeessen]

Service(s)

GitHub

Summary

The GoReleaser based release process updates a repository (jenkins-infra/homebrew) to make the new release available via HomeBrew. With the migration of the tools from "jmMeessen" to "jenkins-infra" this process doesn't work anymore.

The error message is

Error: Could not create installation access token.
(...)
  [cause]: Error: Could not retrieve installation.
(...)
    [cause]: RequestError [HttpError]: Not Found

This process is handled by the "release" GitHub action. It sets up on-the-fly a token via a GitHub app to allow the GHA to update another repository in the Org (the "homebrew" repository) via a commit. In this case, the failure comes the local secrets to access the app are not set.

This is the code where the token is generated (release.yml#L19-24):

    - uses: tibdex/github-app-token@v2
      id: generate_homebrew_token
      with:
        app_id: ${{ secrets.HOMEBREW_APP_ID }}
        private_key: ${{ secrets.HOMEBREW_APP_PRIVKEY }}

and this is how the token is retrieved and used in the release step (release.yml#L24-L31):

    - name: Release via goreleaser
      uses: goreleaser/goreleaser-action@v5
      with:
        distribution: goreleaser
        args: release
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        HOMEBREW: ${{ steps.generate_homebrew_token.outputs.token }}

As far as I remember, the solution is to install an application that has update/commit access in jenkins-infra/homebrew repository and to add the APP_ID and the APP_PRIVKEY as secrets in the repository.

Reproduction steps

No response

[lemeurherve]

For reference, from #4017 (comment):

[...] the first two repositories have to be configured to let goreleaser publishings corresponding taps to https://github.com/jenkins-infra/homebrew-tap.

Like what's done in https://github.com/jenkins-infra/jenkins-version/blob/1342d2771b99c6de6afee408d4b1916c1555cab3/.goreleaser.yml#L63-L76.

[jmMeessen]

I tried it out the suggested method , expecting the GITHUB_TOKEN of jenkins-infra to have some unusual magic. And it didn't work.

Looking closely at the GitHub action used by jenkins-version, I can demonstrate that it uses the GH App technique and Tibdex/github-app-token the same way as my program did.

      - uses: tibdex/github-app-token@3beb63f4bd073e61482598c45c71c1019b59b73a # v2
        id: generate-token
        with:
          app_id: ${{ secrets.JENKINS_ADMIN_APP_ID }}
          private_key: ${{ secrets.JENKINS_ADMIN_APP_PRIVKEY }}

https://github.com/jenkins-infra/jenkins-version/blob/1342d2771b99c6de6afee408d4b1916c1555cab3/.github/workflows/release.yaml#L54C1-L58C64

The only difference is that instead of loading the temporary token in HOMEBREW (obvious name IMHO), the designer of jenkins-version loads it in GITHUB-TOKEN.

@lemeurherve maybe the used secrets name are a hint to what GH App to use (JENKINS_ADMIN_APP).

I am restoring my original code.

[jmMeessen]

Any news about the planning of this issue @lemeurherve? I see that it is still in triage. This is blocking the transfer of the system and I have only 6 days left (and other issues to solve).

If it can help, I can help you by explaining how it works (I implemented it with three applications) although I have only limited access to the jenkins-infra org. I believe that the application to install and configure in the two migrated application is JENKINS_ADMIN_APP.

[dduportal]

Any news about the planning of this issue @lemeurherve? I see that it is still in triage. This is blocking the transfer of the system and I have only 6 days left (and other issues to solve).

If it can help, I can help you by explaining how it works (I implemented it with three applications) although I have only limited access to the jenkins-infra org. I believe that the application to install and configure in the two migrated application is JENKINS_ADMIN_APP.

For info, the jenkins-infra team is doing triage every Tuesday during the weekly meeting (except for "level 1" support requests or production issue of course): this issue will be triaged later today and most probably be part of the upcoming milestone since worked already started.

@jmMeessen Just to be sure (as I'm only starting to check this topic while preparing the weekly meeting): can you confirm the migrated repository is specifying the existing https://github.com/jenkins-infra/homebrew-tap (and not https://github.com/jenkins-infra/homebrew which does not exist)?

Another point: I confirm that @jmMeessen said that a GH Application is required for the homebrew release to the tap as the Github Action GITHUB_TOKEN is always scoped to the repository where it runs. We need to set up one / reuse eventually existing one with permissions to https://github.com/jenkins-infra/homebrew-tap and insert the 2 credentials in the migrated repositories.

[jmMeessen]

Sorry for the unnecessary question about the delayed triage. I am not aware of the internal organisation of the INFRA team (and I was led to think that this was a normal "after sales service" of the migration ticket)

can you confirm the migrated repository is specifying the existing

https://github.com/jenkins-infra/homebrew-tap I confirm and it was specifically asked during the migration on Friday. The configuration has been adapted accordingly. See https://github.com/jenkins-infra/jenkins-contribution-extractor/blob/abe61e18870c4e05cfa8f92d79e7c980c48f12b3/.goreleaser.yml#L54 See https://github.com/jenkins-infra/jenkins-contribution-aggregator/blob/113387c4baee59247bf49ae5cdffb667a4e57325/.goreleaser.yml#L55 /- Jmm Le mar. 18 juin 2024 à 11:09, Damien Duportal ***@***.***> a écrit :

…

Any news about the planning of this issue @lemeurherve <https://github.com/lemeurherve>? I see that it is still in triage. This is blocking the transfer of the system and I have only 6 days left (and other issues to solve). If it can help, I can help you by explaining how it works (I implemented it with three applications) although I have only limited access to the jenkins-infra org. I believe that the application to install and configure in the two migrated application is JENKINS_ADMIN_APP. For info, the jenkins-infra team is doing triage every Tuesday during the weekly meeting (except for "level 1" support requests or production issue of course): this issue will be triaged later today and most probably be part of the upcoming milestone since worked already started. @jmMeessen <https://github.com/jmMeessen> Just to be sure (as I'm only starting to check this topic while preparing the weekly meeting): can you confirm the migrated repository is specifying the existing https://github.com/jenkins-infra/homebrew-tap (and not https://github.com/jenkins-infra/homebrew which does not exist)? Another point: I confirm that @jmMeessen <https://github.com/jmMeessen> said that a GH Application is required for the homebrew release to the tap as the Github Action GITHUB_TOKEN is always scoped to the repository where it runs. We need to set up one / reuse eventually existing one with permissions to https://github.com/jenkins-infra/homebrew-tap and insert the 2 credentials in the migrated repositories. — Reply to this email directly, view it on GitHub <#4141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABN3S7L57VTF2JMEWYTW5Q3ZH72LXAVCNFSM6AAAAABJKTS35GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZVGU4TKNBVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dduportal]

triaged: Related to #4017 , added to milestone, taken by @lemeurherve

[dduportal]

Did a quick check on the existing jenkins-infra/homebrew-tap repository. It was used by jenkins-infra/uc (now archived) and jenkins-infra/jenkins-version.

I understand that the GitHub applications we used to use are old and were misunderstood (by us) in the past: the scoped both updatecli, homebrew tap, binary releases inside GHAs which is too vast.

@lemeurherve a few pointers to unblock this issue quickly:

• I've decreased the scope of the GH Application "updatecli GHA in jenkins-infra org": it cannot access jenkins-infra/uc (archived) and jenkins-infra/homebrew-tap (no updatecli GHA workflow) anymore. Only updatecli GHA workflow should be able to use it!
  • Note: this GH application was/is used in GHA through the secrets JENKINS_ADMIN_APP_ID and JENKINS_ADMIN_APP_PRIVKEY:
    • Example with uc release process (not valid anymore as the repo is archived) https://github.com/jenkins-infra/uc/blob/11537cefa1dc2e433ba976f52da92b23d7e46a12/.github/workflows/release.yaml#L66-L67
    • Example with jenkins-version (not working anymore since I removed the permission to the homebrew tap): https://github.com/jenkins-infra/jenkins-version/blob/0e49bf3c54dc7b93002b254ab4c7d1784d34a4ff/.github/workflows/release.yaml#L54-L58
• I'm going to setup the new permission model and apply it to jenkins-version to validate it:
  • Creating a new GH application only for goreleaser process (homebrew-tap but not only - see note below) and installing it only on required repositories
  • Setting up GH organization secrets for this app and share them only on the repositories which needs them
  • Editing the release workflow of jv to use the new secrets
• Then, @lemeurherve should be able to do the same on @jmMeessen 's CLI repository to unblock the publication here

Note about goreleaser: as per https://goreleaser.com/ci/actions/#token-permissions, we can only provide one GH token to goreleaser. Since it manages publication in the repo where it runs (GH release binaries, tag pushing, etc.) AND homebrew-tap in another repository, it means we cannot have a GH application only for homebrew: its scope is wider for goreleaser.

[lemeurherve]

New "goreleaser in jenkins-infra" GitHub App created (App ID: 925062), with "Contents: Read & Write" permissions, installed on selected repositories (https://github.com/jenkins-infra/jenkins-version & https://github.com/jenkins-infra/jenkins-contribution-extractor).

Tested this new GitHub App on jenkins-version (jenkins-infra/jenkins-version#201) with success:
jenkins-infra/homebrew-tap@4499c9d

Opening a pull request on https://github.com/jenkins-infra/jenkins-contribution-extractor to use this new GitHub App credentials.

[lemeurherve]

Opened jenkins-infra/jenkins-contribution-extractor#53

[lemeurherve]

Retrigered the last failed goreleaser job https://github.com/jenkins-infra/jenkins-contribution-extractor/actions/runs/9548130664, got the following error:

⨯ release failed after 8s error=1 error occurred:

• homebrew tap formula: could not update "Formula/jenkins-contribution-extractor.rb": PUT https://api.github.com/repos/jenkins-infra/homebrew-tap/contents/Formula/jenkins-contribution-extractor.rb: 403 Resource not accessible by integration []

Currently looking at what's wrong.

[lemeurherve]

As I rerun the job, it used the same commit, thus it didn't include the new credentials.

Created a new release to trigger goreleaser which worked as expected:

• https://github.com/jenkins-infra/jenkins-contribution-extractor/releases/tag/v0.2.18
• jenkins-infra/homebrew-tap@3560409

Closing this issue as resolved, thanks @dduportal for the help!

[dduportal]

Reopening until @jmMeessen can confirm (most probably Thursday) that it is good for him

[lemeurherve]

While doing this in pair we forgot to do the same than #4141 (comment) for the second Go repository https://github.com/jenkins-infra/jenkins-contribution-aggregator.

Taking care of it.

[lemeurherve]

Installed the "goreleaser in jenkins-infra" GitHub App to https://github.com/jenkins-infra/jenkins-contribution-aggregator and opened jenkins-infra/jenkins-contribution-aggregator#40 to use its credentials.

Also removed the remnant HOMEBREW_APP_ID and HOMEBREW_APP_PRIVKEY repository secrets from these two Go repositories as they aren't used anymore.

[jmMeessen]

I confirm that (for both repositories)

• the trigger of a release works with the access rights of a maintainer (defining and pushing a tag to the main branch of the repository)
• the automatic HomeBrew tap update works with the GitHub Action
• the new executables can be installed locally (or in a container) with the brew install command.

This issue/task can be considered as completed.

I thank all those who participated to make this possible. It was critical seen the severe time constrain we had to complete and stabilise the migration in the short time left.

[gounthar]

Thanks a lot to all the people that were involved in that migration. 🙏