Issue with GitHub Bot PAT Resulting in "Bad Credential" Errors

[ellahathaway]

We are currently facing an issue where usage of the GitHub bot PAT results in "bad credential" errors. This problem arises during our release pipeline runs, specifically when attempting to upload PRs or publish the release announcement. It also occurs in the sdk-diff pipeline and license scan pipeline runs when these need to submit a PR to update the baselines and exclusions.

Upon investigation, we found that this problem originated from one of the Personal Access Tokens (PATs) being rotated on Monday (06/03), but the corresponding secret was not updated in the Key Vault at the same time.

Steps taken so far to address this issue include:

• Creation of a PAT named BotAccount-dotnet-sb-bot-pat, aligning with the Secret Manager's procedure.

Next steps to resolve this issue are:

• Update the secret in the Key Vault (this is the variable group).
• Coordinate between A&D and ProdCon on the future management of this secret's rotation
  • There has been confusion due to the manual updating and renaming of this secret, which complicates the process for those managing the secrets manually
  • If this secret has been referenced anywhere with the old name (as it was in this variable group)) and it wasn't updated in the Key Vault, we are likely to encounter issues
  • Likewise, if we reference the new name of the secret, we risk running into problems if the secret name is ever restored back to what the person managing the rotation manually expects it to be.
• The dotnet-release repository needs to set up a daily or weekly pipeline to manage the cycling of secrets in order to prevent similar issues in the future.
  • The Eng Services Team has mentioned that this responsibility falls on whoever is in charge of the dotnet-release repository

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ellahathaway]

Following up offline - will update once I get a concrete understanding of what happened and how it was fixed.

[ellahathaway]

The problem originated from the PAT rotation that occurred. It is likely that when the GH token was regenerated, a mismatch occurred between the secret in the KeyValue and the GH PAT.

In more detail, we rotate these secrets internally with the secret manager, so it seems that the rotation date in the secret manager was different from the actual secret expiration date. This is possible because the secret manager has a hardcoded value for the expiration date, so it can be configured separately from the GitHub token. In other words, the secret manager sets these dates for 6 months in advance but that doesn't happen when we create the GitHub token, so it expires without us getting the internal notification.

Here's what occurred to resolve the problem:

1. Logged into the GitHub bot and recreated the token.
2. Put the new token into the KeyVault so that it'd get pulled by the pipelines.
3. @dkurepa opened a PR to improve this process & changed the secret metadata so that we can rotate before it expires.

I think that Djuradj's PR should solve this issue in the future. Once I find the link for it I'll attach it to this issue and close. I don't think that there's any further action to by taken beyond this.

[ellahathaway]

Closing with the merge of dotnet/dnceng#3143