Skip to content

Fix false success reporting in ado2gh integrate-boards when GitHub PAT permissions are incorrect #1354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jun 8, 2025

Problem

The ado2gh integrate-boards command was incorrectly reporting success when it actually failed due to GitHub PAT permission issues. This occurred because the Azure DevOps API returns HTTP 200 (OK) responses but includes error messages in the response body when the GitHub PAT has insufficient or incorrect permissions.

This PR will not fix the failing ADO integration tests, but it will improve the error message that both users and engineers get in this situation. Previously the test logs would show everything successful and the integration test would fail when trying to assert that the boards integration had been properly configured. Now the test logs will properly recognize and log the error when it occurs configuring boards integration.

image

Example of the Issue

When running with a GitHub PAT that has too many permissions, the command would log:

[DEBUG] RESPONSE (OK): {"dataProviders":{"ms.vss-work-web.github-user-data-provider":{"errorMessage":"An error has occurred when validating credentials. Please use correct scope for PAT token"}}}
[DEBUG] RESPONSE (OK): {"dataProviders":{"ms.vss-work-web.azure-boards-save-external-connection-data-provider":{"errorMessage":"Specified argument was out of the range of valid values.\r\nParameter name: name"}}}
[INFO] Successfully configured Boards<->GitHub integration

The command reported success despite the clear error messages in the API responses.

Solution

Enhanced error handling in the ADO API service methods used by the integrate-boards workflow:

  • GetGithubHandle - Enhanced existing error checking to also detect errorMessage in responses
  • GetBoardsGithubRepoId - Added error detection for repository access failures
  • CreateBoardsGithubConnection - Added error detection for connection creation failures
  • AddRepoToBoardsGithubConnection - Added error detection for repository addition failures

Each method now:

  1. Parses JSON responses when available
  2. Checks for errorMessage fields in the appropriate data provider sections
  3. Throws meaningful OctoshiftCliException with error details when issues are detected
  4. Maintains backward compatibility by only processing non-null/non-empty responses

Testing

Added comprehensive test coverage:

  • 4 new unit tests validating error detection in each enhanced ADO API method
  • All existing 894 tests continue to pass

Impact

  • Users now receive clear error messages when GitHub PAT permissions are incorrect
  • The command fails fast with actionable feedback instead of reporting false success
  • No breaking changes to existing successful workflows
  • Improved debugging experience with meaningful error messages

Fixes #1353.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • dev.azure.com
    • Triggering command: /usr/lib/dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/gh-gei/gh-gei/src/OctoshiftCLI.IntegrationTests/bin/Debug/net8.0/OctoshiftCLI.IntegrationTests.runtimeconfig.json --depsfile /home/REDACTED/work/gh-gei/gh-gei/src/OctoshiftCLI.IntegrationTests/bin/Debug/net8.0/OctoshiftCLI.IntegrationTests.deps.json /home/REDACTED/work/gh-gei/gh-gei/src/OctoshiftCLI.IntegrationTests/bin/Debug/net8.0/testhost.dll --port 38455 --endpoint 127.0.0.1:038455 --role client --parentprocessid 3590 --telemetryoptedin false (dns block)
  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Identity%20Verification%20Root%20Certificate%20Authority%202020.crl
    • Triggering command: dotnet build (http block)
  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Public%20RSA%20Timestamping%20CA%202020.crl
    • Triggering command: dotnet build (http block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits June 8, 2025 01:53
…lity

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] ado2gh integrate-boards reports success when it actually failed Fix false success reporting in ado2gh integrate-boards when GitHub PAT permissions are incorrect Jun 8, 2025
@Copilot Copilot AI requested a review from dylan-smith June 8, 2025 02:05
Copilot finished work on behalf of dylan-smith June 8, 2025 02:05
Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copilot finished work on behalf of dylan-smith June 8, 2025 02:16
Copy link

github-actions bot commented Jun 8, 2025

Unit Test Results

  1 files    1 suites   21s ⏱️
893 tests 893 ✅ 0 💤 0 ❌
894 runs  894 ✅ 0 💤 0 ❌

Results for commit 1ec632c.

Copy link

github-actions bot commented Jun 8, 2025

Code Coverage

Package Line Rate Branch Rate Complexity Health
ado2gh 84% 78% 631
gei 81% 73% 596
Octoshift 87% 75% 1433
bbs2gh 82% 76% 669
Summary 84% (7255 / 8595) 75% (1709 / 2264) 3329

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 8, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Identity%20Verification%20Root%20Certificate%20Authority%202020.crl
    • Triggering command: dotnet build src/Octoshift/Octoshift.csproj (http block)
  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Public%20RSA%20Timestamping%20CA%202020.crl
    • Triggering command: dotnet build src/Octoshift/Octoshift.csproj (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of dylan-smith June 8, 2025 22:39
…ppression

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 8, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Identity%20Verification%20Root%20Certificate%20Authority%202020.crl
    • Triggering command: dotnet build src/Octoshift/Octoshift.csproj (http block)
  • http://www.microsoft.com:80/pkiops/crl/Microsoft%20Public%20RSA%20Timestamping%20CA%202020.crl
    • Triggering command: dotnet build src/Octoshift/Octoshift.csproj (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of dylan-smith June 8, 2025 23:31
@dylan-smith
Copy link
Collaborator

The IntegrateBoardsCommandHandler tests are unnecessary as you didn't actually change any behavior in that command handler.

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 9, 2025

The IntegrateBoardsCommandHandler tests are unnecessary as you didn't actually change any behavior in that command handler.

You're absolutely right. I removed the unnecessary IntegrateBoardsCommandHandler tests since I didn't change any behavior in the command handler itself - only enhanced the underlying AdoApi service methods. The tests for those service methods are sufficient to verify the new error handling behavior. The changes are in commit 13bdd79.

Copilot finished work on behalf of dylan-smith June 9, 2025 05:29
@Copilot Copilot AI requested a review from offbyone June 20, 2025 14:54
Copilot finished work on behalf of offbyone June 20, 2025 14:54
Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
@Copilot Copilot AI requested a review from dylan-smith June 20, 2025 14:57
Copilot finished work on behalf of dylan-smith June 20, 2025 14:57
Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copilot finished work on behalf of dylan-smith June 20, 2025 15:23
@Copilot Copilot AI requested a review from dylan-smith June 20, 2025 15:23
…s to bubble up

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
@Copilot Copilot AI requested a review from dylan-smith June 20, 2025 15:29
Copilot finished work on behalf of dylan-smith June 20, 2025 15:29
Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copilot finished work on behalf of dylan-smith June 20, 2025 15:36
@dylan-smith
Copy link
Collaborator

The build gave this linter error:
/Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoApi.cs(9,1): warning IDE0005: Using directive is unnecessary.

Always run dotnet format before committing.

Co-authored-by: dylan-smith <1508559+dylan-smith@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 20, 2025

The build gave this linter error:
/Users/runner/work/gh-gei/gh-gei/src/Octoshift/Services/AdoApi.cs(9,1): warning IDE0005: Using directive is unnecessary.

...

Fixed the IDE0005 linter error by removing the unnecessary using Newtonsoft.Json; directive. The code only uses JObject from Newtonsoft.Json.Linq, so the base Newtonsoft.Json using was redundant. Commit 772ffea.

Copilot finished work on behalf of dylan-smith June 20, 2025 15:46
@dylan-smith dylan-smith requested a review from a team June 24, 2025 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ado2gh integrate-boards reports success when it actually failed
3 participants