RequestFailedException: Service request failed. Status: 403 (This request is not authorized to perform this operation using this permission.)

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=34525

Build leg reported

coreclr Pri0 Runtime Tests Run OSX x64 checked / Send tests to Helix (Unix)

Pull Request

dotnet/runtime#76357

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "x-ms-error-code: AuthorizationPermissionMismatch",
   "BuildRetry": false
}

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
57154	dotnet/roslyn	Test	Log	dotnet/roslyn#64824
56814	dotnet/roslyn	Test	Log
56417	dotnet/roslyn	Test	Log	dotnet/roslyn#64804
56287	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#77052
52971	dotnet/roslyn	Test	Log	dotnet/roslyn#64740
52258	dotnet/runtime	Send to Helix	Log	dotnet/runtime#76949
50744	dotnet/roslyn	Test	Log
42483	dotnet/runtime	Send tests to Helix (Unix)	Log
37694	dotnet/runtime	Send tests to Helix (Unix)	Log
35359	dotnet/roslyn	Test	Log
35045	dotnet/roslyn	Test	Log
34525	dotnet/runtime	Send tests to Helix (Unix)	Log
34499	dotnet/runtime	Send tests to Helix (Unix)	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	4	13

[MattGal]

Added a bit of the error message, this looks like a super rare issue with Azure storage where the drop container's SAS URI received from the Helix API was not valid for the Helix SDK. This happens many 10s of thousands of times a day so if it's reproducible at all it should get lots of hits.

[markwilkie]

Thanks @MattGal !

[ulisesh]

We will wait until the 7-Day hit count get down to 0 to close the issue

[MattGal]

This keeps getting one-off hits, so I'm going to try to harden against it.

Checking out the repo, this has ben reported multiple times and been ignored by the team:
Azure/azure-storage-net#1040
Azure/azure-storage-net#1012

Since none of them got listed as fixed, simply updating the dependency is unlikely to work. Rather I will see if I can stick some retry logic in there.

[MattGal]

@jkotas I have hardened against this in #11364. Once this merges and flows, this mode of failure won't happen; there will be multiple retries before giving up and giving up will simply print a warning.

One thing to note here is that the reason this is so rare, and the reason you're hitting this issue, very likely may be because you are uploading the same payload twice somewhere; the code path in question can only hit if an HTTP 409 throws when uploading the payload (we then check if it exists and do nothing if so). My change should make this non-fatal in all cases either way.

[MattGal]

Given no hits since last Weds (10/19) and a mitigation merged into dotnet/arcade main, I am proactively closing this issue. If you need a fix for this, directly taking a new Arcade update via darc command is the fastest way to get relief.