[QUESTION] AzCopy list fails with "403 Server failed to authenticate the request." #77

MatthewMcD · 2018-10-09T13:59:31Z

I am brand new to AzCopy so forgive me, but I have reviewed the readme.md and really tried to figure this one out. Eventually I'd like to test sync but I am stuck at the starting gate.

Version 10.0.2-Preview

Windows 10

Commands

ps> azcopy.exe login
Login succeeded.
ps> azcopy.exe list https://mystorage.blob.core.windows.net/backups
List is using OAuth token for authentication.

Error:

cannot list blobs for download. Failed with error -> github.com/Azure/azure-storage-azcopy/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob.NewResponseError, /go/src/github.com/Azure/azure-storage-azcopy/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob/zz_generated_response_error.go:28
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:db87066e-c01e-0173-67d6-5fb0b6000000
Time:2018-10-09T13:43:58.0479030Z, Details:
   AuthenticationErrorDetail: Issuer validation failed. Issuer did not match.
   GET https://mystorage.blob.core.windows.net/backups?comp=list&restype=container&timeout=901
   Authorization: REDACTED
   User-Agent: [AzCopy/v10.0.2-Preview Azure-Storage/0.1 (go1.10.3; Windows_NT)]
   X-Ms-Client-Request-Id: [8a047954-e7bf-4d30-77b0-ce3ea113e728]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [422]
   Content-Type: [application/xml]
   Date: [Tue, 09 Oct 2018 13:43:57 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [db87066e-c01e-0173-67d6-5fb0b6000000]

The text was updated successfully, but these errors were encountered:

jiacfan · 2018-10-10T01:41:25Z

Update:

For 403 "AuthenticationErrorDetail: Issuer validation failed. Issuer did not match."
AzCopy v10 has changed to login with default tenant ID "common". It can help to detect custom tenant ID automatically, and customizing --tenant-id for non-Microsoft tenant is no more required.
A special case is Guest user (link), which is used in multi-tenant B2B scenario. In this case, --tenant-id is necessary.
If you see "AuthenticationErrorDetail: Issuer validation failed. Issuer did not match.", please check if you're using B2B account, and specify --tenant-id explicitly.
For 403 "Description=This request is not authorized to perform this operation using this permission."
Please check if role assignment is with enough permission according to link.
Additionally, please note that initial role assignment may take 5 min to take effect (link)

Sorry that as work switch, I just find the replies here are out-of-date, hope above could help. At same time, thanks @JohnRusk for tracking the issue and help to provide suggestion.

===========History reply 2018===========
Hi, @MatthewMcD

Thanks for reaching us.

The error message:
"AuthenticationErrorDetail: Issuer validation failed. Issuer did not match." is returned from service side, which indicates the Issuer for the token doesn't match account's, and it's likely caused that you are login into tenant A, and using account belongs to tenant B.

azcopy.exe login by default login to "microsoft.com" tenant, this can be customized with --tenant-id switch of login command.

Please double check if the tenant you are login to is same as which 'mystorage' account belongs to.

Another thing worth check is if you recently do a subscription migration in ARM? If that's the case, please check the latest tenant, and login accordingly.

Best Regards,
Jiachen

MatthewMcD · 2018-10-10T14:36:29Z

@jiacfan Thanks for the quick response. I forced the tenant with:

azcopy login --tenant-id=MyTennantGUID

and now I get:

403 This request is not authorized to perform this operation using this permission.

This account is a global administrator. I use this account with Azure Storage Explorer to upload files all the time.

PS > azcopy list https://mystorage.blob.core.windows.net/backups
List is using OAuth token for authentication.

cannot list blobs for download. Failed with error -> github.com/Azure/azure-storage-azcopy/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob.NewRe
sponseError, /go/src/github.com/Azure/azure-storage-azcopy/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob/zz_generated_response_error.go:28
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:7fe7a118-e01e-014d-77a5-600697000000
Time:2018-10-10T14:29:36.4603543Z, Details: (none)
   GET https://mystorage.blob.core.windows.net/backups?comp=list&restype=container&timeout=901
   Authorization: REDACTED
   User-Agent: [AzCopy/v10.0.2-Preview Azure-Storage/0.1 (go1.10.3; Windows_NT)]
   X-Ms-Client-Request-Id: [431c1c2e-8db1-43ea-68bc-5ee047094d8c]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Content-Length: [279]
   Content-Type: [application/xml]
   Date: [Wed, 10 Oct 2018 14:29:36 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [7fe7a118-e01e-014d-77a5-600697000000]
   X-Ms-Version: [2018-03-28]

artemuwka · 2018-10-15T21:08:15Z

Hi @MatthewMcD! We've recently introduced the "Blob Data Contributor" role on the storage accounts (https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac). Please make sure your account has this role assigned and try again. Hope this helps!

MatthewMcD · 2018-10-16T15:00:36Z

Thanks @artemuwka , I would have thought that a Global Administrator would have this right. For anyone looking for the Role in the UI it's called "Storage Blob Data Contributor (Preview)". For a Resource Group choose Access Control (IAM) | Add in the blade locate the role Storage Blob Data Contributor (Preview) and assign access to the Users, Groups or Roles as meets your needs.

Thanks!

MatthewMcD · 2018-11-11T13:57:35Z

For anyone stumbling into this like me: AzCopy sync as a Backup Solution

bfleming-ciena · 2019-02-22T22:25:37Z

Good grief, I set blob owner and it didn't work. Then I set contributor - thumbs up.

CIPop · 2019-06-05T18:30:28Z

I just spent 30 minutes trying to figure out what a "tenant-id" is and how to find it.
Turns out you need to log-on to portal, click on
then
. Now search for "Directory ID" and use that GUID.

This is very poorly documented. @jiacfan it would be nice to have he tool explain this could be the case. I'll also open a doc bug against https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs which doesn't work for everybody as written.

JohnRusk · 2019-06-05T20:28:35Z

CC @normesta re @CIPop's comment above. FYI I find it odd that our docs seldom (never?) seem to come out and say:

"Tenant" means "Azure Active Directory Instance". I.e. a tenant is a particular directory. We use the word tenant instead of "directory" because "Active Directory Tenant" avoids the repetition that would arrise with "Active Directory Directory". [At least, I presume that's why we use "tenant" instead of "directory"]

Amd3202 · 2019-12-26T16:50:31Z

I had the above problem... but what's weird is i was able to uplaod 10,000 jpgs into a folder called [container]/images without issue. But the second (local) folder fails (15k items)... I had the same experience as above with azcopy and then i was able to at least get az copy to "work" but i was able to upload files without this role before... i literally just added the role. Not sure if this is a bug or if different clients circumvent this issue, and if so why only for the first folder?

JohnRusk · 2020-01-05T22:11:19Z

That is weird @Amd3202 . I suspect there must be some logical explanation, but from your description I can't tell what that might be.

Amd3202 · 2020-01-06T01:12:17Z

Thanks yeah it's been a weird ride. but we got it working from a mix of using the portal and AZcopy app which is also no fun... oh well! i guess that's why we're lucky enough to be working sunday nights!

JohnRusk · 2020-01-06T01:21:39Z

I'm in a different timezone, so it's Monday already for me :-) Hope the rest of your Sunday goes smoothly and glad to hear you've got it working.

LearnsHappily · 2020-01-12T01:35:31Z

Using AzCopy V10.3.3
What a FRUSTRATING experience!!!

Goal: copy a large file from a VM to a Blob container.

I am a tenant on the "regular" Microsoft cloud and logged in accordingly using "AzCopy login" (without the quotes of course). Actually, I tried it without and then with the parameter tenant-id=
Both cases login was successful and I got a jason token placed in C:\Users\LearnsHappily.azcopy;
Although I am the OWNER of the Blob, I followed the illogical need to add to myself the "contributor" role. [non-trivial and poorly documented requirement]
Then I executed "AzCopy copy" and got repeatedly this message:

RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
Job 561573ff-afb4-144b-41f0-b2c34b353f02 summary
Elapsed Time (Minutes): 0.0669
Total Number Of Transfers: 1
Number of Transfers Completed: 0
Number of Transfers Failed: 1
Number of Transfers Skipped: 0
TotalBytesTransferred: 0

So, I decided to use a SAS token instead, per https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs
I created the SAS token using the portal.
This too failed: "no SAS token or OAuth token is present and the resource is not public"
So I created a new SAS using the Storage Explorer (right-click on the container and create a SAS token for all CRED operations. This too FAILED.

I am using a .bat file to execute the command. I tried using double quotes (per the documentation), single quotes, no quotes. It matters not. I've been unable to get AzCopy running as expected and needed.

What am I supposed to do?

LearnsHappily · 2020-01-12T03:25:55Z

Update

For CMD.EXE (or a .BAT) using a SAS token:

Remove all quotes
Replace all % with %% (escape it)
Replace all & with ^& (escape it)
Execute the command.
It works for me.
Took me 3 hours to fight this hostile utility :-(

JohnRusk · 2020-01-12T20:53:06Z

Hi, sorry to hear about the difficulties you had.

FYI, your initial auth problem may have been the 5 min delay mentioned here: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?toc=%2fazure%2fstorage%2fblobs%2ftoc.json#choose-how-youll-provide-authorization-credentials "Keep in mind that RBAC role assignments can take up to five minutes to propagate."

Yes, using SAS tokens in CMD is a pain, due to the parsing rules. As far as I can tell, those issues are CMD's parsing rules, rather than anything inside Azcopy itself. PowerShell does't have those problems. I encourage AzCopy users to use PowerShell.

ahelwer · 2020-03-27T19:48:34Z

How is it that being an owner on a subscription doesn't give us the blob contributor permission? This does not make sense. I ended up just using SAS tokens for both source & destination storage accounts.

JohnRusk · 2020-03-29T21:08:36Z

@ahelwer I remember seeing an explanation of this that seemed to make sense to me when I read it. I can't seem to find it right now, sorry.

alexivkin · 2020-04-10T06:33:50Z

Utterly ridiculous having to grant a special role to an owner. It goes against established principles and no matter how much microsoft thinks that's it makes sense, if azure is to compete, it should make sense without having to read articles and closed github tickets. -1 point to azure.

kaddul · 2020-08-08T18:07:36Z

Update

For CMD.EXE (or a .BAT) using a SAS token:

Remove all quotes

Replace all % with %% (escape it)

Replace all & with ^& (escape it)
Execute the command.
It works for me.
Took me 3 hours to fight this hostile utility :-(

Hi @LearnsHappily !!!

I was played with SAS token but i couldn't upload one file, my mistake is SAS token

Could you share the method to generate SAS Token?

i was used the Version 2013-08-15 and Later from
https://docs.microsoft.com/en-us/rest/api/storageservices/service-sas-examples

StringToSign = r + \n   
               2013-08-16 + \n  
               2013-08-17 + \n  
               /myaccount/pictures + \n  
               YWJjZGVmZw== + \n  
               2013-08-15 + \n  
               + \n    
               file; attachment + \n  
               + \n  
               + \n  
               binary

but i didn't know what is the field "signedidentifier" or where i found it.
other SAS token with i tried to use is to rest api applied to zcopy
https://docs.microsoft.com/en-us/azure/storage/common/storage-rest-api-auth

StringToSign = VERB + "\n" +  
               Content-Encoding + "\n" +  
               Content-Language + "\n" +  
               Content-Length + "\n" +  
               Content-MD5 + "\n" +  
               Content-Type + "\n" +  
               Date + "\n" +  
               If-Modified-Since + "\n" +  
               If-Match + "\n" +  
               If-None-Match + "\n" +  
               If-Unmodified-Since + "\n" +  
               Range + "\n" +  
               CanonicalizedHeaders +  
               CanonicalizedResource;

And the same result Error,
finally i found that URL and i had the other msitake.
https://docs.microsoft.com/en-us/rest/api/storageservices/create-service-sas

StringToSign = signedpermissions + "\n" +  
               signedstart + "\n" +  
               signedexpiry + "\n" +  
               canonicalizedresource + "\n" +  
               signedidentifier + "\n" +  
               signedversion + "\n" +  
               rscc + "\n" +  
               rscd + "\n" +  
               rsce + "\n" +  
               rscl + "\n" +  
               rsct

zezha-msft assigned jiacfan Oct 9, 2018

MatthewMcD closed this as completed Oct 16, 2018

CIPop mentioned this issue Jun 5, 2019

Getting AuthorizationPermissionMismatch when I follow this guide MicrosoftDocs/azure-docs#32783

Closed

awsles mentioned this issue Aug 4, 2019

Storage Blob Data Contributor Role is the ONLY role which allows uploading MicrosoftDocs/azure-docs#36454

Closed

AriKrumbein mentioned this issue Apr 29, 2020

AzCopy returns 403 Failed to authenticate #991

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QUESTION] AzCopy list fails with "403 Server failed to authenticate the request." #77

[QUESTION] AzCopy list fails with "403 Server failed to authenticate the request." #77

MatthewMcD commented Oct 9, 2018

jiacfan commented Oct 10, 2018 •

edited

Loading

MatthewMcD commented Oct 10, 2018 •

edited

Loading

artemuwka commented Oct 15, 2018

MatthewMcD commented Oct 16, 2018

MatthewMcD commented Nov 11, 2018 •

edited

Loading

bfleming-ciena commented Feb 22, 2019

CIPop commented Jun 5, 2019

JohnRusk commented Jun 5, 2019

Amd3202 commented Dec 26, 2019

JohnRusk commented Jan 5, 2020

Amd3202 commented Jan 6, 2020

JohnRusk commented Jan 6, 2020

LearnsHappily commented Jan 12, 2020 •

edited

Loading

LearnsHappily commented Jan 12, 2020

JohnRusk commented Jan 12, 2020

ahelwer commented Mar 27, 2020 •

edited

Loading

JohnRusk commented Mar 29, 2020

alexivkin commented Apr 10, 2020

kaddul commented Aug 8, 2020

Update

[QUESTION] AzCopy list fails with "403 Server failed to authenticate the request." #77

[QUESTION] AzCopy list fails with "403 Server failed to authenticate the request." #77

Comments

MatthewMcD commented Oct 9, 2018

Version 10.0.2-Preview

Windows 10

Commands

Error:

jiacfan commented Oct 10, 2018 • edited Loading

MatthewMcD commented Oct 10, 2018 • edited Loading

artemuwka commented Oct 15, 2018

MatthewMcD commented Oct 16, 2018

MatthewMcD commented Nov 11, 2018 • edited Loading

bfleming-ciena commented Feb 22, 2019

CIPop commented Jun 5, 2019

JohnRusk commented Jun 5, 2019

Amd3202 commented Dec 26, 2019

JohnRusk commented Jan 5, 2020

Amd3202 commented Jan 6, 2020

JohnRusk commented Jan 6, 2020

LearnsHappily commented Jan 12, 2020 • edited Loading

Using AzCopy V10.3.3 What a FRUSTRATING experience!!!

Goal: copy a large file from a VM to a Blob container.

What am I supposed to do?

LearnsHappily commented Jan 12, 2020

Update

JohnRusk commented Jan 12, 2020

ahelwer commented Mar 27, 2020 • edited Loading

JohnRusk commented Mar 29, 2020

alexivkin commented Apr 10, 2020

kaddul commented Aug 8, 2020

Update

jiacfan commented Oct 10, 2018 •

edited

Loading

MatthewMcD commented Oct 10, 2018 •

edited

Loading

MatthewMcD commented Nov 11, 2018 •

edited

Loading

LearnsHappily commented Jan 12, 2020 •

edited

Loading

Using AzCopy V10.3.3
What a FRUSTRATING experience!!!

ahelwer commented Mar 27, 2020 •

edited

Loading