CannotVerifyCopySource error - recursive copy from blob to file share on same storage account

[jburger]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

azcopy version 10.3.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux-5.0.0-32-generic-x86_64-with-Ubuntu-19.04-disco

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy copy https://[stg-account].blob.core.windows.net/blob-container?[sas-token] https://[stg-account].file.core.windows.net/file-share?[sas-token] --recursive=true

What problem was encountered?

The file was copied but contains null chars instead of the text 'abcd' as expected

How can we reproduce the problem in the simplest way?

1. Create a storage account called storage-account, with a blob container blob-container and a file share called file-share
2. Create a SAS token that isn't limited by IP or otherwise restricted on any scopes
3. Upload a simple text file containing the text "abcd" to the blob container
4. Attempt to copy the contents of blob-container into file-share using the aforementioned command

Have you found a mitigation/solution?

None - tried to use blobxfer and that failed too, see Azure/blobxfer#108

Error logs

Despite being a 'fresh' SAS token with no explicit restrictions I get an auth failure code CannotVerifyCopySource

Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-file-go/azfile.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-file-go@v0.6.0/azfile/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=CannotVerifyCopySource) =====
Description=This request is not authorized to perform this operation.
RequestId:c6673793-b01a-004a-207b-9069d3000000
Time:2019-11-01T06:13:13.2581669Z, Details: 
   Code: CannotVerifyCopySource
   PUT https://[st-account].file.core.windows.net/file-share/testfile.txt?REDACTED
   Content-Length: [0]
   User-Agent: [AzCopy/10.3.1 Azure-Storage/0.6.0 (go1.12; linux)]
   X-Ms-Client-Request-Id: [d850241c-89fd-4599-62fb-388537110d75]
   X-Ms-Copy-Source: [https://[st-account].blob.core.windows.net/blob-container/testfile.txt?REDACTED
   X-Ms-Range: [bytes=0-4]
   X-Ms-Source-Range: [bytes=0-4]
   X-Ms-Version: [2019-02-02]
   X-Ms-Write: [update]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [248]
   Content-Type: [application/xml]
   Date: [Fri, 01 Nov 2019 06:13:12 GMT]
   Server: [Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [d850241c-89fd-4599-62fb-388537110d75]
   X-Ms-Error-Code: [CannotVerifyCopySource]
   X-Ms-Request-Id: [c6673793-b01a-004a-207b-9069d3000000]
   X-Ms-Version: [2019-02-02]

[jburger]

First time azcopy user - apologies if I'm holding this wrong.

[JohnRusk]

Hi, there are two issues here.

The first issues is the authentication failure. This seems to be that one of your SAS tokens (probably the blob one, I think) isn't giving AzCopy the necessary rights. I can't tell from the info whether that's because the token you constructed didn't actually give the necessary rights, or if your token was fine but it wasn't interpreted properly. Or, there's one other possibility: if I recall correctly, you can't do service-to-service copies between the public (i.e. global) Azure cloud and the national (aka sovereign) clouds. (E.g. the US Govt one).

Assuming your source and destination are both in the some cloud (e.g. both in the main Azure public cloud) or both in US Govt, or both in Azure China etc, then it's probably just some kind of SAS issue. I'd suggest that you fire up Storage Explorer, and right click on the source container and the destination file share, and choose on each the "Get Shared Access Signature". For the source, use the default permissions (Read and List) and for the destination tick all the boxes (read, list, write, update, create etc). If your copy works with those SAS's then it must be something about the SAS's that was causing your failure. Please let us know how you get on.

The other issues is that, after failures, AzCopy is supposed to delete destination files that have been created but which don't yet have any content. So, instead of seeing a file full of zeroes, you should have seen no destination file at all. Unfortunately, there is one case where that automatic cleanup doesn't happen: authentication errors. I've added a task to our backlog to make that cleanup happen after authentication failures.

[jburger]

Thanks for the quick response,

I tried creating separate SAS tokens at the container level and didn't get far. I didn't think to create a brand new storage account before now though - which works just fine!

However, it is starting to look like the storage accounts network configuration is a factor. We aren't on a govt cloud here but we are doing some custom things in our network configuration.

The one that worked has public networking:

However, in my failing target environment it was configured like this:

(I confirmed that IP address of the machine running the azcopy commands is in that firewall list)

There is a single virtual network on a private /16 with an Endpoint Status of enabled under main.

I tried allowing read access to logging and metrics and that didn't work either. Is this an unsupported scenario, or should I try adding some ranges in this firewall list?

I've added a task to our backlog

Re: cleanup - ah ok that makes sense - thanks for letting us know!

[JohnRusk]

Incorrect theory about root cause, deleted from this comment to avoid confusion

I'll ask colleagues and see what I can find.

[JohnRusk]

I can reproduce the problem when copying from Blob to File. Am making inquiries and will update here when have some more info.

[JohnRusk]

The team has replied.

It's supposed to be enough to grant the machine running AzCopy access in your firewall rules, as you have done. And in fact that works fine for blob to blob copies. In that case, the system is smart enough to securely use the client IP address to authenticate the service-to-service copy.

The same behaviour is not yet enabled for Blob to Azure Files. It will be enabled soon - e.g. in about 2 weeks. If that's going to hold up your work, please email me and we'll see what we can do. We might be able to do something sooner for you. My email address is in my profile page.

[JohnRusk]

I'm going to close this one now, because it's service-side issue, not a problem with AzCopy itself. As noted above, the service-side fix should be rolled out soon.

[JohnRusk]

Oh, and the second issue here, the failure to cleanup destination files after authentication errors, was fixed in yesterday's release of AzCopy 10.3.2.