Files with uppercase extensions are not uploaded with azcopy

[Jogai]

We are migrating to Azure, but we have a lot of files. I have a copy script for every file extension we want to copy to blob storage. We are using AzCopy 8. For example png files:

For regular png files we get the result:

Finished 1252 of total 1252 file(s).
[2018/08/24 00:36:15] Transfer summary:
-----------------
Total files transferred: 1252
Transfer successfully:   1252
Transfer skipped:        0
Transfer failed:         0
Elapsed time:            00.00:00:26

This command however

.\AzCopy.exe /@:"C:\Path\inputParams.txt" /Pattern:"*.PNG /V:"C:\LogPath\png-cap.log"

inputParams.txt is this:

/Source:"D:\RootPath" 
/Dest:https://blip.blob.core.windows.net/root/ 
/DestKey:{omitted}== 		
/SetContentType
/NC:4
/S 
/XO
/Y

Prints this

Finished 0 of total 0 file(s).
[2018/08/24 00:36:32] Transfer summary:
-----------------
Total files transferred: 0
Transfer successfully:   0
Transfer skipped:        0
Transfer failed:         0
Elapsed time:            00.00:00:17

The output of C:\LogPath\png-cap.log is:

[2018/08/24 00:36:15.362+02:00] >>>>>>>>>>>>>>>>
[2018/08/24 00:36:15.378+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 00:36:15.378+02:00][VERBOSE] 8.0.0 : AzCopy /@:C:\Path\inputParams.txt /Pattern:*.PNG /V:C:\LogPath\png-cap.log
[2018/08/24 00:36:15.471+02:00][VERBOSE] Attempt to parse address 'D:\RootPath' to a directory as a candidate location succeeded.
[2018/08/24 00:36:15.471+02:00][VERBOSE] Source is interpreted as a Local directory: D:\RootPath\.
[2018/08/24 00:36:15.502+02:00][VERBOSE] Attempt to parse address 'https://blip.blob.core.windows.net/root/' to a directory as a candidate location succeeded.
[2018/08/24 00:36:15.518+02:00][VERBOSE] Attempt to parse address 'https://blip.blob.core.windows.net/root/' to a single file location failed: Invalid location 'https://blip.blob.core.windows.net/root/', cannot get valid account, container and blob name.
[2018/08/24 00:36:15.518+02:00][VERBOSE] Destination is interpreted as a Cloud blob directory: https://blip.blob.core.windows.net/root/.
[2018/08/24 00:36:20.370+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 00:36:25.377+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 00:36:30.369+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 00:36:32.647+02:00] Transfer summary:
-----------------
Total files transferred: 0
Transfer successfully:   0
Transfer skipped:        0
Transfer failed:         0
Elapsed time:            00.00:00:17

Is there a way to have the files with capital letters as an extension uploaded?

[zezha-msft]

Hi @Jogai, thanks for reaching out! We’ll investigate and get back to you shortly.

[Jogai]

Much appreciated

[jvallery]

Jogai - This looks to me like you're using the GA'ed azcopy (v8.1) and not the one covered in this repo? We should still help but just making sure you're not testing out v10?

[Jogai]

Correct. I'm happy to try v10 if I could find the download...

AzCopy 8.0.0-netcorepreview Copyright (c) 2018 Microsoft Corp. All Rights Reserved.

[Jogai]

Sorry, was using an older version indeed. The latest version I see onlin is

AzCopy 8.1.0-netcore Copyright (c) 2018 Microsoft Corp. All Rights Reserved.

But the latest version does give the same result

[2018/08/24 01:43:43.055+02:00] >>>>>>>>>>>>>>>>
[2018/08/24 01:43:43.227+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 01:43:43.414+02:00][VERBOSE] 8.1.0 : AzCopy /@:C:\inputParams.txt /Pattern:*.PNG /V:C:\png-cap.log
[2018/08/24 01:43:43.867+02:00][VERBOSE] Attempt to parse address 'E:\RootPath' to a directory as a candidate location succeeded.
[2018/08/24 01:43:43.867+02:00][VERBOSE] Source is interpreted as a Local directory: E:\RootPath\.
[2018/08/24 01:43:43.945+02:00][VERBOSE] Attempt to parse address 'https://blip.blob.core.windows.net/root/' to a directory as a candidate location succeeded.
[2018/08/24 01:43:44.007+02:00][VERBOSE] Attempt to parse address 'https://blip.blob.core.windows.net/root/' to a single file location failed: Invalid location 'https://blip.blob.core.windows.net/root/', cannot get valid account, container and blob name.
[2018/08/24 01:43:44.007+02:00][VERBOSE] Destination is interpreted as a Cloud blob directory: https://blip.blob.core.windows.net/root/.
[2018/08/24 01:43:48.188+02:00][VERBOSE] Finished: 0 file(s), 0 B; Average Speed:0 B/s.
[2018/08/24 01:43:50.918+02:00] Transfer summary:
-----------------
Total files transferred: 0
Transfer successfully:   0
Transfer skipped:        0
Transfer failed:         0
Elapsed time:            00.00:00:08

And I'm sure the files do exist...

[seguler]

@EmmaZhu for azcopy v8

[EmmaZhu]

@Jogai
I just tried to upload files with extension ".TXT" and ".txt" with following command successfully:
AzCopy.exe /source:AzCopySourceFolder /dest:http://myaccount.blob.core.windows.net/container1 /destkey:*** /pattern:*.TXT

From the command line you shared, could you check whether there's file with exact the same name on destination which has newer last modified time? It might have been filtered out for /XO option.

[Jogai]

Sorry, logical error on my part.

I made a script that outputted my parameter files and the commands I needed to run. It ouputs the file types I want to copy in a azcopy statement, ordered by how many files there are with an extension. For png files it gives me this:

.\AzCopy.exe /@:"C:\Path\inputParams.txt" /Pattern:"*.png" /V:"C:\LogPath\png-270560.log"

.\AzCopy.exe /@:"C:\Path\inputParams.txt" /Pattern:"*.PNG" /V:"C:\LogPath\PNG-109203.log"

I always have more files with normal lowercase extension, and that command does copy the uppercase ones as well... Then the uppercase command is run, but because of the XO parameter the files are skipped indeed.

If I look in my log files the count is indeed the sum for both (the file names of the actual log do have the number of files per extension in them).

Thanks for pointing me in the right direction!

Edit: in my defense it was over a million blobs, over 54 GB of data, so I didn't go in there to check a specific file on a specific location....