Cannot sync local dir with Azure Storage Emulator

[rsheptolut]

Which version of the AzCopy was used?

AzCopy 10.2.1

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy sync "C:\LocalDir" "http://127.0.0.1:10000/devstoreaccount1/mycont?%SAS%"

What problem was encountered?

I get a following error:
error parsing the input given by the user. Failed with error Unable to infer the source 'C:\LocalDir' / destination 'http://127.0.0.1:10000/devstoreaccount1/mycont?%SAS%'.

It occurs because I'm trying to sync my local folder with Azure Blob Storage that runs locally in the emulator and its url doesn't contain anything hinting it's a blob storage. The error message does nothing to explain this.

How can we reproduce the problem in the simplest way?

1. Make sure you have Azure Storage Emulator installed and initialized.
2. Create a container called mycont in it, issue an SAS token for it.
3. Create a C:\LocalDir folder with some files in it.
4. Try running the above command with SAS supplied.

Have you found a mitigation/solution?

The solution I've found is simply not using sync and settling with copy. This is a poor workaround, because copy seems to always be overwriting all files and overall does not, you know, sync. The crucial difference that allows copy to work is the fact that it supports the --from-to flag, and supplying --from-to LocalBlob helps AzCopy to figure out what I'm trying to do. Again, no guidance on this anywhere in azcopy help or docs. The following command works:
azcopy copy"C:\LocalDir" "http://127.0.0.1:10000/devstoreaccount1/mycont?%SAS%" --from-to LocalBlob

The same flag doesn't work with sync:

azcopy sync "C:\LocalDir" "http://127.0.0.1:10000/devstoreaccount1/mycont?%SAS%" --from-to LocalBlob
unknown flag: --from-to

Please add support for the --from-to flag to the sync command as there's no reason for the command not to support it. The sync command is also missing a number of other flags that copy has without any apparent reason, for example, --blob-type.

[zezha-msft]

Hi @rsheptolut, thanks for reaching out!

Sync is indeed missing some flags that copy already has. We have plans to make up the difference.

[JohnRusk]

@zezha-msft Any changes to the status of this in 10.3.x or coming in 10.4?

[JohnRusk]

(Just asking to try to get right milestone tag on this issue, if possible)

[InteXX]

I'm very interested in the outcome on this. I need to be able to sync to a local emulator.

[InteXX]

Here's the offending code:

func inferArgumentLocation(arg string) common.Location {
  if arg == pipeLocation {
    return common.ELocation.Pipe()
  }
  if startsWith(arg, "http") {
    // Let's try to parse the argument as a URL
    u, err := url.Parse(arg)
    // NOTE: sometimes, a local path can also be parsed as a url. To avoid thinking it's a URL, check Scheme, Host, and Path
    if err == nil && u.Scheme != "" && u.Host != "" {
      // Is the argument a URL to blob storage?
      switch host := strings.ToLower(u.Host); true {
      // Azure Stack does not have the core.windows.net
      case strings.Contains(host, ".blob"):
        return common.ELocation.Blob()
      case strings.Contains(host, ".file"):
        return common.ELocation.File()
      case strings.Contains(host, ".dfs"):
        return common.ELocation.BlobFS()
      case strings.Contains(host, benchmarkSourceHost):
        return common.ELocation.Benchmark()
        // enable targeting an emulator/stack
      case IPv4Regex.MatchString(host):               // <-- BUG here
        return common.ELocation.Unknown()             // <-- This is what gets returned in case of storage emulator URL
      }

      if common.IsS3URL(*u) {
        return common.ELocation.S3()
      }
    }
  }

  return common.ELocation.Local()
}

Credit to Gaurav Mantri-AIS for finding it.

[bluenote10]

Sync is indeed missing some flags that copy already has. We have plans to make up the difference.

@zezha-msft Any news on that? We are running into the same problem. Since copy already supports the --fromTo argument, it is weird that sync doesn't -- and that's quite a showstopper regarding e.g. Azurite compatibility.

Shouldn't it be sufficient to switch the usage of inferFromTo in sync.go

azure-storage-azcopy/cmd/sync.go

Line 145 in 87cba06

cooked.fromTo = inferFromTo(raw.src, raw.dst)

to validateFromTo as is done in copy.go

azure-storage-azcopy/cmd/copy.go

Line 272 in 87cba06

fromTo, err := validateFromTo(raw.src, raw.dst, raw.fromTo) // TODO: src/dst

?

Internally validateFromTo falls back to inferFromTo anyway if the user didn't specify a --fromTo.

[Oceanswave]

Encountering this bug with Azurite as well - the annoying part is that it mentions to use the --from-to flag, but then when providing the flag indicatesunknown flag: --from-to

Also, want to wish this issue an early 2nd birthday! 🎂

[gapra-msft]

Apologies for not updating this thread sooner. Sync supports the from-to flag starting from v10.13.0 https://github.com/Azure/azure-storage-azcopy/blob/main/ChangeLog.md#version-10130 which was released Oct 2021. This should allow support for syncing a local dir with Azurite. I will go ahead and close this thread for now, but if you are still having any issues, please do re-open this thread or open a new one. Thank you again for your feedback