No build links and can't build for MacOS

[123dev]

Hi

I first reported this here
MicrosoftDocs/azure-docs#13735 (comment)

And am now moving it to this repo as it is more appropriate.

The readme suggests that there should be links for 3 downloads

Download the AzCopy executable using one of the following links:

Windows x64
Linux x64
MacOS x64

But there are no links, so I attempted to build on MacOS myself.
Having no experience with GO, I installed Go and added one by one the respective dependencies
until I encountered the following error.

common/credCache_darwin.go:28:2: cannot find package "github.com/jiacfan/keychain" in any of:
        /usr/local/opt/go/libexec/src/github.com/jiacfan/keychain (from $GOROOT)
        /Users/hrant/go-workspace/src/github.com/jiacfan/keychain (from $GOPATH)
cmd/cancel.go:28:2: cannot find package "github.com/spf13/cobra" in any of:
        /usr/local/opt/go/libexec/src/github.com/spf13/cobra (from $GOROOT)
        /Users/hrant/go-workspace/src/github.com/spf13/cobra (from $GOPATH)
common/credCache_linux.go:26:2: cannot find package "github.com/jiacfan/keyctl" in any of:
        /usr/local/opt/go/libexec/src/github.com/jiacfan/keyctl (from $GOROOT)
        /Users/hrant/go-workspace/src/github.com/jiacfan/keyctl (from $GOPATH)
cmd/cancel.go:28:2: cannot find package "github.com/spf13/cobra" in any of:
        /usr/local/opt/go/libexec/src/github.com/spf13/cobra (from $GOROOT)
        /Users/hrant/go-workspace/src/github.com/spf13/cobra (from $GOPATH)
cmd/cancel.go:28:2: cannot find package "github.com/spf13/cobra" in any of:
        /usr/local/opt/go/libexec/src/github.com/spf13/cobra (from $GOROOT)
        /Users/hrant/go-workspace/src/github.com/spf13/cobra (from $GOPATH)

I can't find github.com/jiacfan/keychain to add that dependency.
Please kindly either provide links for builds or advise on a proper building steps.

Many thanks in advance

[zezha-msft]

Hi @123dev, thanks for reaching out!

We are still in the process of getting the download links ready, this should happen very soon.

In the mean time, please install the dep tool and run dep ensure before doing go build -o azcopy.

[123dev]

Thanks zezha for a very quick reply.

followed your suggestion, and at the step of go build -o azcopy
I'm getting

# github.com/Azure/azure-storage-azcopy/vendor/github.com/jiacfan/keychain
vendor/github.com/jiacfan/keychain/corefoundation_1.10.go:55:33: cannot use nil as type _Ctype_CFAllocatorRef in argument to _Cfunc_CFDataCreate
vendor/github.com/jiacfan/keychain/corefoundation_1.10.go:81: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal
vendor/github.com/jiacfan/keychain/corefoundation_1.10.go:118:41: cannot use nil as type _Ctype_CFAllocatorRef in argument to _Cfunc_CFStringCreateWithBytes
vendor/github.com/jiacfan/keychain/corefoundation_1.10.go:153:232: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal

Thanks

[zezha-msft]

Hi @123dev, please verify that you have the latest code from our master branch. This should work. 😄

[zezha-msft]

Our CI machine seems to be having the same errors as you, but I couldn't reproduce it locally yet. I'm investigating now.

[jiacfan]

Hi, @123dev

Thanks for reporting the issue!

Are you using go 1.11? There is change in go which could lead the build failure mentioned in one of azcopy's dependency, we have fixed that issue and updated dependency in azcopy(dev branch). Now, dev branch can pass build successfully in my local machine with go 1.11, would you like to have a try?

Please note to checkout dev branch's code, run dep ensure again to get latest dependencies, and then run go build.

Thanks,
Jiachen

[123dev]

Hi jiacfan
Thanks for the follow up and the fix.

Indeed I'm using go 1.11
dev build worked, thank you.

Quick question, this azcopy is not backward compatible with the original azcopy?
We already have code that issues azcopy (multi-platform) with parameters such as
/Source
/DestKey
/Dest

And were thinking that this would just be a drop in replacement that supports copying to $web

Thanks

[zezha-msft]

Hi @123dev, the new AzCopy(Version 10) is indeed NOT backward compatible with the current AzCopy(Version 8). However, the new tool is much simpler to use, to see examples, please use ./azcopy copy -h:

Examples:
Upload a single file:
  - azcopy cp "/path/to/file.txt" "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/destination/directory/or/file]"

Upload an entire directory:
  - azcopy cp "/path/to/dir" "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/destination/directory]" --recursive=true

Upload files using wildcards:
  - azcopy cp "/path/*foo/*bar/*.pdf" "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/destination/directory]"

Upload files and/or directories using wildcards:
  - azcopy cp "/path/*foo/*bar*" "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/destination/directory]" --recursive=true

Download a single file:
  - azcopy cp "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/source/file]" "/path/to/file.txt"

Download an entire directory:
  - azcopy cp "https://[account].blob.core.windows.net/[existing-filesystem]/[path/to/source/dir]" "/path/to/file.txt" --recursive=true

The new version also works with SAS tokens only, so that it's much more secure and you don't have to expose your Storage Key. To easily generate an account or container token, please use Storage Explorer, and include the token in the URL that you provide to the copy command.

Please let me know if you have any question. To see what commands are available, you can always view the help pages by ./azcopy -h.

[123dev]

Thank you zszha for the follow up and confirmation of not being backward compatible.
And thanks for the links.

I am able to create SAS and use azcopy to copy files including to $web
Many thanks.

This issue can be closed.

[zezha-msft]

@123dev Awesome!

Please let us know if you encounter any other issue using the new tool.

[123dev]

Hi zezha,

Considering that this new version 10 is not backward compatible with the previous version 8,
I tried to build version 10 on windows as I want to make all our platforms run the same tool the same way.

and encountered the following error.

# runtime
C:\U\Go\src\runtime\lfstack_amd64.go:16:6: lfstackPack redeclared in this block
        previous declaration at C:\U\Go\src\runtime\lfstack_64bit.go:33:45
C:\U\Go\src\runtime\lfstack_amd64.go:20:6: lfstackUnpack redeclared in this block
        previous declaration at C:\U\Go\src\runtime\lfstack_64bit.go:37:33
C:\U\Go\src\runtime\map.go:64:2: bucketCntBits redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:62:18
C:\U\Go\src\runtime\map.go:65:2: bucketCnt redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:63:23
C:\U\Go\src\runtime\map.go:76:2: maxKeySize redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:72:17
C:\U\Go\src\runtime\map.go:77:2: maxValueSize redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:73:17
C:\U\Go\src\runtime\map.go:82:2: dataOffset redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:81:4
C:\U\Go\src\runtime\map.go:91:2: empty redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:87:19
C:\U\Go\src\runtime\map.go:92:2: evacuatedEmpty redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:88:19
C:\U\Go\src\runtime\map.go:93:2: evacuatedX redeclared in this block
        previous declaration at C:\U\Go\src\runtime\hashmap.go:89:19
C:\U\Go\src\runtime\map.go:93:2: too many errors

Steps I took.
Installed Go from here
Installed dep from here
Setup environment
Ran dep ensure
and then go build -o azcopy.exe

Also I would like to know is if azcopy binary is self contained or it has go dependencies?
Can I just copy the binary to another computer and execute it without having to install dependencies?
if no? what do I need to install?

Thanks

[zezha-msft]

Hi @123dev, it looks like your Go environment has some problem, please follow the instructions here to verify that Go was installed properly.

To build AzCopy, you could CD into the directory of azure-storage-azcopy, and either:

1. Run dep ensure followed by go build -o azcopy.exe.
2. Or install Docker, and run make all, which will produce 3 executables, one for Mac, one for Linux, and one for Windows.

And yes, the AzCopy binary is self-contained: you can copy the binary to another computer and execute it just like that.

I apologize for the inconvenience. We should have the download links ready very soon. Thanks!

[123dev]

Thanks zezha for all the assistance,
I re-ran the Go installer and chose repair, and it repaired it, I am no able to build it on windows.
Sorry for the false alarm, my inexperience with Go is the culprit :)

Thanks

[zezha-msft]

Hi @123dev, it's great to hear that you got it working!

Please let us know if you have any feedback on the tool. We are actively working on it to improve the user experience!

[123dev]

Sure zezha,

One thing I came across is that wherever I issue the command azcopy there are two log files in that directory.
azcopy.log which is always 0 bytes long
and .log

Any way to suppress the logging? or at least control where they are output and what name?

So far it's working great on windows and Mac
Great work.

Thanks

[zezha-msft]

Hi @123dev, thanks for the feedbacks!

We have fixed the log file location issue on the dev branch, we'll publish it soon!

In the future, the log files will be placed under a folder called .azcopy in the user's home directory, so that it's still visible, in case the user wants to find them.

[123dev]

Awesome, thanks

[eddiedozier]

@zezha-msft Thank you guys so much for this!