all: change package manager from dep to go mod

[leighmcculloch]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

Summary

Converts the mono repo from using the dep package manager to using the go mod that has become the standard dependency manager in recent releases of Go.

Goal and scope

Make the monorepo more accessible to Go developers using standard Go tooling. Modules will be enabled by default in Go 1.13, and were enabled by default when not using a GOPATH in Go 1.11+.

Close #1634

Summary of changes

• Add a go.mod file that contains the same dependencies and dependency versions/hashes/sources.
• Add a go.sum which is the smallest set of hashes required to validate that the dependencies at those versions remain consistent.
• Add a go.list which is a way for us to visibly see when our dependencies change. Note: This is not a standard file, but is something we can use to validate reproducibility until the go mod commands provide something similar.
• Update CircleCI and TravisCI builds so that they succeed
• Update CircleCI and TravisCI builds so that they fail if any of the above go.* files are out of date.
• Update documentation within the repository
• Test that other projects can import this repo. (This will be much easier, unfortunately, to test after merging, but I am currently attempting to test this before merging.)
• Remove the existing dep files, Gopkg.toml and Gopkg.lock. (This will occur just before merging.)

Follow ups after merging

• Update https://github.com/stellar/integration-tests
• Work with @brahman81 to update apt repository builds
• Update release documentation and other documentation

Known limitations & issues

This change bumps our minimum supported version of Go to 1.11.4. Versions prior to that version contained a bug when Modules are in use that prevented the importing of some packages. There are more details about the issue here: golang/go#30446 (comment).

What shouldn't be reviewed

It is unlikely going to be possible for you to review the go.mod, go.sum, and go.list files. The first two were auto-generated by go mod init and go mod tidy, and by a dozen subsequent go get commands to tell Go to use specific versions for specific libraries. The third is a capture of the output of go list -m all which lists all the dependencies of the project.

I used a tool I wrote yesterday deplockgomoddiff to compare our Gopkg.lock with the output of go list -m all. go list -m all is the recommended command mentioned here for validating dependencies when migrating to Modules to ensure we have alignment. The recommendation is to manually compare the output of that command with the dependencies we expected, but due to our large number of repositories it was impractical to do manually, hence the tool.

[leighmcculloch]

After sanity checking vendor directories generated by both tools, I've identified six packages that are different. I missed them because of a bug in deplockgomoddiff (leighmcculloch/deplockgomoddiff@872f6d2).

• github.com/rcrowley/go-metrics
• golang.org/x/crypto
• golang.org/x/net
• golang.org/x/sys
• golang.org/x/text
• gopkg.in/yaml.v2

I'll get these adjusted before this is merged in, the diff here should change very little.

[bartekn]

LGTM. Minor issues and questions.

[leighmcculloch]

Including the changes in unmerged PRs #1685 #1688 and comparing vendor directories using the pattern below, the vendored dependencies appear essentially same. Capturing commands and output below as a reference if I want to reattempt it.

---

1. Get a vendor directory generated by each:

$ rm -fr vendor
$ dep ensure -v
$ mv vendor ~/devel/vendor-dep
$ go mod vendor
$ mv vendor ~/devel/vendor-mod

2. Find the list of files that are different:

$ diff -qr vendor-dep vendor-mod
...

3. The output was long, but it showed that the only difference was vendor-dep had extra files that vendor-mod didn't have. It's possible Modules is smarter about what files to include or exclude based on things like build constraints, which packages are actually used, etc, so I narrowed it down to only .go and .s (assembly) files, and reduced the output to a list of packages. This is what I get:

$ diff -qr ~/devel/vendor-dep ~/devel/vendor-mod | grep '\.\(go\|s\)$' | cut -d':' -f1 | sort -u | sed 's$^Only in /home/leighmcculloch/devel/vendor-dep/$$' | xargs -I {} echo {}       
github.com/aws/aws-sdk-go                                                                                                                                                                
github.com/aws/aws-sdk-go/service                                                                                                                                                        
github.com/btcsuite/btcd                                                                                                                                                                 
github.com/btcsuite/btcd/btcec                                                                                                                                                           
github.com/btcsuite/btcutil/base58                                                                                                                                                       
github.com/ethereum/go-ethereum/core                                                                                                                                                     
github.com/klauspost/compress/flate                                                                                                                                                      
github.com/klauspost/cpuid                                                                                                                                                               
github.com/lib/pq/oid                                                                                                                                                                    
github.com/stretchr/testify                                                                                                                                                              
golang.org/x/crypto/ssh                                                                                                                                                                  
golang.org/x/net/html/atom                                                                                                                                                               
golang.org/x/net/publicsuffix                                                                                                                                                            
golang.org/x/sys/unix                                                                                                                                                                    
golang.org/x/text                                                                                                                                                                        
golang.org/x/text/encoding/charmap                                                                                                                                                       
golang.org/x/text/encoding/htmlindex                                                                                                                                                     
golang.org/x/text/encoding/internal/identifier                                                                                                                                           
golang.org/x/text/encoding/japanese                                                                                                                                                      
golang.org/x/text/encoding/korean                                                                                                                                                        
golang.org/x/text/encoding/simplifiedchinese                                                                                                                                             
golang.org/x/text/encoding/traditionalchinese                                                                                                                                            
golang.org/x/text/internal                                                                                                                                                               
golang.org/x/text/internal/language                                                                                                                                                      
golang.org/x/text/internal/language/compact                                                                                                                                              
golang.org/x/text/language                                                                                                                                                               
golang.org/x/text/secure                                                                                                                                                                 
golang.org/x/text/unicode                                                                                                                                                                
golang.org/x/text/unicode/bidi                                                                                                                                                           
golang.org/x/text/unicode/norm                                                                                                                                                           
google.golang.org/appengine

4. We can then take that list and compare it to the list of packages that go list returns to see which dep are including that are referenced by the code, or by a dependency we're using.

$ join <(diff -qr ~/devel/vendor-dep ~/devel/vendor-mod | grep '\.\(go\|s\)$' | cut -d':' -f1 | sort -u | sed 's$^Only in /home/leighmcculloch/devel/vendor-dep/$$') <(go list -deps -f '{{if not .Standard}}{{.ImportPath}}{{end}}' all | sort -u)
github.com/btcsuite/btcd/btcec
github.com/btcsuite/btcutil/base58
github.com/klauspost/compress/flate
github.com/klauspost/cpuid
github.com/lib/pq/oid
golang.org/x/net/html/atom
golang.org/x/net/publicsuffix
golang.org/x/sys/unix
golang.org/x/text/encoding/charmap
golang.org/x/text/encoding/htmlindex
golang.org/x/text/encoding/internal/identifier
golang.org/x/text/encoding/japanese
golang.org/x/text/encoding/korean
golang.org/x/text/encoding/simplifiedchinese
golang.org/x/text/encoding/traditionalchinese
golang.org/x/text/internal
golang.org/x/text/internal/language
golang.org/x/text/internal/language/compact
golang.org/x/text/language
golang.org/x/text/unicode/bidi
golang.org/x/text/unicode/norm

---

Modules is definitely more aggressive at pruning packages and files that aren't required, so it isn't surprising we'd have some files not being included. I don't yet fully understand why the above shows packages not being included when they're in the go list, but it's possible the go list command is pulling packages that don't actually get compiled into the application. I suspect that because I sampled the first dependency btcec and I couldn't find it referenced via the code paths we use, or the two packages dependent on it that we import. There's not enough documentation published about the intricacies of go list and go mod vendor that explains why the above is happening.

I think the above is still okay for the following reasons:

• The entire source is capable of building and all tests run and pass. This is a strong signal that Modules has all the packages and files it needs in its module checkout which has the entire packages and not a pruned version of them like what we get with the vendor directory.
• The only diffs between the two vendors were files pruned out by Modules and there were no files that contained different contents that's strong signal that both dep and Modules are downloading the same dependencies. If dependencies were different we'd expect to see some changed files.

---

This test I documented above is just to compare vendor directories and we shouldn't plan to use the vendor directory. go mod vendor is so aggressive at pruning that it breaks some cgo packages that aren't strictly storing files where they need to be storing them for the Go tool to identify that they are required. The github.com/ethereum/go-ethereum package is one package that doesn't do this properly yet.