go_repository: git repositories should be stored in the cache

[jayconrod]

If a go_repository rule is invalidated but @go_repository_cache is not, we shouldn't need to clone a Git repository again. The first time we fetch a repository, we could clone it and store a zip file in the cache. When go_repository is invalidated, we would just need to extract the cached zip.

[Globegitter]

There is also a different approach used here: bazelbuild/bazel#7424 and some relevant discussion here: https://groups.google.com/forum/#!searchin/bazel-dev/buchgr%7Csort:date/bazel-dev/7N_6-RbqBf4/bOggKYkUBgAJ

[jayconrod]

Nice. I hope that gets merged at some point. I filed bazelbuild/bazel#5086, but I've given up hope of it ever being implemented.

[mariusgrigoriu]

I'm finding that go_repositories are being fetched when not expecting it. For example, I make a trivial change to the WORKSPACE file, such as inserting whitespace at the end of the file, and it fetches the go_repository.

Additionally, I'm finding the same repository being downloaded multiple times in a row, extending fetch times. This is mostly evident when downloading in urls mode without a sha256, although I believe this happens with a sha256 and in git mode just based on the duration of the fetch.

Are these all symptoms of this issue or should I go open another issue?

[jayconrod]

@mariusgrigoriu Sorry for the delay. Was at GopherCon last week.

For example, I make a trivial change to the WORKSPACE file, such as inserting whitespace at the end of the file, and it fetches the go_repository.

This may be working as intended. Gazelle, as run by go_repository, reads the WORKSPACE file for configuration. So go_repository rules need to be re-evaluated when WORKSPACE changes. However, the module cache should not be cleared, so go_repository in module mode shouldn't need to download anything.

Additionally, I'm finding the same repository being downloaded multiple times in a row, extending fetch times. This is mostly evident when downloading in urls mode without a sha256, although I believe this happens with a sha256 and in git mode just based on the duration of the fetch.

Use a sha256. In HTTP mode, HTTP downloads are cached by Bazel. The cache key is the expected sha256, which means downloads without sha256 are not cached. It's also important to do this to ensure builds are authentic and reproducible.

VCS downloads are currently not cached, which is this issue. I'd strongly encourage use of module mode instead though.

[mariusgrigoriu]

Using modules sounds good when they work. We're importing Terraform and parts of k8s, neither of which seem to play nicely with module mode. Since k8s already uses Bazel, I think switching to http_archive is a good approach. Seems that terraform should use go_repository in http mode until other issues regarding terraform are resolved.

[jayconrod]

Be aware that the the main Kubernetes repo, k8s.io/kubernetes (a.k.a. github.com/kubernetes/kubernetes) is not intended to be imported by other repos. Most other repos in that org should work though.

[mariusgrigoriu]

Understood. This is all because we're consuming e2e tests. Not sure we can do much until kubernetes/kubernetes#74352 moves the e2e framework into staging. (Apologies for hijacking this thread.)

[evie404]

while not a fix, most go dependencies we use are hosted on github which supplies tar archives over http given a sha. we wrote this as a wrapper to convert existing go_repository with git tags into using urls which allows for caching: https://gist.github.com/rickypai/abadfd810ba13ad295f9987348ffc6af

doesn't work with the more recent go mod stuff though

[jayconrod]

@rickypai Archives served by GitHub do not have stable SHA-256 sums. They haven't changed in a couple years, but it's broken us in the past. Use at your own risk.

[kalbasit]

@rickypai Archives served by GitHub do not have stable SHA-256 sums. They haven't changed in a couple years, but it's broken us in the past. Use at your own risk.

On NixOS side, we have been using GitHub's archives for a few years now with no issues with regards to the sha256 stability:

• https://github.com/NixOS/nixpkgs/blob/bcfa14af1544367bbb2245fc639b470124514385/pkgs/top-level/all-packages.nix#L353
• https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgithub/default.nix

$ git grep 'fetchFromGitHub {' | wc -l
5266

[jayconrod]

@kalbasit I don't think they've changed anything since fall of 2017. However, I spoke with GitHub support ~6 months ago. They're aware of the issue, but they confirmed archives returned from those endpoints are not guaranteed to have stable hashes.

These breaks were really painful. If the hashes change, it retroactively breaks deterministic builds that depend on the old hashes. At the time, it broke every version of rules_go, since we were using http_archive with those endpoints.

[kalbasit]

@kalbasit I don't think they've changed anything since fall of 2017. However, I spoke with GitHub support ~6 months ago. They're aware of the issue, but they confirmed archives returned from those endpoints are not guaranteed to have stable hashes.

These breaks were really painful. If the hashes change, it retroactively breaks deterministic builds that depend on the old hashes. At the time, it broke every version of rules_go, since we were using http_archive with those endpoints.

What do you think about introducing a flag to the update-repos to get it to use http_archive for those who are willing to take that risk?

[jayconrod]

@kalbasit Not sure I follow. update-repos doesn't emit http_archive rules at all. It also no longer emits go_repository rules that clone git repositories (making this issue mostly moot). In the common case, go_repository downloads module zip files from https://proxy.golang.org (or whatever GOPROXY is set to).

[mariusgrigoriu]

We experienced several changing sha256 hashes over the last few months for a subset of archives. In one case, a hash change was only experienced by some people on the team, and it eventually reverted.

[kalbasit]

@kalbasit Not sure I follow. update-repos doesn't emit http_archive rules at all. It also no longer emits go_repository rules that clone git repositories (making this issue mostly moot). In the common case, go_repository downloads module zip files from https://proxy.golang.org (or whatever GOPROXY is set to).

What version of gazelle, rules_go and Go should make this work? I'm currently using Bazel 0.28.0, rules_go at 0.20.2, gazelle 0.19.0 and Go 1.12.9 and I'm still showing it Git cloning.

Definitely an archive from GOPROXY would help, does it go in the cache as described in https://bazel.build/designs/2016/09/30/repository-cache.html?

[jayconrod]

What version of gazelle, rules_go and Go should make this work?

@kalbasit That's a new enough version of Gazelle. Make sure none of your go_repository rules have commit, tag, or remote attributes. Gazelle won't generate go_repository rules with those anymore, but that doesn't change the semantics of old rules. Use version and sum (for module mode) or urls and sha256 (for HTTP mode). See the go_repository documentation.

Definitely an archive from GOPROXY would help, does it go in the cache as described in https://bazel.build/designs/2016/09/30/repository-cache.html?

Modules zips don't get stored in Bazel's cache, but they do get stored in a separate cache within an internal repository. It's a bit of a hack, but it means they don't need to be downloaded whenever WORKSPACE changes. Bazel can't cache them for the same reason as the GitHub archives: module zips don't promise SHA-256 stability. The sums are hashes of the contents, not of the zip files themselves.

[kalbasit]

They are using version and sum so they should be good then. Maybe it's not cloning and I think it is? Is there a way to enable debug to see what's going on behind the scenes?

Here's my go.bzl that I load from the workspace: https://gist.github.com/kalbasit/9ad680ece5d6904a4e96635a38cf18d6

[jayconrod]

You can run bazel info output_base, then look at subdirectories of external within that. Those are the directories where repository rules get evaluated. Look for a .git subdirectory in there.

There are some git_repository rules declared in go_rules_dependencies. That may be what you're seeing.