Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: Building Go projects without assuming remote repository is accessible #24054

Closed
ChimeraCoder opened this issue Feb 23, 2018 · 7 comments

Comments

@ChimeraCoder
Copy link
Contributor

commented Feb 23, 2018

It appears that vgo drops the vendor/ directory. One of the advantages of the vendor/ directory in previous Go dependency management tools is that a project can be built with nothing but a local copy of the repository and the standard library.

There are two situations that I have in mind:

  1. Project A depends on github.com/b/c. The author of github.com/b/c deletes the repository (or, alternatively, has their account suspended by Github).

  2. Project A depends on github.com/b/c. Project A needs to be built in a build system that does not have external network access to Github (ie, behind a private VPN).

With the vendor/ directory, both of these situations are readily solved, because github.com/b/c is vendored inside the repository for Project A. It appears that, with vgo, the absence of the vendor directory means that we need to assume github.com/b/c will exist in perpetuity and will be accessible by the build system.

In theory, it would be possible to build a cache of the packages, but this becomes resource-intensive. For personal projects, there really isn't an automated way to defend against (1). For (2), a company needs to invest resources into building a system that will cache packages locally and make them available. Not all companies that rely on Go will have the ability to do that - mine included.

Our company (Stripe) is primarily not a Go shop, but we do rely on Go for several key pieces of our infrastructure. One of the major advantages of Go which allowed it to gain adoption internally is that, unlike the other languages we use, it required a very minimal build infrastructure - all we needed to do was set up the standard Go toolchain. I don't think that we'd be able to build an internal cache or proxy specifically to support Go builds.

How do we address these issues in a vgo-based workflow?

@gopherbot gopherbot added this to the vgo milestone Feb 23, 2018

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

commented Feb 23, 2018

According to https://research.swtch.com/vgo-tour
vgo vendor generates a vendor directory, however by default vgo will not use it for building.
To do that you would have to do a replace e.g. replace "rsc.io/quote" v1.5.2 => "../quote"

@ChimeraCoder

This comment has been minimized.

Copy link
Contributor Author

commented Feb 23, 2018

From the proposal, it seems that the vendor directory is currently included for backwards-compatibility, but it's not intended to be part of vgo's design.

The replacing workflow looks like a way to address the issue of necessary hot-patches to upstream code. That's another important benefit of our current vendoring workflow, so I'm glad that vgo supports something analogous as well!

But I don't think that's a solution here. From the post, it seems like the assumption is that ../quote would either use a separate repository for version control, or that both would be versioned by the same repository whose root is the parent (or an ancestor) of both. Neither seems ideal. The former means we'd have to create a separate, internal fork of every dependency that we import, which is cumbersome and unwieldy. The latter means that we'd have to fundamentally change the structure of all the Go projects we develop, which would be a huge upheaval - it sounds like it'd essentially mean that each project's repository vendors its entire GOPATH (excluding the standard library) at the top-level, instead of containing the vendored code within the vendor directory. That'd get confusing and hard to manage very quickly.

It also means that, every time we want to update a dependency, we'd have to update it in two steps, not one - we'd first have to update our internal fork (whether in the same repository as our main project or a different one) and then we'd have to update the minimal version specified in go.mod.

The two-step update process adds friction, which is key to the second scenario described in the blog post above (preventing people from accidentally reintroducing known regressions). In that scenario, adding friction is the desired outcome, but I don't think we want to impose the same level of friction every time we update a dependency.

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Feb 25, 2018

First, let me +1 the point that having to set up services is a significant barrier to adoption in companies. I worked for one of the biggest Go shops in a sense, and even there having to set up a service, getting it on the right networks for both developers and CI, and maintaining it, would have been a problem and friction.

Second, it looks to me like a file:// GOPROXY solves, if inelegantly, the problem. There's probably tooling we can build to make that work better, for example a simple tool to build the directory. If stored inside, say, vendor/_vgo it would also be excluded from the module builds. Maybe vgo could even look there by default for cached modules.

I, too, am a fan of self-contained projects. In a way they are the static binaries of source trees.

@ChimeraCoder

This comment has been minimized.

Copy link
Contributor Author

commented Feb 26, 2018

@FiloSottile to make sure I understand: in that case, the file:// directive would go inside the go.mod? So that means that the choice to use local copies of the file has to be made on a per-project basis (rather than based on the setup of the build environment)?

@FiloSottile

This comment has been minimized.

Copy link
Member

commented Feb 26, 2018

No, I'm talking about the GOPROXY environment variable.

@FiloSottile

This comment has been minimized.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2018

  1. Vendor at top level of module will keep working (this is a change from original docs).
  2. We'll make proxies very very easy to set up, including as file:/// URLs pointing at a shared file server you have in your company, or as https:/// URLs to static file servers.

@rsc rsc closed this Mar 30, 2018

@golang golang locked and limited conversation to collaborators Mar 30, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants
You can’t perform that action at this time.