Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: Ensuring VGO allows people to build enterprise support #24811

Closed
gaffo opened this issue Apr 11, 2018 · 4 comments

Comments

@gaffo
Copy link

commented Apr 11, 2018

VGO Caching Strategy and Enterprise Development

Hello,
I wanted to bring up some concerns I have around the vgo cache and doing Enterprise / Lots of Micro services development. I work in an environment where I have a lot of services which are only related via their RPC Apis, Twirp in our case.

Can't use internet go repositories directly
Like many enterpries, pulling code live from github or the internet is not really allowed:

  • I can't control when my access to github / etc is comprimised.
  • I can't have my builds fail when github is down.
  • I have to vet / sign off on all libraries in use and vet their licenses.
  • I need to have security review and audits of all the point releases of libraries I use.

Generally somthing like this is done via a tool like Artifactory and only allowing vetted libraries into it.

This can be solved with firewall in some cases. However, tools like maven solve this by having you list repositories and only hitting those with a default of mvncentral (I think) if no repos are configured.

Versions of libraries aren't the same across all projects
I may do a bad job of explaining this but here goes. Often when developing with lots of services you will have many parallel unrelased versions of libraries. So v1.0.3 would mean completely different things depending on using them. Lets say for instance we have a company library loglib that is currently released (internally) to v1.0.2. Now lets say there are two projects who are consuling loglib, chatty and watchy. Chatty has an issue with loglib that they have to fix and are testing that branch. Obviously this is v1.0.3b. Watchy adds a new feature that they also want to bake, this is also I guess v1.0.3b but is totally different code. Whoever gets their code out first really is the one who releases 1.0.3, but internally they both need to reference unreleased versions of the package. Worse they may need to share these unreleased versions with other teams.

Now even worse, lets say I'm a dev for loglib, and working on both watchy and chatty fixes. I need to have a go development env for watchy setup, and a different one for chatty. If I use unified system cache then these 2 projects will be stepping on each other and really these unreleased versions don't resolve across gopaths.

This may sound wierd but artifactory lets you setup project specific views into packages.

In general it might make more sense to tie the cache to the "compilation directory".

Tho I do think that the ability to setup the cache directory with a envvar is a workaround. Just may be awkward and lead to many issues.

Go Version Dependent?
Also a mistake I think glide makes is that the cache is tied to the version of go and system wide which causes major issues when swapping back and forth between go versions.

Inability to provide your own cache system
At my current work and at some previous places we needed the ability to control caching ourselves for various reasons, such as the multi-version one above. Another was at a game studio where we had distributed build engines. Providing a standard cache layout structure or a standard interface for the cache would allow specialized cache systems and resolvers to be placed in where needed.

@gopherbot gopherbot added this to the vgo milestone Apr 11, 2018

@dlsniper

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2018

The MVS proposal, #24301, supports a proxy server which would allow all of the issues above to be solved. Do I miss anything?

@kardianos

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2018

Hi @gaffo
If you haven't already read the following documents, I would recommend you do so: https://research.swtch.com/vgo . It addresses many of the concerns you raise. Let me try to address some of them specifically here:

Can't use internet go repositories directly

That's great! vgo is designed to work in your environment. You'll want to deploy a company proxy probably (see below).

Versions of libraries aren't the same across all projects

That is interesting. vgo doesn't upgrade to point versions automatically (unlike most other systems), which means rather then using un-released versions for some projects, you can actually tag these point releases as real releases, then only update projects to that version that require it. Please refer to MVS (minimal version selection).

Go Version Dependent?

vgo works on Go code. What you are talking about is the build cache I think. This is much improved in go1.10 (like night and day better).

Inability to provide your own cache system

External cache systems are part of the design. There is an initial implementation of one here: https://github.com/gomods/athens


It sounds like vgo is already well designed to meet all of your needs you listed here. Even though some of the tools like vgo and athens are immature early versions, your needs are well known, common, and designed to have first class support in vgo.

@gaffo

This comment has been minimized.

Copy link
Author

commented Apr 12, 2018

@dlsniper, @kardianos I will take a look tomorrow. Thanks for the pointers to the data.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2018

We're aware of all these concerns, and I believe the current module design does allow people to build the enterprise support you are describing. Over time I expect that some of it will be standard instead of having to be built separately.

I'm going to close this issue, though, since there's not a specific bug to fix.

@rsc rsc closed this Jul 6, 2018

@golang golang locked and limited conversation to collaborators Jul 6, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants
You can’t perform that action at this time.