Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: partial vendoring #52604

Open
johnwmstevens opened this issue Apr 28, 2022 · 10 comments
Open

proposal: cmd/go: partial vendoring #52604

johnwmstevens opened this issue Apr 28, 2022 · 10 comments
Projects
Milestone

Comments

@johnwmstevens
Copy link

@johnwmstevens johnwmstevens commented Apr 28, 2022

The go mod vendor capability is great for insulating a working program from changes made in imported modules.

It would benefit from having the ability to download only those modules that are imported from external entities.

Within a single entity, every project that uses a popular module and the vendor capability will store a copy of the same code. With multiple projects referencing this popular module, multiple copies will be stored in the owning entities' source code repository.

Modifying the vendoring capability to download only modules owned by external entities would be a useful change. Reuse of the GOPRIVATE env var might make sense, but if overloading the semantics of this variable is problematic a new one could be added or a switch added to the command.

@gopherbot gopherbot added this to the Proposal milestone Apr 28, 2022
@seankhliao seankhliao changed the title proposal: affected/package: proposal: cmd/go: partial vendoring Apr 28, 2022
@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 28, 2022

previously #30240

@ianlancetaylor ianlancetaylor added the GoCommand label Apr 29, 2022
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 29, 2022

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Apr 29, 2022
@rsc
Copy link
Contributor

@rsc rsc commented May 4, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals May 4, 2022
@bcmills
Copy link
Member

@bcmills bcmills commented May 4, 2022

I think partial vendoring is a fine concept, but I wouldn't want to key it on GOPRIVATE: some folks will want to vendor only the private dependencies, while others will want to vendor everything but those.

However, there is one added wrinkle: the vendor directory does not include go.mod files. It was hit-or-miss until CL 315410 (#42970), and now it's completely consistent but in the direction of not including them. So we wouldn't be able to resolve the full module graph in the presence of partially-vendored dependencies (because we wouldn't have access to the requirements of those dependencies unless ).

...but that's probably ok, given lazy module loading! We would at least still be able to load the dependencies from modules whose versions are listed explicitly in the go.mod file, which should cover all of the packages that would have been available with a full vendor tree anyway.

@johnwmstevens
Copy link
Author

@johnwmstevens johnwmstevens commented May 4, 2022

I'm not fixated on any particular solution, GOPRIVATE was just an off the top of my head suggestion.

My primary goal is to be able to use the vendor capability in an organizational entity (MyOrg) to draw a line between modules "owned" by other entities (IOW: vendors) and those owned by MyOrg.

This, in order to protect the viability of a critical Go program that depends on stuff owned by "someone else" where it is possible that that other stuff may just up and disappear.

The secondary benefit is being able to avoid multiple copies of MyOrg modules in MyOrg's SCRaM system, which should make rolling out critical bug fixes in heavily reused MyOrg modules safer and easier.

@flibustenet
Copy link

@flibustenet flibustenet commented May 4, 2022

Vendoring only private modules would be fine. On some PAAS (Cloud Run for example) it's difficult to include private modules, it could be find to vendor only them.

@hherman1
Copy link

@hherman1 hherman1 commented May 11, 2022

Is the point of this to save disk space? Why is it useful?

@flibustenet
Copy link

@flibustenet flibustenet commented May 11, 2022

@hherman1 for my use case i can commit my very little private module but i would not like to pollute my code history with all the other dependencies.

@johnwmstevens
Copy link
Author

@johnwmstevens johnwmstevens commented May 11, 2022

Is the point of this to save disk space? Why is it useful?

Copying code owned by entity B protects entity A from being unable to compile their Go program because entity B deleted their repository.

Copying one's own code into multiple projects does indeed waste disc space, but also makes search, tracking, management and maintenance more complicated.

Programs owned by A that reuse a module also owned by A become more difficult to maintain by requiring a manual step to update the copy of the reused module in each project.

Assume, say, fifty programs owned by A that all reuse module X which is also owned by A. Fifty copies of that module exist in the SCRaM system and there you have your wasted disc space.

Also, to update each program when a bug in module X is fixed is easy if all fifty projects refer to the source repository of module X, more difficult when each project must be manually updated to copy the new module X code into all fifty projects.

Certain kinds of search in that SCRaM system will also return fifty results, not just one, because of the multiple copies.

The ability to distinguish between owned and not owned code when making a "vendor" copy also assists in management of other entity bounded operations, such as license change, tracking and management.

@hherman1
Copy link

@hherman1 hherman1 commented May 11, 2022

If I understand correctly the manual step of updating censored code doesn’t seem so different from the manual step of incrementing the dependency version in go.mod, so it seems like you would still have the problem you’re describing even with partial vendoring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

8 participants