Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: decide replace, exclude wildcard behavior #26344

Open
rsc opened this issue Jul 12, 2018 · 4 comments

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Jul 12, 2018

In CL 122400 @bcmills pointed out that we haven't defined what

replace (
	example.com/foo v1.0.0 => elucidation.com/foo v1.1.0
	example.com/foo => elucidation.com/foo v1.2.0
)

means for example v1.0.0. Obviously it must mean you get elucidation v1.1.0 not v1.2.0. But that's not documented nor implemented.

Bryan also raised the question of whether to allow

exclude example.com/foo

to make it completely unavailable. Probably we should.

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018
@rsc rsc added the modules label Jul 12, 2018
@rsc rsc changed the title x/vgo: decide replace, exclude wildcard behavior cmd/go: decide replace, exclude wildcard behavior Jul 12, 2018
@rsc

This comment has been minimized.

Copy link
Contributor Author

@rsc rsc commented Aug 17, 2018

Will have to solve in Go 1.12. Until then, "don't do that."

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 17, 2018
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 15, 2019

Part of the problem here is that there are two different possible meanings for a “wildcard replace”.

One is, “replace every single version of this module with this specific source code”. That destroys all of the transitive requirements that appear anywhere in the module graph, which are important in general (see #31248), so it should be used only as a last resort. (For example, it could be used if the upstream origin is completely gone and most versions are not cached in any proxy, or if the origin is completely unusable due to a licensing or security issue and needs to be expunged completely from the module graph.)

The other — from what I can tell, by far the more common case — is “once you have selected a version of this module, pretend that this replacement is an even higher version and use it instead”. That's the replace you want, for example, when preparing upstream patches (#27542).

CC @jayconrod @thepudds @Helcaraxan

@Helcaraxan

This comment has been minimized.

Copy link
Contributor

@Helcaraxan Helcaraxan commented Oct 17, 2019

Adding my own brain-dump to the topic, mixed with some thoughts on adjacent issues (who said "multi-module repos"?).

  • On the exclude mydomain.com/foo/bar to prevent any use of that module origin.

Supporting this would definitely make sense for me. It would provide native support for (security) blacklisting of specific hosts and I would even extend it to not necessarily be a module path but more a pattern: exclude github.com (or exclude github.com/* TBD) would prevent any use of sources provided by GitHub repos. Counter-point to my position here would be that such blacklisting should be managed via a custom module-proxy... but small OSS projects will not have one and the exclude could thus be a "poor man's alternative".

  • With respect to the original example:
replace (
	example.com/foo v1.0.0 => elucidation.com/foo v1.1.0
	example.com/foo => elucidation.com/foo v1.2.0
)

It does seems sensible to make this result in replacing v1.0.0 with v1.1.0 instead of v1.2.0. However we could also argue that finding this in a go.mod should simply result in an error. If we were to turn it into an error I would also argue we add an error when a given replace statement is not used to construct the final build list. For cases where people do want to keep such unused lines around we can implement the // keep comment approach also used in tools such as gazelle to disable the new errors.

  • On the general semantics and use-cases of the "wildcard" replace statement:

For example, it could be used if the upstream origin is completely gone and most versions are not cached in any proxy, or if the origin is completely unusable due to a licensing or security issue and needs to be expunged completely from the module graph.

This ties somewhat back to my note about the exclude above. If you are expunging an entire module I would very much suspect that you will also very likely not be using any forks or other replacements for it: you would start using an entirely different dependency with its own module path. The generalised exclude would be enough to handle such a scenario. For the case where you are in the process of expunging the module and you need to use a temporary fork users should fall back on a version-specific replace statement in addition to the exclude. This would also have the benefit of encouraging people to consider moving away from the fork as they would otherwise need to continuously update the replace statement at each change to keep builds working.

The other — from what I can tell, by far the more common case — is “once you have selected a version of this module, pretend that this replacement is an even higher version and use it instead”. That's the replace you want, for example, when preparing upstream patches (#27542).

The related use-case, although definitely valid, is not one I have personally encountered much yet. In practice this particular replace semantic is used in a much wider fashion. More in line with "pretend that this replacement is this specific version and use it instead".

One example is the way Kubernetes uses it to achieve the continuous pinning of versions for their dependencies. They rely on the wildcard version but they could work as well with the version-specific statement as well, but this would complicate their already complex automation around module-maintenance even further.

Another similar and frequent use is when migrating (old) projects to modules, whether they have been using third-party dependency management or simply relying on GOPATH. Projects frequently encounter the issue that the dependency graph they rely on for building is not compatible with MVS. Migration thus generally means adding a replace statement for each (in)direct dependency to use the exact versions from the existing dependency graph. Then in a second stage slowly iterate over dependencies and one-by-one update them to conform to MVS.


NB: I am not advocating that the way Kubernetes uses replace and their overall philosophy is close to being canonical or even a good example... merely that it is a big community project and thus is shaping the image many Go developers will have, or get, of the use of modules. Even today I have already heard from many individual developers converting to modules that they want, or need in their own words, to learn how to develop multi-module repositories just because Kubernetes is doing so. It requires a lot explaining each time as these developers have no valid reason for actually needing it themselves... but that's a different (albeit related) problem.

@Helcaraxan

This comment has been minimized.

Copy link
Contributor

@Helcaraxan Helcaraxan commented Oct 18, 2019

Instead of polluting this thread with my thoughts on multi-module repos I've posted them into the linked, and more appropriate, issue #27542.

@bcmills bcmills modified the milestones: Backlog, Go1.15 Oct 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.