New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: do not remove cyclic requirements in 'go mod tidy' #27899

Open
bcmills opened this Issue Sep 27, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@bcmills
Member

bcmills commented Sep 27, 2018

As noted in #27858 (comment), when a package is moved between a module and one of is submodules, the module and submodule each need to require some baseline version of the other, for two reasons:

  1. To ensure that go get -u doesn't drop any needed packages.

  2. To ensure that any third module that depends on both the module and submodule will always end up in a configuration that provides only one copy of the package.

It would be really unfortunate if moving a package meant that you could never run go mod tidy the involved modules again, or that you have to apply some manual edit every time you run it.

We should either modify go mod tidy to avoid dropping requirements that are involved in a cycle with the main module, or add some explicit annotation comment to tell it not to drop an apparently-unnecessary requirement.

@bcmills bcmills added this to the Go1.12 milestone Sep 27, 2018

@hyangah

This comment has been minimized.

Contributor

hyangah commented Oct 12, 2018

If two modules don't have actual dependency but require is needed for the second reason, there is no cycle to detect. Explicit annotation from the submodule (maybe #keep) seems easier.

@bcmills

This comment has been minimized.

Member

bcmills commented Oct 15, 2018

@hyangah, I'm not sure what you mean? If you're changing submodule boundaries, there should always be a cycle. (Reason # 1 imposes a requirement edge in one direction, while # 2 imposes the other direction: I can't think of a situation where we would need to preserve the edge but not have a cycle.)

@hyangah

This comment has been minimized.

Contributor

hyangah commented Oct 15, 2018

@bcmills my comment is about how to change go mod tidy to detect the (implicit) cycle automatically and prevents dropping the requirement (or even better, adding the implicit requirement automatically), not about the existence of such implicit cycle. The implicit cycle is not encoded in the import graph from .go files so the go mod tidy should go beyond the latest source code analysis. Moreover, the required version is the one that introduces the new module boundary, which is not automatically known yet until the commit to add the new boundary is checked in.

@thepudds

This comment has been minimized.

thepudds commented Oct 24, 2018

As noted in #27858 (comment), when a package is moved between a module and one of is submodules, the module and submodule each need to require some baseline version of the other
...
We should either modify go mod tidy to avoid dropping requirements that are involved in a cycle with the main module, or add some explicit annotation comment to tell it not to drop an apparently-unnecessary requirement.

Sorry if this is just noise, and I wouldn't suggest this as a long term solution, but adding a comment here in case it triggers some discussion and/or a better idea.

To avoid go mod tidy removing cyclic dependencies that were put in place to allow a module to be broken apart by moving packages between modules, would it work in 1.11 to do something like add a set of modmove.go files or similar that contain import statements describing the required cyclic dependencies such that a later go mod tidy does not remove the require directives?

I have not tested this on an actual repo, but if you wanted to take a module example.com/my/module/ and break out example.com/my/module/foo into a separate module with its own go.mod, then after manually added the necessary cyclic require statements to the go.mod files, you could then encode those dependencies in a set of new modmove.go file, via something like:

In the root of the repo:

# 1. create a modmove.go for the parent module:

mkdir modmove
cat <<EOF > modmove/modmove.go
// +build modmove

package modmove

import (
        _ "example.com/my/module/foo"
)
EOF

# 2. create a modmove.go for the nested module foo:

mkdir foo/modmove
cat <<EOF > foo/modmove/modmove.go
// +build modmove

package modmove

import (
        _ "example.com/my/module"
)
EOF

In theory, a later go mod tidy would not remove the manually added cyclic require statements.

This technique would be somewhat analogous to the tools.go approach for tool dependencies as described in #25922 (comment), but applied in a different way against a different problem.

(Side note: example above creates modmove subdirectories. That might not be required given the modmove build tag?)

@bcmills

This comment has been minimized.

Member

bcmills commented Oct 24, 2018

@thepudds, that's a nice workaround, but it's slightly fragile: if the package that we put in the import line itself moves into or out of some other module with the same prefix, we could end up accidentally creating a cycle between the nested modules rather than the intended cycle with the parent module.

@thepudds

This comment has been minimized.

thepudds commented Oct 24, 2018

@bcmills, agreed -- it would take some care to set up initially, and then take some care to not mess up later if there are additional moves.

Hopefully the core go tooling has some additional logic targeting these types of scenarios in 1.12 or so such that something like this is no longer needed.

That said, I was trying to think if there was something that could work in 1.11 (aside from "never run go mod tidy again”, or needing to repeatedly re-apply a manual edit). Perhaps modmoved.go or similar could be viewed as a candidate temporary workaround for 1.11 if someone needs to split a module in 1.11.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 25, 2018

I don't think we can a priori assume that every cycle should be kept.
If I'm working on A and I know B uses A, I might run 'go get B@master; go test B/...'. That will create a cycle but one that I probably want 'go mod tidy' to remove.

@bcmills

This comment has been minimized.

Member

bcmills commented Nov 16, 2018

In https://golang.org/issue/27858#issuecomment-434713751, @jba suggests:

After a go get -u, the tool should see if any packages required by the main module are no longer in a module, and fetch those packages' modules in the usual way.

That would allow for a single require edge (to remove the duplicate copy) rather than a cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment