proposal: cmd/go: restrict or remove the -modfile flag

[matloob]

cc @adonovan @seankhliao @golang/go-command

The -modfile flag was added to allow tools such as gopls to run the go command logic to make changes to a temporary version of a go.mod file so that those changes could be surfaced to the user and applied in the editor. But the flag is complicated to support and has overlap with overlays.

Ideally commands that modify go.mod (without -mod=mod being provided) (go mod tidy and go get) can output the file contents in a special mode for tools to use instead, and we could remove the -modfile flag.

If we can't remove it, I think we should restrict it to only the subcommands that need to be used by gopls to reduce the complexity we'd have to support in the go command.

[gabyhelp]

Related Issues

• cmd/go: flags to control changes to go.mod, go.sum #34506 (closed)
• x/tools/gopls (and friends): separate -modfile and -mod out of build flags #41826 (closed)
• proposal: cmd/go: support `-modfile` for `go mod init` #78503
• cmd/go: 'go list' should set the 'GoMod' field to reflect -modfile #36220 (closed)
• cmd/go: support -overlay for the go mod subcommand #42491 (closed)
• x/mod: move cmd/go/internal/modfile into x/mod for go.mod parsing and formatting #34924 (closed)
• cmd/go: reject the `-modfile` flag in workspace mode #59996 (closed)
• cmd/go: confusing error message (or incorrect behavior) when using -modfile with go run #51079 (closed)
• cmd/go: can't invoke a tool with -modfile within a workspace #73968 (closed)

Related Code Changes

• cmd/go: add -modfile flag that sets go.mod file to read/write

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

some possible flags for output to stdout

• -w=false like go fmt
• -diff and have the client merge the diff?
• -mod=dry-run ?

[jakebailey]

I thought this feature was pretty heavily used by people who want to manage tools in a separate directory or filename?

It's documented in https://golangci-lint.run/docs/welcome/install/local/, and has a lot of usage with go tool: https://github.com/search?q=%2Fgo+tool+-modfile%2F&type=code

[seankhliao]

I'd say that the setup never worked well enough, and we should maintain a single supported way of managing tools.

we've declined to document it as a supported method https://go-review.googlesource.com/c/website/+/648578/comment/d9ed9318_8e9659f5/

[jakebailey]

I guess I don't agree that a CL review comment is a decision, and "we" is still just you two... 😅

I personally never agreed with those who thought it was bad to mix regular and tool deps, but I have seen situations where more than one go.mod is helpful. For example, better managing (working around?) compliance tools, or cases where you want code gen to be able to have a new-ish minimum Go version while the shipped code is older. I'm sure I'm forgetting other cases.

[mvdan]

I use go tool -modfile quite a bit and it works just fine. I'm not sure what you mean by it not working well enough.

For some projects it's fine or even desirable to do an MVS mix of dependencies with the main go.mod, but for others it's best avoided - such as tools with large dependency graphs which are entirely unrelated to the main project. We've gained go -C in the meantime (#50332), and that probably does satisfy some -modfile use cases, but it doesn't satisfy e.g. go tool -modfile=tools/go.mod staticcheck ./... because go tool -C tools staticcheck ./... then runs the tool in the wrong directory.

I'm not opposed to restricting the use of -modfile, but the go tool -modfile= pattern is already widespread. If you really think it's an anti-pattern that must be broken, the announcement needs to explain why, but bear in mind that users being broken will not be happy either way. Ideally, you either don't break them at all, or you offer a better drop-in alternative.

For example, if you want to lean into overlays as a replacement that could work, but you have to bear in mind that JSON in the command line is a pain. Perhaps offer a non-JSON alternative to do the equivalent of something like:

go tool -overlay='{"Replace": {"go.mod": "tools/go.mod"}}' staticcheck ./...

[thepudds]

we've declined to document it as a supported method

FWIW, use of -modfile with go tool is documented as a supported flag. The cmd/go documentation for 'go tool' says:

The -modfile=file.mod build flag causes tool to use an alternate file instead of the go.mod in the module root directory.

Regarding that specific CL 648578, the entirety of what Michael wrote about that was:

I'm not sure we should suggest using another mod file for tool dependencies... I understand that some users may not want to mix up their tool and code dependencies, but it seems more cumbersome and error-prone and is not the "well lit" path.

I think that's a fair comment, especially when you consider the timing of his comment there was near the Go 1.24 release date, which was when go tool foo for external tools was new. And because that section of doc was in a HOWTO-ish portion of the overall doc, I suspect there could have been a concern that suggesting -modfile there might give a hint of it being a preferred approach. (In addition, "let's first see how people use it in practice" is often a good strategy in general.)

Also FWIW, in addition to the 1.2K+ hits for using -modfile with go tool that @jakebailey's simple search reported, multiple people have written near identical wrappers to make -modfile more convenient with go tool, like https://github.com/andreynering/gomodtool.

My guess is -modfile is fairly load bearing in the broader ecosystem at this point.

That's not to say it couldn't be removed or restricted (cmd/go flags and behavior are outside the Go 1 compatibility promise), and certainly worth analyzing and discussing.

[matloob]

My guess is -modfile is fairly load bearing in the broader ecosystem at this point.

That's not to say it couldn't be removed or restricted (cmd/go flags and behavior are outside the Go 1 compatibility promise), and certainly worth analyzing and discussing.

I certainly don't think we should break common workflows, so removing it altogether doesn't seem viable unfortunately. Perhaps we can find out how it's commonly used and restrict its usage to only those commands?

I use go tool -modfile quite a bit and it works just fine. I'm not sure what you mean by it not working well enough.

I think it's a bit unfortunate to have to juggle multiple modfiles. I agree that using overlays would be more complicated but it seems to me like having to manually switch out the buildlist using -modfile used is a confusing way to use tools without affecting the buildlist for your project. It feels like an indication that we're missing a feature in the go command or some kind of standard tooling for working with go projects. On the tool proposal my personal preference was that each tool would specify a version and would have its own buildlist so that we would build the tool with the dependencies the tool creator had intended but that ship has sailed.

---

I think I had seen a comment in this thread where someone suggested that we recursively invoke the go command with an overlay when -modfile is provided, and that might be a way to simplify the implementation (similar to what was suggested for GO111MODULE=auto in #78350 (comment)). That would fix my issues with implementation complexity.

One (hopefully reasonable?) option for restricting it could be to only support what's currently documented, or is commonly used, and restrict all other subcommands?

[mvdan]

I'd be totally fine with go tool -modfile to internally be rewritten to use an overlay. As long as I'm not required to change all my scripts or deal with JSON in Bash, it's fair for the internals to be simplified.

If it made the transition easier for you, you could forbid using -modfile together with an -overlay which replaces go.mod, or even any -overlay at all. I haven't seen both flags in combination myself. And if someone is already setting up an overlay, they can just replace go.mod that way.

Similarly, I'd also be fine with restricting the -modfile usage to only those commands which are commonly used or documented, although I'd probably try the earlier idea to reimplement with overlays first, to see if that's enough to reduce complexity. I'm sure that most scenarios that support the flag today are used in a surprising way by someone.