proposal: ref/mod: mention whether go.work files should be checked into VCS

[mvdan]

https://go.dev/ref/mod#workspaces describes the current implementation of Go workspaces, as proposed by https://golang.org/issue/45713, which culminated in the proposal doc at https://go.googlesource.com/proposal/+/master/design/45713-workspace.md.

The proposal doc says:

go.work files should not be checked into version control repos containing modules so that the go.work file in a module does not end up overriding the configuration a user created themselves outside of the module. The go.work documentation should contain clear warnings about this.

However, the canonical documentation about go.work at https://go.dev/ref/mod#workspaces makes no mention of this, as far as I can see. In practice, it seems like the common knowledge among Go developers is that the file should never be checked in, because:

• GitHub's default gitignore includes the file since Go: Ignore Go workspace file go.work github/gitignore#3884
• Official Go repositories set an example by not checking them in; for example, x/tools does not include one, even though it has two modules connected by a directory replace directive
• Some third party docs or blog posts echo the recommendation to never check them in, like https://sebastian-holstein.de/post/2021-11-08-go-1.18-features/

First, we should decide whether we actually want to recommend users to never check in go.work files, like the original proposal doc said. If that's the case, then we simply need to add that recommendation to the /ref/mod page.

However, if we don't want to make that general recommendation, then we need to figure out under what circumstances it is recommended or not to check in go.work into VCS. For example, here are some personal guesses of mine:

• For a VCS repository holding a Go library module intended for external use, checking in go.work is likely a bad idea, as it prevents downstream users from easily setting up their own go.work in a parent directory.
• For large monorepos with multiple modules, it likely makes sense to check in go.work, as it makes the setup easier and often has no downsides.
• In some cases, go.work files will naturally live in the parent directory to the VCS clones, so they have no VCS repository to naturally be checked into at all.

cc @bcmills @marwan-at-work @matloob

[mvdan]

@bcmills points out that perhaps there isn't consensus about whether to check go.work files into VCS. We never gave any advice on it, and the community seems to lean on "never check them in".

We could recommend never checking them in, but I think that's a bad idea: in some particular scenarios it can make perfect sense to check them in, such as monorepos holding multiple modules.

We could continue with no recommendation at all, but I worry that then we'll continue to see third party docs and sites make assertive recommendations like "never check them in".

I personally think that we should give some general guidance that isn't too strict one way or another, such as:

go.work files should generally not be checked into VCS, as their purpose is to develop multiple modules at the same time, and each module tends to have its own VCS repository. However, there are certain scenarios where it can make sense to check in go.work files, such as a monorepo holding multiple modules which depend on each other.

This needs a decision that isn't easy to make, and we should probably get input from others, so @bcmills suggests turning this issue into a proposal. Doing that.

[omertuc]

I've noticed that gopls isn't able to "Find references" for an identifier in a "nested" module unless I create a go.work file. Without the go.work file it only finds the declaration/definition itself. With the go.work file it successfully finds all references in all modules in the repo. You can try to "Find references" in this example project and see it works only while the go.work file exists.

I couldn't get VSCode or Neovim to make it work in any other way - I just had to create a go.work file - even with multiple VSCode "LSP workspaces". Maybe I did something wrong, feel free to correct me.

I would hate to have to teach people in my team how to create a go.work file just so gopls will work as expected for them, so it seems very reasonable to just push it into VCS. That's why I was very surprised to see the proposal suggested against doing that.

@mvdan I'm not sure if my example project is what you were referring to as "monorepos", but wanted to give it as an example

Maybe it's just a gopls bug and so this can't be used as an argument in favor of checking in go.work files to VCS, but I'm not familiar enough with gopls to say for sure

[mvdan]

Assuming your nested Go modules all live in the same git repository as the main module, then yes, that's the sort of setup that I mean by a monorepo. A VCS monorepo will typically contain dozens of modules, and in general nested modules might be tricky and could be avoided, but the end result is still the same: it is possible and perhaps reasonable to check in a go.work file.

[bep]

The go.work feature is incredibly useful, but since it's default on (and cannot be turned off without deleting the work file (I think)), I think you should expand your documentation scope to also discuss where to put the go.work file.

If you put it inside your Go module (monorepo or not) and add go.work to your .gitignore, your local go test ./... may be very different from the CI build.

My current working theory about my own use of go.work files is that they cannot live in the Go repos themselves, but typically in separate /myworkspace folders.

[bcmills]

and cannot be turned off without deleting the work file (I think)

FWIW, you should be able to set GOWORK=off to force it off.
(See https://go.dev/ref/mod#environment-variables.)

[bep]

FWIW, you should be able to set GOWORK=off to force it off.

Those environment variables does not automatically propagate across all the environments in an open source project.

I have said this before, but may as well repeat it, I think having this enabled by default was the wrong decision. We had a situation where you had to make sure that any replacements in go.mod had to be short lived and certainly not committed to Git. Now it's very much guess work what Go Module state you build with. Again, I salute the concept of go.work, but it should be something that was off by default and turned on by switches (e.g. in VS Code or by GOWORK=on).

[Southclaws]

I've come across quite a few situations where I've written a library and made use of a go.work file to provide a tangible benefit: I can easily have sub-modules, run go test ./..., have gopls work and not pollute the repo's package go.mod with unnecessary dependencies.

Two use-cases I had recently: I wanted to write some tests for a package that works with errors. These tests want to ensure my library works nicely with all the other errors packages. So this means I'm importing 10 different errors packages into that ./tests/go.mod. I used a go.work file in the root to make everything play nicely and the root level go.mod has no dependencies, so that's really neat.

The other case was I wrote a package that integrates with various other packages. Of course each binding layer is a separate module so, again, the root level go.mod doesn't need to unnecessarily specify a ton of unrelated dependencies when, realistically, a consumer of that package is only ever going to want either zero or one of the integrations packages. Now, I could create a ton of repos, or make a github org, but I simply don't want to incur the administrative burden so a monorepo suits me well. Again, go.work makes this all work nicely with gopls, go test, godoc, etc.

I thought this was fine until someone on the Gopher slack told me this was bad practice, I couldn't really find any definitive answer and was linked to here. All of what I did was based on official documentation, not blog posts or anything else. So it wasn't really clearly communicated that what I was doing (despite making my workflow easier) was actually not the intended use of go.work.

So it seems I've stumbled on some use cases that make my workflow easier, but are considered bad practice. This feels very un-Golang. I'd rather the tooling allow me to do things that are good practice, not not allow them at all, so there's no room for opinions and discussion (like the K&R thing - no arguments, just one style.)

Thanks for raising this though! I'm keen to see a definitive outcome, and I'm not strongly opinionated one way or the other. 😊

[jeffwidman]

@Southclaws for both of your scenarios, when a user imports your package and then creates their own go.mod file, go mod tidy will prune that project's transitive dependencies such that the unused test modules and unused integrations packages will never be imported... you can check, and you shouldn't see anything actually listed as // indirect unless it's actually used.

So I don't think either scenario you describe really makes sense for a go.work module, you're basically putting in manual effort to do what the tooling already does automatically.

[Southclaws]

Ah thanks, I didn't know this! I figured the lack of explicit dependencies/devDependencies (like npm) meant this wasn't the case.

Given that, I see no reason to not ignore go.work files, it would be worth mentioning this in the official docs for sure!

[matloob]

I think we should strongly caution against checking in go.work files, but there are some rare cases where it makes sense to check them in so we have to leave some room for that possibility. Here's proposed wording to add to ref/mod. What do y'all think:

go.work files should usually not be checked into version control though there may be rare cases
where it might make sense to do so: particularly if the workspace modules don't have any other
modules that depend on them. Module developers should think very carefully before checking in
`go.work` files whether it is worth doing so because of the considerable disadvantages:
Checked-in `go.work` files might override developers' own `go.work` files contained
in parent directories and cause confusion when the developers' `go.work` files don't apply. Checked-in
`go.work` files may also result in the wrong versions of modules being tested in continuous integration
systems: `go.work` will override the versions of dependency modules that are in workspaces so those
versions will not be respected in testing.

[eliben]

This would be great, thanks @matloob

I'd maybe organize the text a bit differently (*), but the overall idea is solid.

(*) Start by saying it's not recommended and list the disadvantages as a bullet list.
Then after the list, add a disclaimer for when the guideline can be relaxed.

[mvdan]

Compared to my ealier attempt at #53502 (comment), I like that it explains a couple of pitfalls with checked-in go.work files. I agree with @eliben that I would like to see better formatting, perhaps splitting into 1-2 paragraphs with a bullet list.

However, I'm not a big fan of the wording "rare cases"; I think monorepos (or multi-module repos) are relatively common, particularly in work scenarios. I worry that only listing disadvantages, and saying that there "may be rare cases where it might make sense", is going to give the idea that it's heavily frowned upon. When, in practice, I think an enterprise monorepo with many modules should definitely check in a go.work file. They just need to be mindful of the pitfalls.

[matloob]

How about the following? I softened up the rare cases wording because I don't have visibility into how modules are structured in work scenarios but I want to be really careful about not recommending go.work files be checked in.

In general, it is recommended that go.work files not be checked into version control systems. Module developers should think carefully before checking in go.work files because of the considerable disadvantages- among others:

• Checked-in go.work files might override developers' own go.work files contained
  in parent directories and cause confusion when the developers' go.work files don't apply.
• Checked-in go.work files may also result in the wrong versions of modules being tested in continuous integration
  systems: go.work will override the versions of dependency modules that are in workspaces so those
  versions will not be respected in testing.

With that said there are some cases where it might make sense to do so: particularly if the workspace modules don't have any other modules that depend on them.

[Emptyless]

The gosec tool can only scan multi module projects with a go.work file. I've tried to support scanning without go.work file checked into VCS but this is closed until there is consensus on go.work (securego/gosec#1100). I just want to chime in that the wording doesn't necessarily have to be a recommendation but just acknowledge that there are two ways to deal with the go.work file and describe how to identify what would work for your case. Continuing on @matloob, how about:

Depending on the project, think of the following when deciding to add the go.work file to a version control system:

• Are there modules external to the project that depend on the workspace modules?
• Are the developers of the project aware that if they have a go.work file, it might be overruled by the project go.work if added?
• Are the continuous integration pipelines configured such that the versions specified in go.mod files are respected, e.g. with a GOWORK=off environment variable?

About a monorepo, does it generally make sense to commit the go.work file? Maybe I'm interpreting incorrectly but isn't one of the benefits of using multiple modules in a large monorepo that it decouples them? Modules can publish new (possibly breaking) versions without having to refactor the entire monorepo. This decoupling would break if committing a global go.work listing all modules.

[matloob]

@Emptyless I'm very surprised looking at the comments on your review. We should certainly not force people who work in multi module repositories to check in their go.work files and should support users working without checked-in go.work files. That's how you get the best experience with go.work files. go.work files are meant to be a convenience feature to make certain workflows easier- not a requirement.

I tried to hit the tone in my proposed wording that in some cases it is okay to check in go.work files but I certainly don't want to encourage doing so or even worse require it.

The best experience for a monorepo is to have a single module. Because there are cases where that's not possible, for example because different components of a monorepo are distributed and versioned separately, but that's exactly the decoupling that you mentioned.

[dylan-bourque]

The best experience for a monorepo is to have a single module.

I would think exactly the opposite. In many (most? nearly every?) cases a monorepo will contain multiple things and those things would typically be versioned independently (pretty much the definition of a Go module). Having a monorepo be a single Go module feels like the opposite of what I would want/expect.

I use VS Code and it does in fact require that you have a go.work in place if your workspace contains multiple modules so I think it very much makes sense to either 1) commit go.work/go.sum and accept the trade-offs or 2) add scripts or other developer tooling to create and populate a go.work for the repo from scratch.

At work we've settled on option 2 for repos with >1 module, which are becoming more common for us.

[eliben]

How about the following? I softened up the rare cases wording because I don't have visibility into how modules are structured in work scenarios but I want to be really careful about not recommending go.work files be checked in.

In general, it is recommended that go.work files not be checked into version control systems. Module developers should think carefully before checking in go.work files because of the considerable disadvantages- among others:

* Checked-in `go.work` files might override developers' own `go.work` files contained
  in parent directories and cause confusion when the developers' `go.work` files don't apply.

* Checked-in `go.work` files may also result in the wrong versions of modules being tested in continuous integration
  systems: `go.work` will override the versions of dependency modules that are in workspaces so those
  versions will not be respected in testing.

With that said there are some cases where it might make sense to do so: particularly if the workspace modules don't have any other modules that depend on them.

@matloob did you get any significant objections to this? If not, can this be "implemented" (by mentioning in the right docs)?

[matloob]

I haven't gotten any significant objections. I'll send a CL to add this to ref/mod shortly.

[gopherbot]

Change https://go.dev/cl/580115 mentions this issue: ref/mod: add a warning against checking in go.work files

[troy0820]

Would it be useful, because there is a tutorial on how to use workspaces and the workflow that it represents, to provide a tutorial/workflow where checking the go.work would be beneficial? The use case that say kubernetes/kubernetes has is a lot different than most and that clarity would be beneficial when making decisions like this beyond the understanding of the disadvantages of it all.