Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: mod tidy should join "require" sections if there are more than two #56471

Open
mvdan opened this issue Oct 28, 2022 · 7 comments
Open
Labels
GoCommand cmd/go modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Oct 28, 2022

Since #45965, go.mod files now generally consist of two sections: one require (...) section with direct module dependencies, and one for // indirect module dependencies.

However, if a go.mod file somehow ends up with more than two of those sections, then go mod tidy does not join them back. I think it should. For example, if I join the normal two sections into one, go mod tidy, splits them again.

Here are two instances I recently found of such an unintentional "split" of sections:

Both were spotted manually and fixed, as they were indeed unintentional. Locally, I used for f in **/go.mod; do n=$(grep -F 'require (' $f | wc -l); if [[ $n -gt 2 ]]; then echo $n $f; fi; done with Bash to find any others. I couldn't find any.

Globally, I tried using cs.github.com, but it doesn't appear to support matching regular expressions across multiple lines, like PCRE. But Googe's code search does; https://cs.opensource.google/search?q=path%3Ago.mod+AND+pcre%3Ayes+AND+%28require%5C+%5C%28%28.%7C%5Cn%29*require%5C+%5C%28%28.%7C%5Cn%29*require%29+ found:

None of those four appear to be on purpose. I think they could have appeared unintentionally due to a number of reasons, such as:

  1. Git merges. If two branches make changes to go.mod, depending on how the user resolves the conflicts manually, they could end up with more than two sections. I'm fairly sure that this is what happened in vocdoni-node, as the PR in question lived for over a month and had conflicts to be resolved.
  2. Manual editing. I've seen some users not fully grasp how go get works, and resorting to editing go.mod directly to update or add require lines. When doing it quickly or copy-pasting, I imagine it's tempting to just add a require some-module v1.2.3 at the end of the file, which will work regardless of what the file looks like.

I personally can't think why anyone would want more than two require sections today. The fact that I could only find four examples today in ten minutes of research is a double-edged sword. On one hand it's proof that basically noone wants more than two sections. On the other, it also means that this problem appears rarely, so it might not be a high priority as an improvement for go mod tidy.

Still, I have encountered this problem myself twice now, and that's the magic number that tells me I should file a bug and propose an automated fix. This topic has been brought up a number of times on Slack (January 2022, February 2022, October 2022), so there are at least a couple of other people experiencing the problem and fixing it manually.

In the second of those Slack threads, @bcmills mentions that this might be a quirk with the go mod tidy upgrade from go 1.16 to go 1.17. It could be that some of the third sections came about that way; it's hard to tell for sure. I'm a bit skeptical that the problem will go away with time, as 1.17 was released over a year ago and the extra sections still pop up. For example, that vocdoni PR was finalized in April 2022, and the go.mod file in master was already on go 1.17 since September 2021.

I think go mod tidy should join these extra sections. The only reason I see that as potentially risky is if, in the future, another proposal like #45965 comes along and we want more than two sections. But presumably that shouldn't be a problem, because go mod tidy already complains if a go.mod file has a go X.Y version that's too new.

cc @bcmills @matloob @leitzler @seankhliao @dylan-bourque

@mvdan
Copy link
Member Author

mvdan commented Oct 28, 2022

A slightly better CS regular expression is https://cs.opensource.google/search?q=path%3Ago.mod+AND+pcre%3Ayes+AND+%28require%5C+%28.%7C%5Cn%29*require%5C+%28.%7C%5Cn%29*require%5C+%29+&sq=, which also matches require X Y and not just require (X Y; ...). It found one more, funnily enough, in gopls:

@seankhliao
Copy link
Member

seankhliao commented Oct 28, 2022

see also #51983

@mvdan
Copy link
Member Author

mvdan commented Oct 28, 2022

Properly handling comments could indeed be some work. I'm honestly not worried about it, because in all the instances I've run into this problem, there were no custom comments in any of the require sections. So I'd be fine if we start by only joining require sections without any comments inside or between them, for example.

@heschi heschi added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 28, 2022
@heschi heschi added this to the Backlog milestone Oct 28, 2022
@bcmills
Copy link
Member

bcmills commented Oct 28, 2022

I'm a bit skeptical that the problem will go away with time, as 1.17 was released over a year ago and the extra sections still pop up.

Note that the awkward transition happens when the go directive in your go.mod file crosses that boundary; it's more-or-less independent of upgrading the toolchain. (Current versions of go still try to tidy according to the old format when in a module with an older go version.)

@mvdan
Copy link
Member Author

mvdan commented Nov 8, 2022

Right. Note that the vocdoni example had been on go 1.17 for nearly a year by the time the duplicate section was introduced, so I think the duplication is an ongoing inconvenience regardless.

@mvdan
Copy link
Member Author

mvdan commented Nov 17, 2022

For what it's worth, it happened again in one of the projects mentioned above :) vocdoni/vocdoni-node@ed5fb16#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6

@dylan-bourque
Copy link

dylan-bourque commented Nov 21, 2022

Can't share the specifics, but I just had an internal module end up with 5 require blocks. I'm using Go 1.19.3 and the module has go 1.18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

5 participants