Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow go.mod.local to contain replace/exclude lines #26640

Open
maeglindeveloper opened this issue Jul 27, 2018 · 29 comments
Open

cmd/go: allow go.mod.local to contain replace/exclude lines #26640

maeglindeveloper opened this issue Jul 27, 2018 · 29 comments

Comments

@maeglindeveloper
Copy link

@maeglindeveloper maeglindeveloper commented Jul 27, 2018

Hi everyone,
I'm actually using a go.mod file for dependencies in my go project.

I saw that there is a way to use local packages instead of online repo, using the replace() features, which is really cool when developping multiple packages at the same time.

I was wondering if there is a way split in different files the require dependencies and the replace one. Something like that

go_replace.mod => define the replace part.
go.mod => define the module and the required dep, including the go_replace.mod.

the go_replace.mod will not be pushed on git, and each developer will create it if needed.

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Thanks a lot!

@rsc

@mvdan mvdan added the modules label Jul 27, 2018
@mvdan mvdan added this to the Go1.11 milestone Jul 27, 2018
@mvdan mvdan added the NeedsDecision label Jul 27, 2018
@mvdan
Copy link
Member

@mvdan mvdan commented Jul 27, 2018

Could you clarify what is the purpose of separating the data into two files? So that developers don't commit temporary replace directives by mistake? I'd imagine that there could be tooling to help with that. For example, a tool to add and undo temporary directives, or having CI fail on unexpected replace directives.

I'm also worried about adding more special Go module files, even if they are in theory not to be committed into git.

@flibustenet
Copy link

@flibustenet flibustenet commented Jul 27, 2018

It's dangerous as you can commit files that use something else than what said in go.mod.
With replace in go.mod you see immediately (just before commit) that you will commit with a dependency fork.

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 2, 2018

when [developing] multiple packages at the same time

We certainly do need to address that use-case somehow, but splitting up the module definitions doesn't seem like an appropriate solution: ideally I don't want individual developers to have to edit the module definition to do that.

@rsc rsc changed the title cmd/go go.mod file & replace feature cmd/go: allow go.mod.local to contain replace/exclude lines Aug 9, 2018
@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 9, 2018
@cosban
Copy link

@cosban cosban commented Sep 3, 2018

I fall into the use case of "building multiple packages concurrently" as well.
I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications.
There are multiple solutions to this problem, some more elegant than others.

  • Having a release modifier or command (#26420) that ignores local copies would be a simple (from the user perspective) solution and would not require additional modifications to the go.mod files.
  • The solution proposed in this issue could also work. It could even be combined with the first proposal. @bcmills, when you say you "don't want individual developers to have to edit the module definition to do that", are you referring to them manualy modifying go.mod files? If not, could you please clarify your position a little more so I may understand it better? This could be resolved by allowing the developer to specify that the library is one which will be modified locally as well using something like go get -u <package@version> -local <relative/path/to/package>
  • Allowing users to install local, uncommitted versions could also work for this, but probably isn't the most elegant solution. This is, users could go install their current copy of the package they are developing, and the module system would store it in some sort of development version folder. In this method, version v0.0.0-20180526204631-c40f734b202a could be the latest commit, but v0.0.0-20180526204631-c40f734b202a-devel could be the latest local copy. If the local copy is not found, the module system would be able to revert back to the last commit when using the go get command.

I believe that the first option is the best, but I'm interested in reading what direction you guys are learning toward in the interim.

@wsc1
Copy link

@wsc1 wsc1 commented Oct 3, 2018

Hi all,

I'm not sure that my approach will help others, but in case it does, (taken from go-nuts)

There is a bare bones workaround without local go.mod nor replacements.

Suppose you need to work on module A and module B simultaneously (hopefully you set things up to avoid this, but hey, it's inevitable)

If you only need to edit one package at a time in at least one of A and B, then you don't need
to muck with local go.mod or versions tags or replacements or anything.

Let B be the module which has only one package to edit, and p the package.

Copy p somewhere in A, rename import path to p using existing go imports/ide,
edit both, build, test, etc. A's go.mod will be a little crazy and out of whack.
So what. Copy p back to where it belongs in B, re-run go imports, build/test or go mod tidy
Now commit as you like the two.

In terms of commands and tools, it's very simple: no tools, no replacements, no go.mod.local, no replacements which shouldn't be pushed/commited, no workflow restrictions. Simple "cp -r" and use with editor that understands go imports works fine.

Now, thanks to modules, if your edits correspond to releases, it doesn't even matter in what order
you commit/push the changes after the two modules have been edited.

Yes, it's still a PITA. For me it's much less painful than thinking about workflows and replacements.
But for me it also doesn't happen a lot.

That said, I do think it would be nice to have the go command do for a set of modules, say go build ./...
with ./ containing the modules in question. that would allow updates of all version in one shot for example.

@ohir
Copy link

@ohir ohir commented Oct 14, 2018

bcmills> "ideally I don't want individual developers to have to edit the module definition to do that."

Pondering on how to "merge" go.mod with the hypothetical go.mod.local I came to the point where only simplest solution stayed viable: add a -local knob. Then, if -local flag is present, tools will use go.mod.local instead of go.mod.

I know that knobs breeding is unpopular among core team but consider it, please.

@benhalstead
Copy link

@benhalstead benhalstead commented Feb 11, 2019

go.mod.local or similar would also help our use case. We have developers working on service 'A' which relies on library 'B'.

Our developers will make changes to both the service and the library and so have a replace in the service's go.mod file to use their local version of the library code instead.

The problem is that each developer will have checked the library out onto their filesystem with a different path so each of them need a different path in the 'replace' statement, meaning version control of go.mod is messy.

@zshbleaker
Copy link

@zshbleaker zshbleaker commented Mar 16, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

@bradleyjames
Copy link

@bradleyjames bradleyjames commented Mar 24, 2019

+1 to go.mod.local or just a separate file based approach. It would provide a clean logical separation and file based will be simple to ignore via .gitignore. Also overriding the mod file location (aka opt-in) during development would provide a sanity check in mitigating the risk of pushing to production.

@bradleyjames
Copy link

@bradleyjames bradleyjames commented Mar 24, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

We’re using GitHub Enterprise which works for us. What specific issues are you having?

@pwaller
Copy link
Contributor

@pwaller pwaller commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

I want to be able to do a git bisect with a replace directive in effect. For this to be effective, I want to avoid having a dirty working tree, so editing the go.mod isn't great for this.

I have found myself a few times wanting a replace directive without contaminating the git status output with changes, for example whilst just doing some development whilst working on multiple modules simultaneously.

@SamWhited
Copy link
Member

@SamWhited SamWhited commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

Not everyone can see that link; if it's important, please copy any context needed into this discussion.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 27, 2019

@SamWhited, the discussion on Slack was literally just a link to this issue.
(I mostly treat Slack as a way to quickly route people to the relevant public issues.)

@gunnsth
Copy link

@gunnsth gunnsth commented May 20, 2019

As a developer in a fairly large library I frequently work with smaller applications that use this library. To be able to work with the local version of the library requires setting the replace directive for each small application. Setting a global replace directive would make this a lot easier. Otherwise it is definitely a regression from the GOPATH in terms of convenience.

It seems like a global replace directive could either be specified in a global file or through an environment variable, e.g. GOMODREPLACE=old[@v]=new[@v],... with multiple replacements comma separated.

@owais
Copy link

@owais owais commented Jun 12, 2019

I use the replace directive quite a lot but this request scares me a little (at least the way it was requested originally). I fear people might forget that their local builds are using the .local file and end up testing against the wrong dependencies. Of course a good process with CI will catch such issues but IMO relying on external processes for correctness from the perspective of in-built Go tooling is wrong.

Imagine having to submit a very critical hotfix for something that needs to bypass the regular process. One might forget about the existence of a .local file or out of muscle memory use -local flag to build the program, test it locally and ship the code to prod. The remote build pipeline would not use the -local flag and would result in publishing an expected build. There can be many other such cases. For example, while reviewing a PR, I might want to checkout a branch to build and test the behavior locally but accidentally end up testing the branch with a local copy of the dependency. People learning Go might also find it frustrating to debug issues because of an old .local file lying around.

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@myquant
Copy link

@myquant myquant commented Sep 17, 2019

The problem of my team: say we have 3 coders developing same module, their go.mod might be as following:

coder1
replace my/program/module => c:/module

coder2
replace my/program/module => d:/module

coder3
replace my/program/module => e:/module

One guy might accidentally commit his go.mod to git, then others git pull and build...bomb! Guess go.mod.local is a nice idea to resolve this problem. It can be added to .gitignore.

@rwxrob
Copy link

@rwxrob rwxrob commented Jan 2, 2020

I have regularly been struggling with this same thing and really do not like having to switch back and forth between having replace and not because I frequently do wip commits and every single one would require swapping it out. I really, really like the idea of something global. Now that $GOPATH/env is a thing perhaps that is the perfect place for it.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jan 29, 2020

Closing in favor of #34506, which adds a -modfile flag to Go 1.14. It's not exactly what's proposed in this issue, but it's close, and it solves a lot of other problems.

For this workflow I'd recommend:

  • Copy go.mod to go.local.mod. Add go.local.mod to .gitignore (or equivalent for your workspace).
  • Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.
  • Any any replace and exclude directives or other local edits.
  • Before submitting and in CI, make sure to test without the local file: go env -u GOFLAGS or just -modfile=. Probably also go mod tidy.
@jayconrod jayconrod closed this Jan 29, 2020
@AndreKR
Copy link

@AndreKR AndreKR commented Jan 29, 2020

@jayconrod I don't think #34506 is a replacement (no pun intended) for this one. From that issue:

Note however, that these local files are used instead of the regular files, not in addition to, so some synchronization might be required.

So the modfile setting is not at all helpful for the scenario where you want the dependencies from the repository (and send any changes to the repository) and just some local replacements.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jan 29, 2020

@AndreKR I understand it's not quite the same. With this, you can work out of the same code base without modifying the original go.mod.

We discussed whether -modfile should mean instead of or in addition to, I think on #34506 or one of the last few golang-tools calls. It seemed like we could serve more use cases by keeping the semantics simple and not touching the original go.mod file.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 29, 2020

I agree with @AndreKR: I don't think #34506 addresses the use-case here, which is to make a set of replace directives easy to revert (or to .gitignore) while still updating the require directives in the main go.mod.

@bcmills bcmills reopened this Jan 29, 2020
@myitcv
Copy link
Member

@myitcv myitcv commented Jan 29, 2020

Agree, @bcmills.

Also to point out that

Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.

is a global environment change, whereas what is being discussed here is very much a project-local effect.

@pdf
Copy link

@pdf pdf commented Jan 29, 2020

Allowing modfile to be specified multiple times, or accept multiple filenames, merge the contents, and fail silently if one of the files doesn't exist would solve this I think.

ETA: Though, it doesn't really standardise the mechanism (every developer/project may choose a different filename to .gitignore), and hence requires much more error-prone local modifications in a team or open source project.

@p554157atch
Copy link

@p554157atch p554157atch commented Aug 20, 2020

Hello, please check this proposal as well: #40933

We can combine two different mechanisms,
using a go.local.mod plus importing replace definitions into go.local.mod from an existing local file.

For example go.local.mod for 20 different packages can have:

module hello
go 1.14
replace_definitons /home/coder/replace_definitions.mod

If we made changes to /home/coder/replace_definitions.mod, then changes will be applied to all the 20 go.local.mod files that import the replace_definitions.mod file.
go mod tidy takes care of un-used definitions.

This way git repository won't have unnecessary commits, plus we will be able to specify replace definitions in one local file and changes will be reflected in all local modules that import this file.

Making simultaneous edits on local modules that require constant updated to replace definitions will become much easier if we follow this method.

Let me know what you think.

@sheen4n

This comment has been minimized.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 26, 2020

A reminder about https://github.com/golang/go/wiki/NoPlusOne - if you want to add your +1, just give a thumbs up reaction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.