Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow go.mod.local to contain replace/exclude lines #26640

Open
maeglindeveloper opened this issue Jul 27, 2018 · 18 comments

Comments

Projects
None yet
@maeglindeveloper
Copy link

commented Jul 27, 2018

Hi everyone,
I'm actually using a go.mod file for dependencies in my go project.

I saw that there is a way to use local packages instead of online repo, using the replace() features, which is really cool when developping multiple packages at the same time.

I was wondering if there is a way split in different files the require dependencies and the replace one. Something like that

go_replace.mod => define the replace part.
go.mod => define the module and the required dep, including the go_replace.mod.

the go_replace.mod will not be pushed on git, and each developer will create it if needed.

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Thanks a lot!

@rsc

@mvdan mvdan added the modules label Jul 27, 2018

@mvdan mvdan added this to the Go1.11 milestone Jul 27, 2018

@mvdan mvdan added the NeedsDecision label Jul 27, 2018

@mvdan

This comment has been minimized.

Copy link
Member

commented Jul 27, 2018

Could you clarify what is the purpose of separating the data into two files? So that developers don't commit temporary replace directives by mistake? I'd imagine that there could be tooling to help with that. For example, a tool to add and undo temporary directives, or having CI fail on unexpected replace directives.

I'm also worried about adding more special Go module files, even if they are in theory not to be committed into git.

@flibustenet

This comment has been minimized.

Copy link

commented Jul 27, 2018

It's dangerous as you can commit files that use something else than what said in go.mod.
With replace in go.mod you see immediately (just before commit) that you will commit with a dependency fork.

@bcmills

This comment has been minimized.

Copy link
Member

commented Aug 2, 2018

when [developing] multiple packages at the same time

We certainly do need to address that use-case somehow, but splitting up the module definitions doesn't seem like an appropriate solution: ideally I don't want individual developers to have to edit the module definition to do that.

@rsc rsc changed the title cmd/go go.mod file & replace feature cmd/go: allow go.mod.local to contain replace/exclude lines Aug 9, 2018

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 9, 2018

@myitcv myitcv referenced this issue Aug 12, 2018

Closed

Add README #7

@cosban

This comment has been minimized.

Copy link

commented Sep 3, 2018

I fall into the use case of "building multiple packages concurrently" as well.
I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications.
There are multiple solutions to this problem, some more elegant than others.

  • Having a release modifier or command (#26420) that ignores local copies would be a simple (from the user perspective) solution and would not require additional modifications to the go.mod files.
  • The solution proposed in this issue could also work. It could even be combined with the first proposal. @bcmills, when you say you "don't want individual developers to have to edit the module definition to do that", are you referring to them manualy modifying go.mod files? If not, could you please clarify your position a little more so I may understand it better? This could be resolved by allowing the developer to specify that the library is one which will be modified locally as well using something like go get -u <package@version> -local <relative/path/to/package>
  • Allowing users to install local, uncommitted versions could also work for this, but probably isn't the most elegant solution. This is, users could go install their current copy of the package they are developing, and the module system would store it in some sort of development version folder. In this method, version v0.0.0-20180526204631-c40f734b202a could be the latest commit, but v0.0.0-20180526204631-c40f734b202a-devel could be the latest local copy. If the local copy is not found, the module system would be able to revert back to the last commit when using the go get command.

I believe that the first option is the best, but I'm interested in reading what direction you guys are learning toward in the interim.

@wsc1

This comment has been minimized.

Copy link

commented Oct 3, 2018

Hi all,

I'm not sure that my approach will help others, but in case it does, (taken from go-nuts)

There is a bare bones workaround without local go.mod nor replacements.

Suppose you need to work on module A and module B simultaneously (hopefully you set things up to avoid this, but hey, it's inevitable)

If you only need to edit one package at a time in at least one of A and B, then you don't need
to muck with local go.mod or versions tags or replacements or anything.

Let B be the module which has only one package to edit, and p the package.

Copy p somewhere in A, rename import path to p using existing go imports/ide,
edit both, build, test, etc. A's go.mod will be a little crazy and out of whack.
So what. Copy p back to where it belongs in B, re-run go imports, build/test or go mod tidy
Now commit as you like the two.

In terms of commands and tools, it's very simple: no tools, no replacements, no go.mod.local, no replacements which shouldn't be pushed/commited, no workflow restrictions. Simple "cp -r" and use with editor that understands go imports works fine.

Now, thanks to modules, if your edits correspond to releases, it doesn't even matter in what order
you commit/push the changes after the two modules have been edited.

Yes, it's still a PITA. For me it's much less painful than thinking about workflows and replacements.
But for me it also doesn't happen a lot.

That said, I do think it would be nice to have the go command do for a set of modules, say go build ./...
with ./ containing the modules in question. that would allow updates of all version in one shot for example.

@ohir

This comment has been minimized.

Copy link

commented Oct 14, 2018

bcmills> "ideally I don't want individual developers to have to edit the module definition to do that."

Pondering on how to "merge" go.mod with the hypothetical go.mod.local I came to the point where only simplest solution stayed viable: add a -local knob. Then, if -local flag is present, tools will use go.mod.local instead of go.mod.

I know that knobs breeding is unpopular among core team but consider it, please.

@benhalstead

This comment has been minimized.

Copy link

commented Feb 11, 2019

go.mod.local or similar would also help our use case. We have developers working on service 'A' which relies on library 'B'.

Our developers will make changes to both the service and the library and so have a replace in the service's go.mod file to use their local version of the library code instead.

The problem is that each developer will have checked the library out onto their filesystem with a different path so each of them need a different path in the 'replace' statement, meaning version control of go.mod is messy.

@zshbleaker

This comment has been minimized.

Copy link

commented Mar 16, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

@bradleyjames

This comment has been minimized.

Copy link

commented Mar 24, 2019

+1 to go.mod.local or just a separate file based approach. It would provide a clean logical separation and file based will be simple to ignore via .gitignore. Also overriding the mod file location (aka opt-in) during development would provide a sanity check in mitigating the risk of pushing to production.

@bradleyjames

This comment has been minimized.

Copy link

commented Mar 24, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

We’re using GitHub Enterprise which works for us. What specific issues are you having?

@pwaller

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

I want to be able to do a git bisect with a replace directive in effect. For this to be effective, I want to avoid having a dirty working tree, so editing the go.mod isn't great for this.

I have found myself a few times wanting a replace directive without contaminating the git status output with changes, for example whilst just doing some development whilst working on multiple modules simultaneously.

@SamWhited

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

Not everyone can see that link; if it's important, please copy any context needed into this discussion.

@bcmills

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

@SamWhited, the discussion on Slack was literally just a link to this issue.
(I mostly treat Slack as a way to quickly route people to the relevant public issues.)

@thepudds

This comment has been minimized.

Copy link

commented Apr 20, 2019

As far I as understand the main proposal in this issue, a go.mod.local file residing in a module would be allowed to contain replace and exclude directives that apply when inside that module (with part of the benefit being that that go.mod.local file would not be checked in).

If so, a go.mod.local file containing replace statements would be fairly simple to setup if you had, say, two modules foo and bar that you needed to work on simultaneously. If foo depends on bar, it might be as simple as creating a single go.mod.local file inside foo that reads:

replace bar => ../bar

However, I think explicitly specifying each replace in go.mod.local files would be fairly challenging to manually setup if instead you have something like 6 interdependent modules with up to 6 go.mod.local files, which can translate to much more than 6 total replace statements, especially as you might sometimes want to work in module foo but sometimes work in module bar, and foo might depend on bar and baz while bar depends on baz and qux, etc. Dependency cycles between modules add to the total possible count of needed replace statements.

One could imagine the go tooling being able to take a set of modules that are laid down in a shared file tree containing M modules and automatically create the N needed replace statements to write to M go.mod.local files.

Even with help from the tooling, that seems a bit complex, and could get stale or otherwise allow for more user error.

Three alternatives:

  1. Rather than having M go.mod.local files to describe the relationship between M modules, instead have a single go.mod.local file that is in a shared parent in the file tree of the M modules. That single go.mod.local file could have all the replace statements in that single location. This would at least centralize the information, while keeping it explicit. It would also mean there are fewer replace statements overall in typical cases (e.g., there could be a single replace baz => ./baz if the paths are relative to the location of the single go.mod.local file, rather than needing to repeat variations of replace baz => ../baz if there are multiple go.mod.local files). This single go.mod.local file could be created programmatically by the go tool (e.g., creating the single go.mod.local file based on running against a shared file tree containing M modules and populating the single go.mod.local file with the various replace statements needed for each of those modules to find each other's local copy).

  2. A different approach would be to still have a single go.mod.local file that is in a shared parent in the file tree of the M modules, but rather than having N replace statements, the presence the file itself would be sufficient to imply that the M modules should be used locally, and the go.mod.local file itself could be empty. This is the same end result as if the go.mod.local file was created automatically with the N replace statements in 1., but in this case the replace statements would be implicit. Perhaps the file is called go.mod.local to mean "please use the local go.mod files", or perhaps the sentinel file is called something like go.workspace or go.space or some other better name.

  3. Rather than treating 1. vs. 2. as mutually exclusive, both could be supported options, where the behavior in 1. could be triggered by explicit replace statements like replace foo => ./foo, and the behavior in 2. could be triggered by some other statement inside the file (perhaps replace all, or replace relative, or similar).

Perhaps those are not good approaches, but of those three options, option 2. makes the most sense to me personally. It would help with a particular set of pain points around working with multiple modules simultaneously, but would also allow people to more easily organize their code around projects, and could also retain some of the benefits of how at least some gophers enjoy the way GOPATH has encouraged them to organize their code. Someone could for example create ~/go/src/go.workspace, but also ~/project1/go.workspace and ~/project2/go.workspace.

@kortschak

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2019

The second option also has the nice property that even if the go.mod.local/go.worskspace(/go.local?) were accidentally committed and pushed upstream, it would not leak internal information to the outside world. This is a concern that has driven some people to want a non-go.mod approach; the issue of accidentally pushing local filepaths to the outside world.

@akyoto

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

As an author of ~30 Go packages where testing 1 new package version equals 5-6 replace statements in the current system, I highly appreciate the idea of having a single global file to indicate all your local copies, as @thepudds mentioned. This single file could then be used for any go build in any directory.

Advantages:

  • Can't accidentally commit replace statements in go.mod anymore.
  • It takes less time to test my new version in the other 29 local packages that depend on it.
  • It opens up the possibility for a go link ~/projects/... style command where link finds all your git repositories inside ~/projects that have a go.mod file and automatically adds a global replace entry for each local copy.
@gunnsth

This comment has been minimized.

Copy link

commented May 20, 2019

As a developer in a fairly large library I frequently work with smaller applications that use this library. To be able to work with the local version of the library requires setting the replace directive for each small application. Setting a global replace directive would make this a lot easier. Otherwise it is definitely a regression from the GOPATH in terms of convenience.

It seems like a global replace directive could either be specified in a global file or through an environment variable, e.g. GOMODREPLACE=old[@v]=new[@v],... with multiple replacements comma separated.

@owais

This comment has been minimized.

Copy link

commented Jun 12, 2019

I use the replace directive quite a lot but this request scares me a little (at least the way it was requested originally). I fear people might forget that their local builds are using the .local file and end up testing against the wrong dependencies. Of course a good process with CI will catch such issues but IMO relying on external processes for correctness from the perspective of in-built Go tooling is wrong.

Imagine having to submit a very critical hotfix for something that needs to bypass the regular process. One might forget about the existence of a .local file or out of muscle memory use -local flag to build the program, test it locally and ship the code to prod. The remote build pipeline would not use the -local flag and would result in publishing an expected build. There can be many other such cases. For example, while reviewing a PR, I might want to checkout a branch to build and test the behavior locally but accidentally end up testing the branch with a local copy of the dependency. People learning Go might also find it frustrating to debug issues because of an old .local file lying around.

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.