Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow go.mod.local to contain replace/exclude lines #26640

Open
maeglindeveloper opened this issue Jul 27, 2018 · 42 comments
Open

cmd/go: allow go.mod.local to contain replace/exclude lines #26640

maeglindeveloper opened this issue Jul 27, 2018 · 42 comments

Comments

@maeglindeveloper
Copy link

@maeglindeveloper maeglindeveloper commented Jul 27, 2018

Hi everyone,
I'm actually using a go.mod file for dependencies in my go project.

I saw that there is a way to use local packages instead of online repo, using the replace() features, which is really cool when developping multiple packages at the same time.

I was wondering if there is a way split in different files the require dependencies and the replace one. Something like that

go_replace.mod => define the replace part.
go.mod => define the module and the required dep, including the go_replace.mod.

the go_replace.mod will not be pushed on git, and each developer will create it if needed.

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Thanks a lot!

@rsc

@mvdan mvdan added the modules label Jul 27, 2018
@mvdan mvdan added this to the Go1.11 milestone Jul 27, 2018
@mvdan
Copy link
Member

@mvdan mvdan commented Jul 27, 2018

Could you clarify what is the purpose of separating the data into two files? So that developers don't commit temporary replace directives by mistake? I'd imagine that there could be tooling to help with that. For example, a tool to add and undo temporary directives, or having CI fail on unexpected replace directives.

I'm also worried about adding more special Go module files, even if they are in theory not to be committed into git.

@flibustenet
Copy link

@flibustenet flibustenet commented Jul 27, 2018

It's dangerous as you can commit files that use something else than what said in go.mod.
With replace in go.mod you see immediately (just before commit) that you will commit with a dependency fork.

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 2, 2018

when [developing] multiple packages at the same time

We certainly do need to address that use-case somehow, but splitting up the module definitions doesn't seem like an appropriate solution: ideally I don't want individual developers to have to edit the module definition to do that.

@rsc rsc changed the title cmd/go go.mod file & replace feature cmd/go: allow go.mod.local to contain replace/exclude lines Aug 9, 2018
@rsc rsc removed this from the Go1.11 milestone Aug 9, 2018
@rsc rsc added this to the Go1.12 milestone Aug 9, 2018
@cosban
Copy link

@cosban cosban commented Sep 3, 2018

I fall into the use case of "building multiple packages concurrently" as well.
I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications.
There are multiple solutions to this problem, some more elegant than others.

  • Having a release modifier or command (#26420) that ignores local copies would be a simple (from the user perspective) solution and would not require additional modifications to the go.mod files.
  • The solution proposed in this issue could also work. It could even be combined with the first proposal. @bcmills, when you say you "don't want individual developers to have to edit the module definition to do that", are you referring to them manualy modifying go.mod files? If not, could you please clarify your position a little more so I may understand it better? This could be resolved by allowing the developer to specify that the library is one which will be modified locally as well using something like go get -u <package@version> -local <relative/path/to/package>
  • Allowing users to install local, uncommitted versions could also work for this, but probably isn't the most elegant solution. This is, users could go install their current copy of the package they are developing, and the module system would store it in some sort of development version folder. In this method, version v0.0.0-20180526204631-c40f734b202a could be the latest commit, but v0.0.0-20180526204631-c40f734b202a-devel could be the latest local copy. If the local copy is not found, the module system would be able to revert back to the last commit when using the go get command.

I believe that the first option is the best, but I'm interested in reading what direction you guys are learning toward in the interim.

@wsc1
Copy link

@wsc1 wsc1 commented Oct 3, 2018

Hi all,

I'm not sure that my approach will help others, but in case it does, (taken from go-nuts)

There is a bare bones workaround without local go.mod nor replacements.

Suppose you need to work on module A and module B simultaneously (hopefully you set things up to avoid this, but hey, it's inevitable)

If you only need to edit one package at a time in at least one of A and B, then you don't need
to muck with local go.mod or versions tags or replacements or anything.

Let B be the module which has only one package to edit, and p the package.

Copy p somewhere in A, rename import path to p using existing go imports/ide,
edit both, build, test, etc. A's go.mod will be a little crazy and out of whack.
So what. Copy p back to where it belongs in B, re-run go imports, build/test or go mod tidy
Now commit as you like the two.

In terms of commands and tools, it's very simple: no tools, no replacements, no go.mod.local, no replacements which shouldn't be pushed/commited, no workflow restrictions. Simple "cp -r" and use with editor that understands go imports works fine.

Now, thanks to modules, if your edits correspond to releases, it doesn't even matter in what order
you commit/push the changes after the two modules have been edited.

Yes, it's still a PITA. For me it's much less painful than thinking about workflows and replacements.
But for me it also doesn't happen a lot.

That said, I do think it would be nice to have the go command do for a set of modules, say go build ./...
with ./ containing the modules in question. that would allow updates of all version in one shot for example.

@ohir
Copy link

@ohir ohir commented Oct 14, 2018

bcmills> "ideally I don't want individual developers to have to edit the module definition to do that."

Pondering on how to "merge" go.mod with the hypothetical go.mod.local I came to the point where only simplest solution stayed viable: add a -local knob. Then, if -local flag is present, tools will use go.mod.local instead of go.mod.

I know that knobs breeding is unpopular among core team but consider it, please.

@benhalstead
Copy link

@benhalstead benhalstead commented Feb 11, 2019

go.mod.local or similar would also help our use case. We have developers working on service 'A' which relies on library 'B'.

Our developers will make changes to both the service and the library and so have a replace in the service's go.mod file to use their local version of the library code instead.

The problem is that each developer will have checked the library out onto their filesystem with a different path so each of them need a different path in the 'replace' statement, meaning version control of go.mod is messy.

@zshbleaker
Copy link

@zshbleaker zshbleaker commented Mar 16, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

@bradleyjames
Copy link

@bradleyjames bradleyjames commented Mar 24, 2019

+1 to go.mod.local or just a separate file based approach. It would provide a clean logical separation and file based will be simple to ignore via .gitignore. Also overriding the mod file location (aka opt-in) during development would provide a sanity check in mitigating the risk of pushing to production.

@bradleyjames
Copy link

@bradleyjames bradleyjames commented Mar 24, 2019

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

We’re using GitHub Enterprise which works for us. What specific issues are you having?

@pwaller
Copy link
Contributor

@pwaller pwaller commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

I want to be able to do a git bisect with a replace directive in effect. For this to be effective, I want to avoid having a dirty working tree, so editing the go.mod isn't great for this.

I have found myself a few times wanting a replace directive without contaminating the git status output with changes, for example whilst just doing some development whilst working on multiple modules simultaneously.

@SamWhited
Copy link
Member

@SamWhited SamWhited commented Mar 27, 2019

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

Not everyone can see that link; if it's important, please copy any context needed into this discussion.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 27, 2019

@SamWhited, the discussion on Slack was literally just a link to this issue.
(I mostly treat Slack as a way to quickly route people to the relevant public issues.)

@p554157atch
Copy link

@p554157atch p554157atch commented Aug 20, 2020

Hello, please check this proposal as well: #40933

We can combine two different mechanisms,
using a go.local.mod plus importing replace definitions into go.local.mod from an existing local file.

For example go.local.mod for 20 different packages can have:

module hello
go 1.14
replace_definitons /home/coder/replace_definitions.mod

If we made changes to /home/coder/replace_definitions.mod, then changes will be applied to all the 20 go.local.mod files that import the replace_definitions.mod file.
go mod tidy takes care of un-used definitions.

This way git repository won't have unnecessary commits, plus we will be able to specify replace definitions in one local file and changes will be reflected in all local modules that import this file.

Making simultaneous edits on local modules that require constant updated to replace definitions will become much easier if we follow this method.

Let me know what you think.

@sheen4n

This comment has been minimized.

@mvdan

This comment was marked as off-topic.

@pipe01
Copy link

@pipe01 pipe01 commented Dec 2, 2020

An implementation I think could come in handy would be to look up on every parent directory for a replace.mod file, which contains replace and perhaps other directives.

@niemeyer
Copy link
Contributor

@niemeyer niemeyer commented Jan 7, 2021

With GO111MODULE defaulting to on by default in 1.16, it would be nice to address this issue somehow.

For clarity, the key problem problem most developers seem to be observing is not the lack of an additional mod file, but the fact there's no way to experiment with changes in local dependencies (for debugging, for example) without hacking on files that need to be skipped on every commit. An additional overriding file that could be ignored would be one way to address it, but it has the disadvantage of being silent about it. Ideally the final solution will make it visible that there are local overrides active during operation.

While this issue isn't sorted out upstream, developers may be interested in the --assume-unchanged flag of git update-index:

--[no-]assume-unchanged
    When this flag is specified, the object names recorded for the paths are
    not updated. Instead, this option sets/unsets the "assume unchanged" bit
    for the paths. When the "assume unchanged" bit is on, the user promises not
    to change the file and allows Git to assume that the working tree file
    matches what is recorded in the index. If you want to change the working
    tree file, you need to unset the bit to tell Git. This is sometimes helpful
    when working with a big project on a filesystem that has very slow lstat(2)
    system call (e.g. cifs).

    Git will fail (gracefully) in case it needs to modify this file in the
    index e.g. when merging in a commit; thus, in case the assumed-untracked
    file is changed upstream, you will need to handle the situation manually.

Some git aliases can make this a bit more convenient to use:

ghost = update-index --assume-unchanged
unghost = update-index --no-assume-unchanged
ghosted = ! git ls-files -v | grep '^[a-z]'
noghost = ! git ls-files -v | awk '/^[a-z]/ {system(\"git update-index --no-assume-unchanged \"$2)}'

This also has the disadvantage of being silent during operation. That's relatively easy to fix by automatically decorating commit messages to remind that go.mod is dirty.

@sabutterworth
Copy link

@sabutterworth sabutterworth commented Mar 2, 2021

I recently proposed #44660 with exactly the same issue in mind - I frequently need to use go.mod directives to fiddle with non-VCS packages (or newer code in local than in VCS) but NOT have them in the go.mod long-term.

Someone pointed me at this proposal which has a lot of discussion on it so I decided to close my proposal and add my suggestion here - to reduce the amount of 'duplicate' issues.

Suggestion:
When go is run without -modfile it looks for a file named go.mod.dev in the folder path resolved by GOMOD (or if -modfile is specified look for {thatmodfile}.dev at the same path). If found (and the module is the same) it uses a merge strategy to combine that with the contents of go.mod to resolve the correct directives to use - with directives from the .dev file having a higher precedence over the things in the go.mod file when conflicts are encountered.

@PeterFeicht
Copy link

@PeterFeicht PeterFeicht commented Mar 3, 2021

@sabutterworth so you're suggesting that even when I explicitly specify a .mod file by using -modfile, then the go command would still use another file in addition to that? That doesn't sound very sensible.

@ohir
Copy link

@ohir ohir commented Mar 3, 2021

@sabutterworth

I frequently need to use go.mod directives to fiddle with non-VCS packages (or newer code in local than in VCS) but NOT have them in the go.mod long-term.

This is common in bootstrap and debug phase.

When go is run without -modfile it looks for a file named go.mod.dev in the folder path resolved by GOMOD (or if -modfile is specified look for {thatmodfile}.dev at the same path). If found (and the module is the same) it uses a merge strategy to combine that with the contents of go.mod to resolve the correct directives to use - with directives from the .dev file having a higher precedence over the things in the go.mod file when conflicts are encountered.

For experiments go.mod should not be consulted at all. Just we need to tell compiler where our experimental code is. Then - after fiddling there - we may commit and publish new or fixed interdependent modules.

@complyue
Copy link

@complyue complyue commented Apr 10, 2021

FYI, in Haskell ecosystem, library authors tend to like Cabal, while application authors tend to like Stack; There is cabal.project.local supported, but not with similar mechanism upon stack.yaml , see: https://www.reddit.com/r/haskell/comments/hvybba/is_there_a_stack_equivalent_of_cabalprojectlocal/

Seems Go doesn't suffer from a split like cabal/stack wrt build tooling, but I kinda feel a similarity that GOPATH leans toward library authors (tend to develop multiple inter-related or even lock-step projects concurrently), while vending/go.mod leans toward application authors (focus on a single package module for most of the time).

So I think go.mod.local seemingly will be favored by lib authors, but can harm reproducibility that app authors value a lot. I observed many design decisions stack made to emphasis reproducibility, not to support stack.local.yaml might be one among others.

Personally I like #44347 GOTINKER over go.mod.local here, that seems to have the potential to make both lib and app authors happy.

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 12, 2021

I kinda feel a similarity that GOPATH leans toward library authors (tend to develop multiple inter-related or even lock-step projects concurrently), while vending/go.mod leans toward application authors (focus on a single package module for most of the time).

My impression is the opposite: I think most library authors are using modules at this point, because modules make it easier for their downstream consumers to resolve and manage transitive dependencies, and libraries already have unique import paths (generally beginning with the domain name hosting the library).

I think the majority of GOPATH users at this point are developers of private or self-contained applications. Private apps may be more difficult to migrate to modules because the existing package paths might not begin with a domain name compatible with go get.

@complyue
Copy link

@complyue complyue commented Apr 12, 2021

Sure that's the case, private apps are not on my mind previously, I agree they'll have to mass refactoring their import paths for the migration.

For typical library authors, I assume one would split functionality toward finer grained lib projects, but he must have some motivating use cases to be prototyped and tested with all those libs, or further plus one or more apps involved.

Suppose each lib and each app goes as a separate go.mod, or this isn't the case?

When you are planning several libs altogether, or to add major, inter-related features affecting multiple libs, they need to be prototyped & tested with WIP branches. Temporarily placing local replaces in one go.mod is not a big deal, but how about maintaining ~10 intermediate go.mod files at the same time? For myself, I would feel painful in such workflows, even if CI can be configured to automate removal of those local replaces, it's very unwieldy in preparing a staging area for effective feature branches of multiple libs. And I can't see automation in tooling can bring a rival experience as smooth as the traditional GOPATH provides.

@gilgad13
Copy link

@gilgad13 gilgad13 commented Apr 13, 2021

Placing this issue in context with Russ's well-written plea here: #37755 (comment)

Yes, go modules are obviously superior to GOPATH. They enable reproducible builds. They allow different components to depend on different versions of the same third-party repository. They solve all the security problems recently highlighted with pip, in triplicate. However, at engineering companies of a certain size, you’ve already had to solve these problems. And, furthermore, you’ve had to solve them for a variety of languages, so your solution is cross-language. Existing processes and tools are defined in terms of this meta-builder, so institutional inertia requires compatibility.  It also allows dependencies on specific go and protoc versions.  At $dayjob, we have a custom Make-based framework, for Google I’m guessing Bazel / Blaze fills that role, and for a project like Debian its the control files and auto builder network. If you already have such a system, integrating it with GOPATH is easy, just ask that system to form a valid GOPATH with the correct versions in place everywhere (as Debian does). With go modules, it's much, much harder. The official answer as I understand it is to call directly in to go tool compile and go tool link.  However, bypassing the high-level go build command means you may need to reimplement editor and tool integrations, which is a very high bar.  Even at a conceptual level its hard because now both the go compiler and the overarching tool want to control how dependencies are fulfilled. I haven’t seen a clear story on how to integrate go modules into such a system.

I think some participants in this thread (myself included) are attempting to recreate their previous GOPATH workflows using the replace directive.  For this use case, the -modfile flag (perhaps encoded in GOFLAGS) is nice, but is still difficult to work with because you need to create an override file for each dependency you pull in to the "workspace", and regardless of whether you pass the -modfile argument as a relative or absolute path it is difficult to select the correct modfile based on the the directory a command is being run in.  I believe this is why there is interest in having a single file that defines a "workspace", which overrides the module discovery process for all checked-out modules within it.

@complyue
Copy link

@complyue complyue commented Apr 14, 2021

I just realized that go.mod corresponds to what's called "package" elsewhere, I feel we need a construct bigger than the "Go module" in Go tooling's concept space, to properly model the source structure of Go software. So I'd add that I don't think go.mod.local is the solution, for it be at the same level as go.mod.

Go uses the term "package" for directory of source files, while in more general software engineering context, "package" as in "Package Manager", "Pip - the Package Installer for Python" and etc. refers to something bigger than "module". So now Go has the relative size of "package" and "module" flipped, this will likely to confuse even exclusive Go developers if unaware, for they are likely, e.g. to work with "apt - Advanced Package Tool" day-to-day in using their servers, workstations or laptops.

We need to start talking about the concept for the thing bigger than "package" or "Go module" in tooling related issues, currently the data model of Go tooling wrt source project structure lacks a layer of project workspace, so maybe the term "project" or "workspace" or my personal idea "go.farm" will do, just don't stop at go.mod as the biggest unit of Go software.

Update: cross link #27542 (comment)

@thepudds
Copy link

@thepudds thepudds commented Apr 15, 2021

Hi @complyue @gilgad13, You might be interested in #26640 (comment), which is a sketch from a while ago of how a go.workspace or go.space could work to allow working multiple modules simultaneously without needing to do anything manually regarding replace, and would allow people to more easily organize their code around projects. Of the alternatives sketched there, I personally prefer option 2 there.

One question that came up in a prior discussion of that sketch is how would a release happen, including to check if a given module still builds and passes tests using the published versions of the modules in the local workspace. One approach would be to use mv as an extremely minimalistic “API“ to temporarily rename the go.workspace file or move modules temporarily out of the workspace. That has the benefit of being minimalistic, tangible, and explicit, but might be too minimalistic. Alternatively, a new -mod=foo mode might be able to say “use published versions, not workspace versions”.

@complyue
Copy link

@complyue complyue commented Apr 15, 2021

@thepudds I confess I missed your posts (as well as many others') by not thoroughly reading all the discussions, a shame but I don't have too much time/energy available on Go stuff as time being. And I'm glad to know you already sketched the idea and many have discussed about it.

With respect to the release concern, I'm no expert at it, but AFAICT, similar concerns had driven a gang of Haskellers to have created Stackage, it recruit curators to work on nightly snapshots of the central registry/repository of Haskell packages (https://hackage.haskell.org), making sure all packages are vetted (compiles, passing test suites, etc.) then regularly release such snapshots as LTS Haskell. Stackage Curators are great people and smart CI tools making all that happen.

Back to the release concern we talk about here, I do think it's okay if all modules in a local go.workspace are engineered in a lock-step fashion, but for wider collaboration among upstream/downstream projects (inevitable for open source development), I don't think that solvable by a simple tool. More intensive processes/workflows with dedication of human forces might be inevitable, and https://www.snoyman.com/blog/2018/11/why-i-believe-stackage-succeeded may provide an illustration for what it will look like after done right.

@slatermorgan
Copy link

@slatermorgan slatermorgan commented Aug 12, 2021

I too have come across this problem, it would be lovely to have go.space for local development!

For the time being, couldn't a solution be some sort of bash script which could append the replace directives from another file to the go.mod before building the project in the local development environment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet