New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: track tools/tooling updates to support modules #24661

Open
flibustenet opened this Issue Apr 3, 2018 · 50 comments

Comments

Projects
None yet
@flibustenet

flibustenet commented Apr 3, 2018

Many Go tools that previously worked with GOPATH need to be upgraded in order to work with modules.

This is a tracking issue for the conversion of tools to work with modules

@gopherbot gopherbot added this to the vgo milestone Apr 3, 2018

@rsc rsc changed the title from x/vgo how to works with tools like guru ? to x/vgo: integrate with guru, goimports, etc Apr 3, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Apr 3, 2018

Yes, this is an issue we're very aware of. Will leave this open to track it.

@myitcv

This comment has been minimized.

Member

myitcv commented Apr 7, 2018

@rsc - just to check this issue effectively encapsulates what we discussed in golang-dev?

@myitcv myitcv changed the title from x/vgo: integrate with guru, goimports, etc to x/vgo: integrate with guru, goimports, and other tools Apr 9, 2018

@myitcv myitcv changed the title from x/vgo: integrate with guru, goimports, and other tools to x/vgo: integrate with guru, goimports, and other tools/tooling Apr 9, 2018

@myitcv

This comment has been minimized.

Member

myitcv commented Apr 9, 2018

Changed the title so that this issue matches on use of the words "tool", "tools" or "tooling"

Will keep a list of tools here too (please edit):

  • Requiring build output:
    • guru
    • gocode
    • stringer
  • Requiring source code
@gopherbot

This comment has been minimized.

gopherbot commented Apr 9, 2018

Change https://golang.org/cl/105855 mentions this issue: x/vgo: add deplist command to get build information about (test) deps

myitcv added a commit to myitcv/vgo that referenced this issue Apr 30, 2018

x/vgo: add deplist command to get build information about (test) deps
This CL is a work in progress for golang/go#24661. Following the pattern laid
down by the vet command, we want to pass the build details of the
transitive (test) deps of a list of packages onto vetters/linters etc
that need to load imports of said packages for go/types (and similar)
analysis.

Fixes golang/go#24661

Change-Id: If1496dd6a3ed501ad6f124226a05f5d57284c57d

myitcv added a commit to myitcv/vgo that referenced this issue May 20, 2018

x/vgo: add deplist command to get build information about (test) deps
This CL is a work in progress for golang/go#24661. Following the pattern laid
down by the vet command, we want to pass the build details of the
transitive (test) deps of a list of packages onto vetters/linters etc
that need to load imports of said packages for go/types (and similar)
analysis.

Fixes golang/go#24661

Change-Id: If1496dd6a3ed501ad6f124226a05f5d57284c57d
@rsc

This comment has been minimized.

Contributor

rsc commented Jun 6, 2018

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018

@rsc rsc added the modules label Jul 12, 2018

@rsc rsc changed the title from x/vgo: integrate with guru, goimports, and other tools/tooling to cmd/go: integrate with guru, goimports, and other tools/tooling Jul 12, 2018

@gopherbot

This comment has been minimized.

gopherbot commented Jul 20, 2018

Change https://golang.org/cl/125296 mentions this issue: go/build: invoke go command to find modules during Import, Context.Import

gopherbot pushed a commit that referenced this issue Jul 28, 2018

go/build: invoke go command to find modules during Import, Context.Im…
…port

The introduction of modules has broken (intentionally) the rule
that the source code for a package x/y/z is in GOPATH/src/x/y/z
(or GOROOT/src/x/y/z). This breaks the code in go/build.Import,
which uses that rule to find the directory for a package.

In the long term, the fix is to move programs that load packages
off of go/build and onto golang.org/x/tools/go/packages, which
we hope will eventually become go/packages. That code invokes
the go command to learn what it needs to know about where
packages are.

In the short term, though, there are lots of programs that use go/build
and will not be able to find code in module dependencies.
To help those programs, go/build now runs the go command to
ask where a package's source code can be found, if it sees that
modules are in use. (If modules are not in use, it falls back to the
usual lookup code and does not invoke the go command, so that
existing uses are unaffected and not slowed down.)

Helps #24661.
Fixes #26504.

Change-Id: I0dac68854cf5011005c3b2272810245d81b7cc5a
Reviewed-on: https://go-review.googlesource.com/125296
Reviewed-by: Michael Matloob <matloob@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@myitcv

This comment has been minimized.

Member

myitcv commented Aug 1, 2018

@alandonovan @ianthehat - is it worth creating a number of issues, specific to each tool that's being worked on?

I keep directing people to this umbrella issue... but maybe there's merit in getting a bit more granularity on things? Equally, maybe not.

So I'm definitely wouldn't push back if you said "let's keep things here" 👍

@sam3d

This comment has been minimized.

sam3d commented Aug 2, 2018

How about the case of goimports importing packages that are both:

  • Within the module's local directory
  • Outside of the system scoped $GOPATH

Assuming the following directory structure:

./
├── go.mod
├── main.go
└── test
    └── test.go

With ./go.mod containing:

module example.com

goimports will happily leave alone and even format ./main.go:

package main

import (
	"log"

	"example.com/test"
)

func main() {
	log.Print("Hello, world!")
	test.Start()
}

With ./test/test.go:

package test

import "fmt"

func Start() {
	fmt.Println("Started properly")
}

However, if the "example.com/test" import is removed from main.go, it wont get added or found again. If its import path is changed in any way to render it incorrect (say to "example.com/tes"), goimports simply deletes the line.

It's odd - goimports can clearly tell that package test exists when it's been provided already, but it can't actively seek and add it if it hasn't been.

Is this something that requires looking into the module cache for, or is this a different issue?

@ianthehat

This comment has been minimized.

ianthehat commented Aug 2, 2018

goimports will assume that the import line "example.com/test" imports a package called test if it cannot find any sources for that package, and it knows the test symbol is used, so it won't delete the line. It does not mean that it knew the package existed...

goimports scans your GOPATH and GOROOT, anything not there it will not find right now.

It will be made to use x/tools/go/packages plus some extra stuff that is beyond the scope of that package. When it does, this case will work (does not involve the module cache, just the list of packages in the current module, which you can already get with go mod -packages

@sam3d

This comment has been minimized.

sam3d commented Aug 2, 2018

Thank you for explaining @ianthehat! I've misunderstood how goimports handles symbols.

@metakeule

This comment has been minimized.

metakeule commented Aug 11, 2018

I have two questions:

  1. will x/tools/go/packages move to the standard library with Go 1.12?
  2. I can't see any information about the version of a plugin, so how is a tool that assists in importing (like GoSublime) supposed to get a list of possible imports (with versions)?
@metakeule

This comment has been minimized.

metakeule commented Aug 11, 2018

Also: How to prevent that import completion will get really slow with larger projects? (see mdempsky/gocode#32)

@myitcv

This comment has been minimized.

Member

myitcv commented Aug 11, 2018

@metakeule

will x/tools/go/packages move to the standard library with Go 1.12?

The plan is for it to move to the standard library at some point, yes, once it has stabilised (ref).

I can't see any information about the version of a plugin

This is similar to the goimports "problem"; see #26717

How to prevent that import completion will get really slow with larger projects?

Many of the problems describe in mdempsky/gocode#32 relate to speed problems brought about by the use of the -source flag, which drives the use of the source importer. go/packages will work using the underlying build cache and so should not suffer these problems.

But in case lag does become an issue for gocode with its current architecture, go/packages will be offering support for long-running programs. So it's conceivable that gocode's cache will switch to be module-based, where it effectively "watches" for relevant changes and refreshes its cache accordingly. Its cache will then be primed for fast responses to ad-hoc queries.

cc @dominikh (in case there's any overlap of interest here with your work)

@solumos

This comment has been minimized.

solumos commented Nov 2, 2018

Are there any updates on issues related to go/packages or similar? I see that stringer is still unchecked above, and there's no activity on the linked PR (I'm planning a similar refactor from build to packages for percolate/charlatan#29).

Apologies if this isn't the right place for a question like this.

@ianthehat

This comment has been minimized.

ianthehat commented Nov 2, 2018

Stringer is a bit special, it requires us to vendor go/packages into the standard library, which we are not quite ready to do.
In general, go/packages is ready for use but still slower than the systems it is replacing, we are waiting on speed improvements to go list before merging some of the tools. Others that might be slightly less speed sensitive or not trigger the slow paths have just been converted already.
I would think for something like charlatan it should be fine, let us know how you get on.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 2, 2018

Minor point, but why is stringer special? It doesn't live in the standard library, it lives in x/tools/cmd/stringer.

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 2, 2018

In general, go/packages is ready for use but still slower than the systems it is replacing, we are waiting on speed improvements to go list before merging some of the tools.

Could we not put the modules-enabled changes behind a "standard" build tag, e.g. go111module (for some symmetry with the GO111MODULE env variable)?

It's a shame that the go tool's build-related commands don't allow the -tags flag to be repeated, otherwise it might have been possible to make use of build-tag guarded tools/code in more transparent way by setting GOFLAGS=-tags=go111module in one's profile/.bashrc/wherever.

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 4, 2018

Updated my previous comment; my testing with multiple -tags flags was clearly bogus. The GOFLAGS approach does work. Which, to my mind, makes maintaining a modules-aware version of a tool/package alongside the non-module-aware code in x/tools and friends a more attractive option.

The overhead will be similar to maintaining a separate branch/fork (of each tool), but the cost in terms of people now not needing to switch references to another repo will be much lower, which is a win.

@ianthehat

This comment has been minimized.

ianthehat commented Nov 5, 2018

I don't think that would really help.
The interesting cases right now are the IDE integrations, and asking them to build two copies from the same repository with different tags is probably not any easier than asking them to build two copies from different repositories. There is also a certain amount of overhead in having the person writing the fix have to maintain code in a repository they are not a maintainer of.
Maybe we should discuss this in the meeting tomorrow?

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 7, 2018

Maybe we should discuss this in the meeting tomorrow?

Just to note here, we concluded that a build-tags based approach is likely no better, indeed potentially worse, than the current planned approach. So no need to diverge from that for now.

@atombender

This comment has been minimized.

atombender commented Nov 10, 2018

@flibustenet Can you add Mockery to the list? Fixed.

@atombender

This comment has been minimized.

atombender commented Nov 10, 2018

@flibustenet Also, go-sumtype. Fixed.

@mtibben

This comment has been minimized.

mtibben commented Nov 13, 2018

Another tool: gqlgen

@powerman

This comment has been minimized.

powerman commented Nov 15, 2018

golangci-lint now works with go modules (uses go/packages), add it please

This is great as it let us work around incompatibility of many individual lint tools embedded in golangci-lint.

Maybe it's worth to re-prioritize work on tools in the list and give refactoring tools like gorename higher priority, because absence of such tools is a real blocker for moving projects out of GOPATH.

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 15, 2018

@powerman - gorename came up on the on the last golang-tools call. What editor do you use out of interest?

@keegancsmith

This comment has been minimized.

Contributor

keegancsmith commented Nov 15, 2018

I use gorename a lot. I do it from emacs. As a workaround I just run go mod vendor (also fixes other tools on that list, eg I use guru references a lot).

@myitcv

This comment has been minimized.

Member

myitcv commented Nov 15, 2018

Thanks @keegancsmith - I've added an agenda item for the next call on 27 Nov to make sure we cover this more thoroughly. In the meantime, @ianthehat / @alandonovan might be able to provide some more detail.

@powerman

This comment has been minimized.

powerman commented Nov 15, 2018

@powerman - gorename came up on the on the last golang-tools call. What editor do you use out of interest?

This is good news, thanks! (I've checked #27571 but there is no activity in last two months, that's why I've asked about priorities.) I'm using vim-go.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 15, 2018

@groob

This comment has been minimized.

Contributor

groob commented Nov 23, 2018

Is anyone tracking eg yet?
We use eg for a number of rewrites, but unfortunately it is not able to find dependencies without GOPATH and/or vendor/.

I end up with something like this in our Makefile:

 eg is not module aware, so we add the vendor/ directory, run generate without modules
# and then remove the vendor directory.
eg:
	go mod vendor
	GO111MODULE=off eg -w -t some.template github.com/groob/pkg
	rm -rf vendor/
@bcmills

This comment has been minimized.

Member

bcmills commented Nov 28, 2018

Does the godoc entry above cover cmd/doc, or should there be a separate entry for it?

@agnivade

This comment has been minimized.

Member

agnivade commented Nov 29, 2018

It doesn't actually. godoc is tracked separately here - #26827. It is just there on the list for completeness.

@bcmills bcmills added the help wanted label Nov 29, 2018

@casualjim

This comment has been minimized.

casualjim commented Nov 30, 2018

adding go-swagger to this list

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment