Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: missing dependendency in go.mod file #25371

Closed
ardan-bkennedy opened this issue May 13, 2018 · 4 comments

Comments

@ardan-bkennedy
Copy link

commented May 13, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.2 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Using this project
https://github.com/ardanlabs/service

I added an empty go.mod file at the root of the project.
I ran the vgo build command
Once that was done, the go.mod file contained the following

module github.com/ardanlabs/service

require (
	github.com/dimfeld/httptreemux v0.0.0-20180213074414-7f532489e773
	github.com/pborman/uuid v0.0.0-20180122190007-c65b2f87fee3
	github.com/pkg/errors v0.8.0
	go.opencensus.io v0.9.0
	gopkg.in/go-playground/validator.v8 v8.18.2
	gopkg.in/mgo.v2 v2.0.0-20160818020120-3f83fa500528
)

What did you expect to see?

There is a dependency in my vendor folder that is missing from the go.mod file that is being used.

github.com/openzipkin/zipkin-go

I expected to see that in the go.mod file.

When I run the vgo list -m command I do see this in the list.

$ vgo list -m -u
MODULE                                            VERSION                             LATEST
github.com/ardanlabs/service                      -                                   -
cloud.google.com/go                               v0.21.0 (2018-04-09 13:38)          v0.22.0 (2018-05-09 10:25)
github.com/beorn7/perks                           v0.0.0-20180321164747-3a771d992973  -
github.com/dimfeld/httptreemux                    v0.0.0-20180213074414-7f532489e773  -
github.com/golang/protobuf                        v1.0.0 (2018-01-25 13:43)           v1.1.0 (2018-04-30 11:52)
github.com/googleapis/gax-go                      v0.0.0-20170915024731-317e0006254c  v1.0.0 (2017-09-01 17:52)
github.com/matttproud/golang_protobuf_extensions  v1.0.0 (2016-04-23 10:36)           -
github.com/openzipkin/zipkin-go                   v0.1.0 (2018-02-27 02:13)           -
github.com/pborman/uuid                           v0.0.0-20180122190007-c65b2f87fee3  -
github.com/pkg/errors                             v0.8.0 (2016-09-28 18:48)           -
github.com/prometheus/client_golang               v0.8.0 (2016-08-17 08:48)           v0.9.0-pre1 (2017-08-31 06:19)
github.com/prometheus/client_model                v0.0.0-20171117100541-99fa1f4be8e5  -
github.com/prometheus/common                      v0.0.0-20180413074202-d0f7cd64bda4  v0.0.0-20180426121432-d811d2e9bf89
github.com/prometheus/procfs                      v0.0.0-20180408092902-8b1c2da0d56d  -
go.opencensus.io                                  v0.9.0 (2018-04-30 11:08)           -
golang.org/x/net                                  v0.0.0-20180406214816-61147c48b25b  v0.0.0-20180511174649-2491c5de3490
golang.org/x/oauth2                               v0.0.0-20180402223937-921ae394b943  v0.0.0-20180503012634-cdc340f7c179
golang.org/x/sync                                 v0.0.0-20180314180146-1d60e4601c6f  -
golang.org/x/text                                 v0.3.0 (2017-12-14 05:08)           -
google.golang.org/api                             v0.0.0-20180413000347-fca24fcb4112  v0.0.0-20180513000358-15aea8711db9
google.golang.org/appengine                       v1.0.0 (2016-09-29 13:31)           -
google.golang.org/genproto                        v0.0.0-20180409222037-51d0944304c3  v0.0.0-20180427144745-86e600f69ee4
google.golang.org/grpc                            v1.11.3 (2018-04-09 13:31)          v1.12.0 (2018-05-08 14:01)
gopkg.in/go-playground/validator.v8               v8.18.2 (2017-07-29 22:02)          -
gopkg.in/mgo.v2                                   v2.0.0-20160818020120-3f83fa500528  -

What did you see instead?

I just expected all my vendored dependencies to show up in the go.mod file.

@bcmills bcmills changed the title Missing dependendency in go.mod file x/vgo: missing dependendency in go.mod file May 13, 2018

@gopherbot gopherbot added this to the vgo milestone May 13, 2018

@oiooj

This comment has been minimized.

Copy link
Member

commented May 14, 2018

This issue occured if your project and one of your imports have a common dep, and the version is also equal.

In your codebase, github.com/ardanlabs/service needs github.com/openzipkin/zipkin-go, and go.opencensus.io also needs github.com/openzipkin/zipkin-go.

go.opencensus.io will be written into go.mod, but github.com/openzipkin/zipkin-go will not be.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jun 15, 2018

As @oiooj notes, the go.mod file only contains requirements that are not implied transitively.
(Also recall that vgo build ignores the vendor/ directory in general.)

You should find that vgo vendor still copies the appropriate version into your vendor/ directory.

We certainly do need to work on some concise documentation for these sorts of properties, but I think that's a separate matter from this issue itself. Please feel free to reopen if you disagree.

@bcmills bcmills closed this Jun 15, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2018

Actually, @bcmills realized that the build list algorithm assumes that direct dependencies are listed explicitly in go.mod, so we'll start doing that, even when the result is not minimal. Will send a CL probably tomorrow.

@rsc rsc reopened this Jun 27, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Jun 28, 2018

Change https://golang.org/cl/121304 mentions this issue: cmd/go/internal/vgo: track directly-used vs indirectly-used modules

@rsc rsc closed this Jul 3, 2018

gopherbot pushed a commit to golang/vgo that referenced this issue Jul 10, 2018
cmd/go/internal/vgo: track directly-used vs indirectly-used modules
A cleanup pass in mvs.BuildList discards modules that are not reachable
in the requirement graph as satisfied for this build. For example, suppose:

	A -> B1, C1
	B1 -> D1
	B2 -> nothing
	C1 -> nothing
	D1 -> nothing
	D2 -> nothing

The effective build list is A, B1, C1, D1 (no cleanup possible).

Suppose that we update from B1 to B2. The effective build list
becomes A, B2, C1, D1, and since there is no path through those
module versions from A to D, the cleanup pass drops D.

This cleanup, which is not in https://research.swtch.com/vgo-mvs,
aims to avoid user confusion by not listing irrelevant modules in
the output of commands like "vgo list -m all".

Unfortunately, the cleanup is not sound in general, because
there is no guarantee all of A's needs are listed as direct requirements.
For example, maybe A imports D. In that case, dropping D and then
building A will re-add the latest version of D (D2 instead of D1).
The most common time this happens is after an upgrade.

The fix is to make sure that go.mod does list all of the modules
required directly by A, and to make sure that the go.mod
minimizer (Algorithm R in the blog post) does not remove
direct requirements in the name of simplifying go.mod.

The way this is done is to annotate the requirements NOT used
directly by A with a known comment, "// indirect".

For example suppose A imports rsc.io/quote. Then the go.mod
looks like it always has:

	module m

	require rsc.io/quote v1.5.2

But now suppose we upgrade our packages to their latest versions.
Then go.mod becomes:

	module m

	require (
        		golang.org/x/text v0.3.0 // indirect
        		rsc.io/quote v1.5.2
        		rsc.io/sampler v1.99.99 // indirect
        	)

The "// indirect" comments indicate that this requirement is used
only to upgrade something needed outside module m, not to satisfy
any packages in module m itself.

Vgo adds and removes these comments automatically.
If we add a direct import of golang.org/x/text to some package in m,
then the first time we build that package vgo strips the "// indirect"
on the golang.org/x/text requirement line. If we then remove that
package, the requirement remains listed as direct (the conservative
choice) until the next "vgo mod -sync", which considers all packages
in m and can mark the requirement indirect again.
Algorithm R is modified to be given a set of import paths that must
be preserved in the final output (all the ones not marked // indirect).

Maintenance of this extra information makes the cleanup pass safe.

Seeing all directly-imported modules in go.mod
and distinguishing between directly- and indirectly-imported modules in go.mod
are two of the most commonly-requested features,
so it's extra nice that the fix for the cleanup-induced bug
makes go.mod closer to what users expect.

Fixes golang/go#24042.
Fixes golang/go#25371.
Fixes golang/go#25969.

Change-Id: I4ed0729b867723fe90e836c2325f740b55b2b27b
Reviewed-on: https://go-review.googlesource.com/121304
Reviewed-by: Bryan C. Mills <bcmills@google.com>

@golang golang locked and limited conversation to collaborators Jul 3, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants
You can’t perform that action at this time.