New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dep cannot vendor dependencies that do not have Go source code #1306

Open
davecheney opened this Issue Oct 24, 2017 · 30 comments

Comments

Projects
None yet
@davecheney
Contributor

davecheney commented Oct 24, 2017

What version of dep are you using (dep version)?

What dep command did you run?

I am trying to convert a project that uses glide to vendor two repositories containing protocol buffer definitions. The abstract from glide.yaml is

hash: 2f5e7adae8a174aebf478c2aac823837001c82d40f647fd9da15be270739057c
updated: 2017-10-23T16:03:08.007589083-04:00
imports:
- name: github.com/envoyproxy/data-plane-api
  version: 5b29f002b44f3f6e3921bd3b4535aeb91f89e84a
- name: github.com/googleapis/googleapis
  version: 5c6df0cd18c6a429eab739fb711c27f6e1393366
- name: github.com/golang/protobuf
  version: ab9f9a6dab164b7d1246e0e688b0ab7b94d8553e
  subpackages:
  - jsonpb
  - proto
  - protoc-gen-go/descriptor
  - ptypes
  - ptypes/any
  - ptypes/duration
  - ptypes/struct
  - ptypes/timestamp
  - ptypes/wrappers

The two repositories in question are github.com/envoyproxy/data-plane-api and github.com/googleapis/googleapis. When I try to add

+
+[[constraint]]
+  name="github.com/envoyproxy/data-plane-api"
+  revision="5b29f002b44f3f6e3921bd3b4535aeb91f89e84a"
+
+[[constraint]]
+  name="github.com/googleapis/googleapis"
+  revision="5c6df0cd18c6a429eab739fb711c27f6e1393366"

dep ensure refuses to vendor the repositories

What did you expect to see?

those two repositories vendored into vendor/

What did you see instead?

% dep ensure
ensure Solve(): No versions of github.com/envoyproxy/data-plane-api met constraints:
        5b29f002b44f3f6e3921bd3b4535aeb91f89e84a: Could not introduce github.com/envoyproxy/data-plane-api@5b29f002b44f3f6e3921bd3b4535aeb91f89e84a, as its subpackage github.com/envoyproxy/data-plane-api does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)
        master: Could not introduce github.com/envoyproxy/data-plane-api@master, as its subpackage github.com/envoyproxy/data-plane-api does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)
        typo: Could not introduce github.com/envoyproxy/data-plane-api@typo, as it is not allowed by constraint 5b29f002b44f3f6e3921bd3b4535aeb91f89e84a from project github.com/heptio/zzz.
@markcampanelli

This comment has been minimized.

markcampanelli commented Oct 25, 2017

I hit same issue when trying to use dep to manage some non-Go build tools.

@markuswustenberg

This comment has been minimized.

markuswustenberg commented Oct 26, 2017

This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):

$ dep ensure -v -add github.com/docker/docker       
Fetching sources...
(1/1) github.com/docker/docker

Root project is "code.uber.internal/infra/dep-no-go-code"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)	? attempt github.com/docker/docker with 1 pkgs; 206 versions to try
(1)	    try github.com/docker/docker@v1.13.1
(2)	✗   github.com/docker/docker at v1.13.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.13.0
(2)	✗   github.com/docker/docker at v1.13.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.6
(2)	✗   github.com/docker/docker at v1.12.6 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.5
(2)	✗   github.com/docker/docker at v1.12.5 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.4
(2)	✗   github.com/docker/docker at v1.12.4 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.3
(2)	✗   github.com/docker/docker at v1.12.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.2
(2)	✗   github.com/docker/docker at v1.12.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.1
(2)	✗   github.com/docker/docker at v1.12.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.12.0
(2)	✗   github.com/docker/docker at v1.12.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.2
(2)	✗   github.com/docker/docker at v1.11.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.1
(2)	✗   github.com/docker/docker at v1.11.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.11.0
(2)	✗   github.com/docker/docker at v1.11.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.3
(2)	✗   github.com/docker/docker at v1.10.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.2
(2)	✗   github.com/docker/docker at v1.10.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.1
(2)	✗   github.com/docker/docker at v1.10.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.10.0
(2)	✗   github.com/docker/docker at v1.10.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.9.1
(2)	✗   github.com/docker/docker at v1.9.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.9.0
(2)	✗   github.com/docker/docker at v1.9.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.3
(2)	✗   github.com/docker/docker at v1.8.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.2
(2)	✗   github.com/docker/docker at v1.8.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.1
(2)	✗   github.com/docker/docker at v1.8.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.8.0
(2)	✗   github.com/docker/docker at v1.8.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.7.1
(2)	✗   github.com/docker/docker at v1.7.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.7.0
(2)	✗   github.com/docker/docker at v1.7.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.2
(2)	✗   github.com/docker/docker at v1.6.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.1
(2)	✗   github.com/docker/docker at v1.6.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.6.0
(2)	✗   github.com/docker/docker at v1.6.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.5.0
(2)	✗   github.com/docker/docker at v1.5.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.4.1
(2)	✗   github.com/docker/docker at v1.4.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.4.0
(2)	✗   github.com/docker/docker at v1.4.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.3
(2)	✗   github.com/docker/docker at v1.3.3 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.2
(2)	✗   github.com/docker/docker at v1.3.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.1
(2)	✗   github.com/docker/docker at v1.3.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.3.0
(2)	✗   github.com/docker/docker at v1.3.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.2.0
(2)	✗   github.com/docker/docker at v1.2.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.2
(2)	✗   github.com/docker/docker at v1.1.2 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.1
(2)	✗   github.com/docker/docker at v1.1.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.1.0
(2)	✗   github.com/docker/docker at v1.1.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.0.1
(2)	✗   github.com/docker/docker at v1.0.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v1.0.0
(2)	✗   github.com/docker/docker at v1.0.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.12.0
(2)	✗   github.com/docker/docker at v0.12.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.11.1
(2)	✗   github.com/docker/docker at v0.11.1 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.11.0
(2)	✗   github.com/docker/docker at v0.11.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.10.0
(2)	✗   github.com/docker/docker at v0.10.0 has problem subpkg(s):
(2)	  	github.com/docker/docker has err (*build.NoGoError); required by (root).
(1)	    try github.com/docker/docker@v0.9.1
(1)	✓ select github.com/docker/docker@v0.9.1 w/1 pkgs

Version:

$ dep version
dep:
 version     : devel
 build date  : 
 git hash    : 
 go version  : go1.9.1
 go compiler : gc
 platform    : darwin/amd64
@sdboyer

This comment has been minimized.

Member

sdboyer commented Oct 27, 2017

yeah...this is tricky. i'm considering the possibility that we change the rules on required to not complain if the named packages lack Go code. alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import. that would open up the possibility of false positives with real imports...but that seems a lot less likely to be a real problem than the false negatives being described here.

the only other alternative i can think of right now is something like #269, which is a whole other can of worms.

@markuswustenberg

This also happens when fetching e.g. github.com/docker/docker, which doesn't have Go code in the root (since v0.9.1 at least):

yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable. this is one of those confounded expectations arising from bumping up against the model, again - folks are thinking, "i want to grab this down and just have it there so i can reference stuff from it," but dep's model is "you don't actually use these, so we're gonna get rid of it."

this is gonna be a point of friction for a while, unfortunately. the easiest way to make e.g. your editor's autocomplete work is to just trick it by also having that project on GOPATH - and by the time that we get around to adding the new storage area, loosely outlined here, i think most of this pain should go away. but it's gonna be a while.

@markuswustenberg

This comment has been minimized.

markuswustenberg commented Oct 27, 2017

Thanks for your answer @sdboyer.

alternatively, we could relax the "no go code" rule in general and consider any files that may possibly be usable to Go good enough to make a package "valid" for import.

In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction. Maybe there could just be an option to override this behaviour?

yeah, the design here implicitly expects that you give -add a package - not necessarily a project root - that is actually importable

In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?

@sdboyer

This comment has been minimized.

Member

sdboyer commented Oct 27, 2017

In my view, any arbitrary binary is potentially usable by Go code, so that's seems a bit of an odd restriction.

the key here is the definition of "usable" - dep is, for now, primarily concerned with what is necessary to compile Go code. not generate; not run. it's nice when we can also capture those things, but for now, they're not our primary objective. that puts arbitrary binaries out of scope, unless i've missed something quite big about extensible compiler capabilities. (entirely possible!)

In this case, I'm trying to grab the docker client which is under github.com/docker/docker/client, but dep won't let me import that directly. Is there any way to get this working with the current tools?

dep ensure -add github.com/docker/docker/client

@discordianfish

This comment has been minimized.

discordianfish commented Nov 10, 2017

I have a related issue. Not sure if I should open a new issue for that though:
I'm building a Dockerfile and I want to add Gopkg.* first, then run dep ensure before I add the code in a next step. That's a common pattern which allows Docker to cache the dependency-install step, so that I don't have to run this every time I change the code. Due to the 'no go code' rule this isn't possible when using dep.

@sdboyer

This comment has been minimized.

Member

sdboyer commented Nov 10, 2017

@discordianfish this issue is specifically about the case where there's no Go code in a dependency, which is different from having no Go code in the current project. dep ensure -vendor-only should cover your case.

@discordianfish

This comment has been minimized.

discordianfish commented Nov 10, 2017

Oh perfect, thanks! Sorry for the noise.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Nov 10, 2017

@jhayotte

This comment has been minimized.

jhayotte commented Nov 21, 2017

Hi @sdboyer,
I tried out to migrate from Glide to Dep and dependencies including a list of protofile like https://github.com/googleapis/googleapis/tree/master/google/api cannot be vendored.

Would be nice to have a workaround in the meantime. Many people are trying to use dep and are blocked by this issue :/

@sdboyer

This comment has been minimized.

Member

sdboyer commented Nov 21, 2017

yeah, i don't see any problems with expanding the definition of required to cover this case. so we have a path to a solution - just, need the elbow grease to get it done.

@kris-nova

This comment has been minimized.

Contributor

kris-nova commented Nov 21, 2017

Happy to lend some commits while I am enjoying my time of being unemployed if you want to assign this or any other issues to me

@jhayotte

This comment has been minimized.

jhayotte commented Nov 22, 2017

Hi @sdboyer,

I can contribute to this issue as well. We just need to clarify your expectations.

Do you wish that we remove the constraint looking for *.go files? (IMO that's what we should do) or do we handle it as a warning?

dep/gps/pkgtree/pkgtree.go

Lines 219 to 226 in 13df556

gofiles, err := filepath.Glob(filepath.Join(p.Dir, "*.go"))
if err != nil {
return err
}
if len(gofiles) == 0 {
return &build.NoGoError{Dir: p.Dir}
}

@travisjeffery

This comment has been minimized.

travisjeffery commented Nov 23, 2017

Here's a PR for it #1398. Feedback welcome, thanks.

@sdboyer

This comment has been minimized.

Member

sdboyer commented Nov 23, 2017

i'm delighted to see all the offers to help ❤️ ❤️ this is how these things get done!

but yeah, lemme clarify the requirements here.

we can't mess with ListPackages() in the way @jhayotte has outlined. that's fixing this problem by lying about the state of the world, and ListPackages() needs to not do that. we can't change what it reports in order to achieve a desired outcome in the solver; instead, we need to change the solver to deal with this more appropriately.

to the specifics. right now, the behavior of the solver is that:

  1. if a target import path has either been included in the required list from Gopkg.toml, or if it's in an import statement (we do not currently differentiate between these two cases)
  2. and that target import path does not exist within its containing project
  3. or that target import path has any errors within its containing project
  4. then consider it a failure condition

what we want to do here is tease apart actual import statements from Gopkg.toml's required. the behavior for imports should remain the same, but:

  1. if a target import path has been given in the required list from Gopkg.toml
  2. and that target import path does not exist within its containing project
  3. or that target import path has an error OTHER THAN a *build.NoGoError
  4. then consider it a failure condition

this achieves the outcome i described in an earlier comment: #1306 (comment)

there's going to be more work to do here. in addition to likely needing to update some of gps' tests, we'll want at least one new one to cover this particular path (and probably a few to cover combinations of conditions). crucially, we're also going to need to change how the hasherdoes its work, as we're changing the semantics of imports and required's so that they're no longer identical: we'll need to break them out into separate groups.

this change will also then entail a bump o the solver-version that you see in Gopkg.lock, which we'll have to do a little bit of footwork around, as i don't think we've accommodated that in dep yet.

@johanbrandhorst

This comment has been minimized.

Member

johanbrandhorst commented Dec 18, 2017

Lots of offers of work but I see no open PRs? Is anyone working on this at the moment? It's a blocker for our transition (and I unfortunately do not have spare cycles at this time 😞 ).

@sdboyer

This comment has been minimized.

Member

sdboyer commented Dec 23, 2017

nope, nobody is on this at the moment, unfortunately. it involves solver changes, so it's a rather tricky task to take on.

carolynvs added a commit to carolynvs/service-catalog that referenced this issue Jan 17, 2018

Run dep init to migrate from glide
* This does not include the code generators, such as gogen, because of
an open issue in dep: golang/dep#1306.
* This adds a new dependency that was not previously identified with
glide: github.com/petar/GOLLRB
@carolynvs

This comment has been minimized.

Collaborator

carolynvs commented Jan 18, 2018

I am working on this, there was a lot of overlap between the changes needed for this and the transitive constraint prototype so I had a head start. 😀

carolynvs added a commit to carolynvs/service-catalog that referenced this issue Jan 18, 2018

Run dep init to migrate from glide
* This does not include the code generators, such as gogen, because of
an open issue in dep: golang/dep#1306.
* This adds a new dependency that was not previously identified with
glide: github.com/petar/GOLLRB
@dt665m

This comment has been minimized.

dt665m commented Mar 20, 2018

Is there currently a work around for npulling non .go file dependencies? I have some cgo and some dynamic c++ libs that aren't getting pulled by dep. Is there any way to force dep to pull the whole repository? (in this case, github).

@carolynvs

This comment has been minimized.

Collaborator

carolynvs commented Mar 20, 2018

AFAIK, if the repo doesn't contain go code anywhere, there is no workaround yet. It is on our backlog to address. However if there is at least one directory with go code, then you can require that package from the repo, and then don't use pruning of non-go files or unused packages so that the entire repo is vendored.

@zhexuany

This comment has been minimized.

zhexuany commented Mar 28, 2018

@sdboyer I hit a confusing error.

In a project which depends on our one project tidb, I ran dep ensure and dep gave me the following error:

Solving failure: No versions of github.com/pingcap/tidb met constraints:
    master: Could not introduce github.com/pingcap/tidb@master due to multiple problematic subpackages:
    Subpackage github.com/pingcap/tidb/sessionctx does not contain usable Go code (*build.NoGoError).. (Package is required by (root).)    Subpackage github.com/pingcap/tidb/store/mockstore is missing. (Package is required by (root).)    Subpackage github.com/pingcap/tidb/session is missing. (Package is required by (root).)

The thing is subpackage session and sessionctx contains useable go code. Could you please help m out?

session's link: https://github.com/pingcap/tidb/tree/master/session
sessionctx's link: https://github.com/pingcap/tidb/tree/master/sessionctx

@jackmanlabs

This comment has been minimized.

jackmanlabs commented Mar 30, 2018

FYI, this is also an issue with https://github.com/go-qml/qml. This package depends on C++ files located in a subdirectory.

@hamid-elaosta

This comment has been minimized.

hamid-elaosta commented Apr 26, 2018

@carolynvs This is an interesting point, because what you suggest above does not appear to be the behaviour, perhaps I'm missing something.

I have a Go project A, which I'm trying to vendor into another Go project B. From A I am using only a Proto Buf .proto file, which resides in a subpackage of A. I am vendoring the top-level package, not the sub-package, and I have no pruning or other cleanup/removal configured in my toml, yet dep insists it cannot vendor A because its sub-package (the one I'm getting the proto file from) does not contain Go code.

Is there an override I can use to vendor the entirety of project A? I had assumed vendoring its top-level would vendor it entirely, regardless if I'm using only a file from a sub-package in my B project.

@hamid-elaosta

This comment has been minimized.

hamid-elaosta commented Apr 26, 2018

@carolynvs Thanks for the links. Unfortunately neither seems to work (and dep complains about redundant prune options) I'm not sure if it's some combination of my project requirements that is preventing it but the error message didn't change.

The project I'm attempting to vendor is on a specific branch (non-master), the constraint is overridden with an SSH path because it's a private (corporate) git server, and my local copy was not go-g[o]t but git cloned (because go-get won't check out over ssh from our git server).

EDIT: I should add, dep ensure worked when I had a go file in the sub-package that contains the proto, but it was removed in a recent update to that project (it's now generated after checkout).

@tangblue

This comment has been minimized.

tangblue commented May 16, 2018

Hit the same issue when trying to use dep to get HTML static file from another project.

@mysugar

This comment has been minimized.

mysugar commented Jun 1, 2018

I tried to use dep to get github.com/spinlock/jemalloc-go which is a c project.
Then i update the Gopkg.toml file , here is my file

# [prune]
#   non-go = false
#   go-tests = true
#   unused-packages = true
required = ["github.com/spinlock/jemalloc-go"]
[[override]]
  name = "github.com/spinlock/jemalloc-go"
  revision = "26719b2ee6186ef794cd50b604f8d765d3be5a30"
[prune]

Then i got all files in my vendor dir. So the config worked for me.Hope it can help others.
My dep version is 0.4.1.

ghost pushed a commit to spdk/spdk that referenced this issue Jun 28, 2018

go: empty Go package
The only purpose at the moment is to allow vendoring of the SPDK
source code into Go projects which manage dependencies with the "dep"
tool. That tool is designed for Go projects and as a sanity check
rejects any dependency which does not have at least one Go file, at
least at the moment (golang/dep#1306).

Change-Id: I22458d0895729ad7d8ea53e7541e9089da1d448a
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Reviewed-on: https://review.gerrithub.io/416831
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
@ZhenhangTung

This comment has been minimized.

ZhenhangTung commented Jul 25, 2018

@zhexuany I think your problem is not related to this issue, and you could open a new issue or reach out to the community if this problem still bothers you.

The thing is subpackage session and sessionctx contains useable go code. Could you please help m out?

The context.go file in sessionctx package was created in this commit, which is on 22 Feb, so not sure if you was importing the version before this commit, since you didn't give us any conext or logs when dep was running.
From my understanding, useable go code doen't mean that you have some subpackages which contains go files, it means that there should be go files under this package.
Hope this could help you.

@msolimans

This comment has been minimized.

msolimans commented Aug 2, 2018

tried to avoid pruning non-go files but did not work by just turning it off using non-go = false however removing root prune at all was good enough to do the magic for me and to include everything I want. didn't like to include unused-packages too but at least it is working for now

andrewkroh added a commit to andrewkroh/beats that referenced this issue Aug 23, 2018

Use golang/dep to manage vendor
This consolidates all vendor files into the top-level vendor directory so that they can be fully managed
by golang/dep.

The main benefit to this change is that it makes vendoring elastic/beats inside of custom
Beats easier because dep understands transitive dependencies.

The downside is that dep is experimental and will be replaced at some point by go modules. Several
bugs (or missing features) make this migration less than ideal.

- Symlinks in dependencies are not removed by pruning. golang/dep#1625
- Ran out of file handles on macOS. golang/dep#1044
- Can't vendor non-go directories. golang/dep#1306
  This is the most limiting feature from my point of view because we still have to
  manage build scripts (.py, Makefile, packaging templates) separatately and on our own.

Notes About the Changes

Skip notice file generation temporarily. It will need to be updated to read from Gopkg.lock in TOML format.

Pin github.com/docker/docker to use the 17.05.x branch. We have a patched version of this
branch with one extra commit that allows us to build on netbsd.

Pin github.com/rcrowly/go-metrics with a contraint because libbeat/monitoring will not
build with newer versions of go-metrics.

Pin github.com/docker/distribution to the version used by github.com/docker/docker
as specified in thier verndor data (they don't use dep).

This prunes non-go files with one exception for a project that uses non-standard license file
naming. So we had to keep non-go files in order to no lose the license files.

Update github.com/golang/protobuf

Update golang.org/x/net

There is a dev-tools/vendor directory containing non-go vendor files used in packaging.
Dep will not manage this project because it does not contain any Go files.

Packaging executes package_test.go after generating packages. And third-party projects
need to execute this test file from elastic/beats. But since dep will not vendor
the dependencies of this test file I added a work-around that adds unused imports to
non-test files so that they end up in the dependency graph.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Aug 23, 2018

Use golang/dep to manage vendor
This consolidates all vendor files into the top-level vendor directory so that they can be fully managed
by golang/dep.

The main benefit to this change is that it makes vendoring elastic/beats inside of custom
Beats easier because dep understands transitive dependencies.

The downside is that dep is experimental and will be replaced at some point by go modules. Several
bugs (or missing features) make this migration less than ideal.

- Symlinks in dependencies are not removed by pruning. golang/dep#1625
- Ran out of file handles on macOS. golang/dep#1044
- Can't vendor non-go directories. golang/dep#1306
  This is the most limiting feature from my point of view because we still have to
  manage build scripts (.py, Makefile, packaging templates) separatately and on our own.

== Notes About the Changes

Skip notice file generation temporarily. It will need to be updated to read from Gopkg.lock in TOML format.

Pin github.com/docker/docker to use the 17.05.x branch. We have a patched version of this
branch with one extra commit that allows us to build on netbsd.

Pin github.com/rcrowly/go-metrics with a contraint because libbeat/monitoring will not
build with newer versions of go-metrics.

Pin github.com/docker/distribution to the version used by github.com/docker/docker
as specified in thier verndor data (they don't use dep).

This prunes non-go files with one exception for a project that uses non-standard license file
naming. So we had to keep non-go files in order to no lose the license files.

Update github.com/golang/protobuf

Update golang.org/x/net

There is a dev-tools/vendor directory containing non-go vendor files used in packaging.
Dep will not manage this project because it does not contain any Go files.

Packaging executes package_test.go after generating packages. And third-party projects
need to execute this test file from elastic/beats. But since dep will not vendor
the dependencies of this test file I added a work-around that adds unused imports to
non-test files so that they end up in the dependency graph.

I added the 'mage syncTools' target to mimic what apm-server does by rsyncing the build
tools to their own directory.

Add script to copy build tools
@tico8

This comment has been minimized.

tico8 commented Dec 13, 2018

dependencies that do not have Go source code

My case was solved by following options.
I wanted to get github.com/gogo/protobuf(do not have Go source) into "vender/".

[prune]
  go-tests = true
  unused-packages = true
  non-go = true
  
  [[prune.project]]
    name = "github.com/gogo/protobuf"
    non-go = false
    unused-packages = false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment