support nested vendor/ dirs

[aybabtme]

The Go tool supports recursively looking up dependencies that may be vendored in ./vendor directories and their parents. It seems that dep doesn't support vendoring dependencies in nested dirs. Example:

src/do.co/monorepo/vendor
src/do.co/monorepo/large_project/with_many/specific_needs/vendor

In this example, monorepo/vendor contains global, generally needed dependencies that may have been vendored using another tool, while monorepo/large_project/with_many/specific_needs/vendor contains dependencies needed only by that project, which we want to use dep to manage.

This seems to currently not be possible. dep appears to be ignoring the dependencies vendored in monorepo/vendor and try to pull them again in monorepo/large_project/with_many/specific_needs/vendor.

[sdboyer]

hi! thanks for the issue.

yes, flattening vendor directories up to the single, topmost level is foundational to dep's design. nested vendor directories can cause numerous problems, and eliminating them was an intentional choice. changing that design is not something we can do.

dep is not primarily designed for use in monorepos right now. it is possible, but only for monorepo layouts with specific properties. i have a nascent plan about how we might make it useful for arbitrary monorepos, but i need a) time and b) folks with monorepos willing to collaborate with me on figuring it out 😉

in the meantime, though, i'm marking this for docs, as we should at least have an FAQ entry that details how dep can be used within a monorepo today.

[aybabtme]

@sdboyer perfect! My contrived example comes from the fact that we can't use dep right now to vendor for our whole monorepo, it appears not to work. But I thought, maybe I can introduce it at a smaller level.

I'll try to loop back with info about why dep doesn't work on our monorepo, if that's useful to you.

[sdboyer]

@aybabtme that'd be great! i have a few orgs i've talked to about their monorepos, but i still don't have anything written about where the real pain points are.

[aybabtme]

@sdboyer cool is there a specific doc/ticket where we should provide this info?

Our biggest hurdle so far has been speed: we have thousands of vendored dependencies and similarly many internal packages, this seems to cause problems to dep as it takes many minutes to run commands. Because of this, making dep work with our repo has been hard in itself, since progress and iteration time is very high.

Another thing is that our code layout is:

$ echo $GOPATH
~/cthulhu/docode
$ tree docode/src/ -L 2
docode/src/
└── do
    ├── doge
    ├── exp
    ├── services
    ├── teams
    ├── tools
    └── vendor

[aybabtme]

cc @fatih

[sdboyer]

is there a specific doc/ticket where we should provide this info?

this ticket's a fine place to give that feedback. you can also email me, if you'd rather keep it private.

Our biggest hurdle so far has been speed

yeah, we've been deferring on performance for a while, but we have a lot of headroom, and are making headway on the major things that will help - #121, #431. a month or two, maybe?

[sdboyer]

i'm gonna close this out, just to try to keep the issue queue tidy. when y'all have a chance to get to that feedback, please feel free to reopen 😄

[bhcleek]

@sdboyer I have some more specific feedback for you about this. If you'll re-open this issue, I can put it here. Otherwise, please let me know if you'd rather I just open a new issue.

[sdboyer]

@bhcleek new issue is best, i think.