Introduce the dependency vendoring tool: `dep`. #551

mandrigin · 2018-01-16T21:04:02Z

Use commits from go-ethereum@release/1.7 for most of the dependencies.

TODO

push 1.7.3 + dep branch to status-im/go-ethereum
use status-im/go-ethereum as a source in Gopkg.toml
~~create an issue to use dep prune when it is ready~~ (no need to since prune will be a part of ensure)
create an issue to create a CI check based on this comment → issue #556
rebase fix for make test-e2e when it is available

Closes #223

mandrigin · 2018-01-16T21:20:22Z

Split changes into 2 commits to simplify reviewing:

actual changes in the configs;
dependencies update.

adambabik · 2018-01-17T09:05:37Z

Have you created a Gopkg.toml in our go-ethereum fork already?

mandrigin · 2018-01-17T09:20:06Z

@adambabik I did it in mandrigin/go-ethereum. There is a TODO for moving that to the status-im fork.
I just wanted to synchronize the effort with @divan about cleaning up status-im/go-ethereum.

mandrigin · 2018-01-17T21:34:57Z

rebased to the current develop, so make test-e2e works again.

mandrigin · 2018-01-17T21:43:00Z

Double-checked difference between vendor directories in upstream go-ethereum@1.7 and this PR. In github.com there are no differences in source files. We just have a different set of packages here, but if the packages exists in both upstream and here the revision is the same.

mandrigin · 2018-01-18T19:34:26Z

@divan @adambabik I created develop branch in status-im/go-ethereum, pointed this PR to it.

mandrigin · 2018-01-18T19:37:10Z

Hmm, is it by design, to lint vendor? Don't we need to ignore that?

mandrigin · 2018-01-18T19:56:46Z

Resolved WIP status. I'll take a look at the linter problem later.

adambabik · 2018-01-19T11:06:29Z

Hmm, is it by design, to lint vendor? Don't we need to ignore that?

vendor/ is not linted (we specifically pass packages in lint.mk to gometalinter) but if there are files in linted packages that import some dependencies and types do not match, it can complain. In this case, it probably won't even compile.

mandrigin · 2018-01-19T11:20:30Z

@adambabik, hmm, all the tests pass on my machine, both e2e and unit. Is that possible if it doesn’t compile?

adambabik · 2018-01-19T16:30:46Z

Cache maybe in Travis? Have you checked that on Travis and your local machine there are the same files?

mandrigin · 2018-01-20T18:37:04Z

@adambabik so it was an incompatibility of 2 libraries btcd and btcutil, but in the packages that we never use ourselves. Found a compatible version of btcutil (btcd is used by Geth).

mandrigin · 2018-01-20T19:09:48Z

@adambabik all the checks are green now.

adambabik · 2018-01-21T12:34:59Z

@mandrigin awesome! Btw. these packages are used in extkeys which is used in geth/account.

adambabik · 2018-01-21T13:02:55Z

@mandrigin are these constraints https://github.com/status-im/go-ethereum/blob/develop/vendor/vendor.json included in any Gopkg.* file?

mandrigin · 2018-01-21T14:09:02Z

Yes, in the lock file.

mandrigin · 2018-01-22T15:05:40Z

@divan @adambabik I added a short doc[1] on how to use dep and what are the ideas behind it. Please, review and share your opinion there.

dshulyak · 2018-01-23T12:01:15Z

@mandrigin please do a push to status-go repository, it will run tests on public network

they are failing currently (#565) and it would be nice to see if this PR will affect them in any way

mandrigin · 2018-01-23T12:15:06Z

@dshulyak done! https://travis-ci.org/status-im/status-go/builds/332265950 should have a test on a public network.

dshulyak · 2018-01-23T13:00:04Z

thanks, looks like it doesn't, test hangs the same way :)

adambabik · 2018-01-24T01:24:51Z

@mandrigin can you please post instruction how I can test it from scratch? Can I just delete vendor/ from status-go and dep ensure should do the trick or I need to do something more?

dshulyak

works well, i have couple of questions/suggestions

I am seeing this warning, shouldn't we use override for transitive deps?

However, these projects are not direct dependencies of the current project:
they are not imported in any .go files, nor are they in the 'required' list in
Gopkg.toml. Dep only applies [[constraint]] rules to direct dependencies, so
these rules will have no effect.

Either import/require packages from these projects so that they become direct
dependencies, or convert each [[constraint]] to an [[override]] to enforce rules
on these projects, if they happen to be transitive dependencies,

Usually it is good practice to strip tests from vendor directory. Can we add find vendor -name '*_test.go' -delete until it is not support by dep ensure?

dshulyak · 2018-01-24T09:12:07Z

DEPENDENCIES.md

+   to be in-sync there. We want to reduce the regression scope. 
+   Hence, we pin down all the dependencies of `go-ethereum` with SHAs in `Gopkg.toml` when
+   importing a new version of upstream. (This is considered a bad practice for
+   `dep` but we are willing


something is missing here)

fixed in the latest push

dshulyak · 2018-01-24T09:21:34Z

DEPENDENCIES.md

+2. Commit changes.
+
+
+## Updating a dependency


i think it worth noting how to install dependencies from scratch, e.g dep ensure

Do we ever get a repo state when you need to install all deps from scratch? (aka with empty vendor)?

i guess you are right

fixed in the latest push

dshulyak · 2018-01-24T09:23:27Z

Gopkg.toml

@@ -0,0 +1,109 @@
+[[constraint]] # Required to be compatible with `btcd`


i would like to separate our deps and go-ethereum, what do you think about it?

I tried it, it doesn't seem to worth it.

If we have SHAs in the lock file (as we should) it isn't propagated to the main dependency. That makes the versions we use in status-go slightly different from what go-ethereum is using. That is exactly what we want to avoid.

If we have SHAs as constraints in go-ethereum/Gopkg.lock (via manual labour), we will have dependency resolution issues if a dependency is used both in status-go and go-ethereum (otto, for instance).

So the most stable method I found is to keep all the dependencies in Gopkg.toml of status-go.

i meant just to put our own deps on top of the file, then go-ethereum and then all of the go-ethereum deps. but maybe thats not worth it

Ah, no, that is a good idea, I'll do that.
One thing to note is that we won't have much of our dependencies in the toml file anyways.
The idea of this toml/lock split that the toml file contains only constrained dependencies. If we don't care about the version, we don't add this dep in there. So basically we will have:

btcutils (constrained because btcd)

go-ethereum (constrained because a custom source and a custom branch)

geth dependencies.

dep ensures that SHAs are in the lock file and the components aren't updated until you specifically ask for it.

fixed in the latest push

dshulyak · 2018-01-24T09:50:27Z

Also our vendor became 112M with tests, 86M without tests. And before that it was 32M. Do you know what might be the reason for such big change?

mandrigin · 2018-01-24T10:09:47Z

@dshulyak that's because dep ensure w/o dep prune keeps all the sub-packages of a dependency. For instance, if we only use github.com/foo/package/subpackage1, it will download the whole github.com/foo/package. dep prune is supposed to fix that, but it will also kill all C sources that we want to keep.

mandrigin · 2018-01-24T10:15:17Z

@dshulyak

I am seeing this warning, shouldn't we use override for transitive deps?

How to reproduce that? Is it with clean vendor?

dshulyak · 2018-01-24T10:21:19Z

@mandrigin at first i removed vendor and saw this warning, but after initial sync i can reproduce it as well

i suspect it might be related to dep version, i installed it in the morning)

dep:
 version     : devel
 build date  : 
 git hash    : 
 go version  : go1.9.2
 go compiler : gc
 platform    : linux/amd64

mandrigin · 2018-01-24T11:50:08Z

@dshulyak this warning... I can't reproduce it locally 😢

dshulyak · 2018-01-24T12:33:20Z

@mandrigin I just discovered that the code required for pruning unused packages and test files was merged and will be a part of next release. but in the documentation, we are recommending to use dep tool straight from GitHub, so maybe it makes sense to rely on those features and make our vendor folder smaller from the beginning. what do you think?

dshulyak · 2018-01-24T12:35:05Z

Even if we can only remove tests it will reduce the size of vendor by ~30M

mandrigin · 2018-01-24T12:45:22Z

@dshulyak I think it might make sense have a follow-up issue when the dep's functionality is released. Knowing which dependencies we use looks more important for me than the repo size.

dshulyak

sure, sounds good to me

dshulyak · 2018-01-24T12:47:36Z

regarding the warning, the result was identical anyway, so I assume we can figure out its meaning later

mandrigin · 2018-01-24T12:53:26Z

@dshulyak #569

Use commits from `go-ethereum@release/1.7` for most of the dependencies.

themue

The number of dependencies is alway horrifying.

LGTM

dshulyak · 2018-01-25T12:27:01Z

@mandrigin please rebase, i fixed mistake that blocked CI #572

dshulyak · 2018-01-25T12:43:36Z

good news, you don't have to :) restart of the job helped

mandrigin requested review from adambabik and divan January 16, 2018 21:04

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch from 107480d to 211bece Compare January 16, 2018 21:18

divan added the in progress label Jan 17, 2018

divan mentioned this pull request Jan 17, 2018

Use single codepath for sending transactions to a local and remote nodes #527

Merged

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch 2 times, most recently from 04ee720 to 5813f1a Compare January 17, 2018 21:33

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch 2 times, most recently from 823ee75 to 887468d Compare January 18, 2018 19:33

mandrigin changed the title ~~WIP: Introduce the dependency vendoring tool: dep.~~ Introduce the dependency vendoring tool: dep. Jan 18, 2018

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch 2 times, most recently from 76fe80b to d8c66d3 Compare January 20, 2018 18:36

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch from d8c66d3 to 5e93a5a Compare January 20, 2018 18:52

dshulyak suggested changes Jan 24, 2018

View reviewed changes

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch 2 times, most recently from fcf5ef7 to fe46ebb Compare January 24, 2018 11:49

dshulyak approved these changes Jan 24, 2018

View reviewed changes

dshulyak mentioned this pull request Jan 24, 2018

Remove unused packages and tests #570

Closed

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch 2 times, most recently from cbae543 to 4f748ce Compare January 25, 2018 10:53

mandrigin added 2 commits January 25, 2018 12:16

Introduce dep, the dependency vendoring tool.

db39fda

Use commits from `go-ethereum@release/1.7` for most of the dependencies.

Update dependencies.

9c50dbd

mandrigin force-pushed the igorm/investigate-vendoring-tool-223-dep-geth173 branch from 4f748ce to 9c50dbd Compare January 25, 2018 11:16

themue approved these changes Jan 25, 2018

View reviewed changes

mandrigin merged commit fb3d2ff into status-im:develop Jan 25, 2018

dshulyak mentioned this pull request Jan 25, 2018

Add an option to keep non-go files to dep #569

Closed

adambabik mentioned this pull request Jan 29, 2018

Update geth-patches readme with dep instruction and creating patches #578

Closed

		@@ -0,0 +1,109 @@
		[[constraint]] # Required to be compatible with `btcd`

Introduce the dependency vendoring tool: dep. #551

Introduce the dependency vendoring tool: dep. #551

Conversation

mandrigin commented Jan 16, 2018 • edited

TODO

mandrigin commented Jan 16, 2018

adambabik commented Jan 17, 2018

mandrigin commented Jan 17, 2018

mandrigin commented Jan 17, 2018

mandrigin commented Jan 17, 2018

mandrigin commented Jan 18, 2018

mandrigin commented Jan 18, 2018

mandrigin commented Jan 18, 2018

adambabik commented Jan 19, 2018

mandrigin commented Jan 19, 2018

adambabik commented Jan 19, 2018

mandrigin commented Jan 20, 2018

mandrigin commented Jan 20, 2018

adambabik commented Jan 21, 2018

adambabik commented Jan 21, 2018

mandrigin commented Jan 21, 2018

mandrigin commented Jan 22, 2018

dshulyak commented Jan 23, 2018

mandrigin commented Jan 23, 2018

dshulyak commented Jan 23, 2018

adambabik commented Jan 24, 2018

dshulyak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dshulyak Jan 24, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dshulyak commented Jan 24, 2018

mandrigin commented Jan 24, 2018 • edited

mandrigin commented Jan 24, 2018 • edited

dshulyak commented Jan 24, 2018

mandrigin commented Jan 24, 2018

dshulyak commented Jan 24, 2018 • edited

dshulyak commented Jan 24, 2018

mandrigin commented Jan 24, 2018

dshulyak left a comment

Choose a reason for hiding this comment

dshulyak commented Jan 24, 2018

mandrigin commented Jan 24, 2018

themue left a comment

Choose a reason for hiding this comment

dshulyak commented Jan 25, 2018

dshulyak commented Jan 25, 2018

Introduce the dependency vendoring tool: `dep`. #551

Introduce the dependency vendoring tool: `dep`. #551

mandrigin commented Jan 16, 2018 •

edited

dshulyak Jan 24, 2018 •

edited

mandrigin commented Jan 24, 2018 •

edited

mandrigin commented Jan 24, 2018 •

edited

dshulyak commented Jan 24, 2018 •

edited