New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⛵️ Move k8s.io/test-infra over to dep #5301

Merged
merged 4 commits into from Nov 2, 2017

Conversation

@cblecker
Member

cblecker commented Nov 1, 2017

I have no idea how this will go, but let's try it!
This actually seems like it could work!

What I did:

dep init -no-examples
rm -rf Godeps/ _vendor-* vendor/
# Edit Gopkg.toml to ignore github.com/aws/aws-sdk-go* and remove it as a contraint
dep ensure && dep prune
verify/update-bazel.sh

@BenTheElder @stevekuznetsov @fejta @Q-Lee

@cjwagner

This comment has been minimized.

Show comment
Hide comment
@cjwagner

cjwagner Nov 1, 2017

Member

/retest

Member

cjwagner commented Nov 1, 2017

/retest

@stevekuznetsov

This comment has been minimized.

Show comment
Hide comment
@stevekuznetsov

stevekuznetsov Nov 1, 2017

Contributor

I wouldn't necessarily have expected the BUILD files to change or for us to pull in 200k more LoC -- is that a misinterpretation of the current vendor state by dep? Are we pulling in sub-packages we don't need?

Contributor

stevekuznetsov commented Nov 1, 2017

I wouldn't necessarily have expected the BUILD files to change or for us to pull in 200k more LoC -- is that a misinterpretation of the current vendor state by dep? Are we pulling in sub-packages we don't need?

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 1, 2017

Member

Doing this without the prune is even worse:

+6,512,203 −19,285
Member

cblecker commented Nov 1, 2017

Doing this without the prune is even worse:

+6,512,203 −19,285
@spxtr

This comment has been minimized.

Show comment
Hide comment
@spxtr

spxtr Nov 1, 2017

Member

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

Member

spxtr commented Nov 1, 2017

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

@BenTheElder

This comment has been minimized.

Show comment
Hide comment
@BenTheElder

BenTheElder Nov 1, 2017

Member

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md#ignored

Member

BenTheElder commented Nov 1, 2017

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md#ignored

Show outdated Hide outdated BUILD Outdated

@cblecker cblecker changed the title from [WIP] Move k8s.io/test-infra over to dep to ⛵️ Move k8s.io/test-infra over to dep Nov 2, 2017

@spxtr

This comment has been minimized.

Show comment
Hide comment
@spxtr

spxtr Nov 2, 2017

Member

Probably we should add some basic instructions to the readme.

Member

spxtr commented Nov 2, 2017

Probably we should add some basic instructions to the readme.

cblecker added some commits Nov 2, 2017

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 2, 2017

Member

Okay. So this works as-is. I was tinkering with also trying to break the dependency on k/k, as the imports we use are all ones that have been split off into the apimachinery/apiserver repos. However there was some bazel funkiness with that, so it'll have to wait for another PR. This one is big enough!

@kubernetes/test-infra-maintainers PTAL :)

Member

cblecker commented Nov 2, 2017

Okay. So this works as-is. I was tinkering with also trying to break the dependency on k/k, as the imports we use are all ones that have been split off into the apimachinery/apiserver repos. However there was some bazel funkiness with that, so it'll have to wait for another PR. This one is big enough!

@kubernetes/test-infra-maintainers PTAL :)

@BenTheElder

This comment has been minimized.

Show comment
Hide comment
@BenTheElder

BenTheElder Nov 2, 2017

Member

LGTM, thanks for all your work on this! Hopefully this will move us to dep sanity :-)
/lgtm
/hold
This PR is big enough that I'd love someone else to /lgtm and /hold cancel after reviewing it themseleves @cjwagner @kargakis.

Member

BenTheElder commented Nov 2, 2017

LGTM, thanks for all your work on this! Hopefully this will move us to dep sanity :-)
/lgtm
/hold
This PR is big enough that I'd love someone else to /lgtm and /hold cancel after reviewing it themseleves @cjwagner @kargakis.

@krzyzacy

This comment has been minimized.

Show comment
Hide comment
@krzyzacy

krzyzacy Nov 2, 2017

Member

/cc @ixdy @rmmh
LGTM, and try to get more eyes on this

Member

krzyzacy commented Nov 2, 2017

/cc @ixdy @rmmh
LGTM, and try to get more eyes on this

@k8s-ci-robot k8s-ci-robot requested review from ixdy and rmmh Nov 2, 2017

@kargakis

This comment has been minimized.

Show comment
Hide comment
@kargakis

kargakis Nov 2, 2017

Member

It seems that most of the additions are test code, BUILD, or other documentation files. There is also some interesting cleanup.

/lgtm

Member

kargakis commented Nov 2, 2017

It seems that most of the additions are test code, BUILD, or other documentation files. There is also some interesting cleanup.

/lgtm

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Nov 2, 2017

Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, cblecker, kargakis

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

Contributor

k8s-ci-robot commented Nov 2, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, cblecker, kargakis

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@stevekuznetsov

This comment has been minimized.

Show comment
Hide comment
@stevekuznetsov

stevekuznetsov Nov 2, 2017

Contributor

100k LOC diff is still weird -- looks like we were not vendoring tests in the past, but now we are? Is that ok? Do we want them?

Contributor

stevekuznetsov commented Nov 2, 2017

100k LOC diff is still weird -- looks like we were not vendoring tests in the past, but now we are? Is that ok? Do we want them?

@@ -0,0 +1,26 @@
# kubernetes/test-infra dependency management

This comment has been minimized.

@stevekuznetsov

stevekuznetsov Nov 2, 2017

Contributor

Can we link to this from the main README so contributors can find it without digging too much?

@stevekuznetsov

stevekuznetsov Nov 2, 2017

Contributor

Can we link to this from the main README so contributors can find it without digging too much?

This comment has been minimized.

@cblecker

cblecker Nov 2, 2017

Member

Done

@cblecker

cblecker Nov 2, 2017

Member

Done

@stevekuznetsov

This comment has been minimized.

Show comment
Hide comment
@stevekuznetsov

stevekuznetsov Nov 2, 2017

Contributor

Of the additions, the majority are tests:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | wc -l
432
$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -E '_test.go$' | wc -l
281

Then we also pull in a lot of new non-code files:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | xargs -L 1 basename | sort | uniq -c | sort -rn | head -n 15
     18 README.md
     16 .gitignore
     12 CONTRIBUTING.md
      6 .travis.yml
      6 BUILD
      6 AUTHORS
      5 CONTRIBUTORS
      4 OWNERS
      4 Makefile
      4 .gitattributes
      4 codereview.cfg
      3 README
      3 CHANGELOG.md
      3 AUTHORS.md
      2 util.go

So updated vendored code is small:

$ git diff --diff-filter=AM --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | grep -E '.go$' | xargs wc -l
   829 vendor/github.com/influxdata/influxdb/client/influxdb.go
   501 vendor/github.com/influxdata/influxdb/client/v2/client.go
    44 vendor/github.com/influxdata/influxdb/errors.go
     1 vendor/github.com/influxdata/influxdb/influxdb.go
    27 vendor/github.com/influxdata/influxdb/models/inline_fnv.go
    38 vendor/github.com/influxdata/influxdb/models/inline_strconv_parse.go
  1916 vendor/github.com/influxdata/influxdb/models/points.go
   116 vendor/github.com/influxdata/influxdb/node.go
    95 vendor/github.com/influxdata/influxdb/pkg/escape/bytes.go
    39 vendor/github.com/petar/GoLLRB/llrb/avgvar.go
    93 vendor/github.com/petar/GoLLRB/llrb/iterator.go
    46 vendor/github.com/petar/GoLLRB/llrb/llrb-stats.go
   456 vendor/github.com/petar/GoLLRB/llrb/llrb.go
    17 vendor/github.com/petar/GoLLRB/llrb/util.go
    13 vendor/github.com/shurcooL/githubql/doc.go
    11 vendor/github.com/shurcooL/graphql/doc.go
    23 vendor/golang.org/x/crypto/curve25519/doc.go
    21 vendor/golang.org/x/crypto/ssh/doc.go
   156 vendor/golang.org/x/net/context/context.go
   341 vendor/golang.org/x/oauth2/oauth2.go
    76 vendor/golang.org/x/sys/unix/syscall.go
   100 vendor/golang.org/x/tools/go/gcexportdata/gcexportdata.go
  1041 vendor/golang.org/x/tools/go/gcimporter15/gcimporter.go
   112 vendor/google.golang.org/appengine/appengine.go
   355 vendor/google.golang.org/appengine/internal/urlfetch/urlfetch_service.pb.go
   210 vendor/google.golang.org/appengine/urlfetch/urlfetch.go
    20 vendor/k8s.io/kubernetes/pkg/util/doc.go
    56 vendor/k8s.io/kubernetes/pkg/util/string_flag.go
    48 vendor/k8s.io/kubernetes/pkg/util/template.go
    72 vendor/k8s.io/kubernetes/pkg/util/trace.go
    27 vendor/k8s.io/kubernetes/pkg/util/umask_windows.go
   147 vendor/k8s.io/kubernetes/pkg/util/util.go
  7047 total

So that actually makes me feel like this is a lot smaller of a move than it seems. Seems like we are updating those dependencies?

Contributor

stevekuznetsov commented Nov 2, 2017

Of the additions, the majority are tests:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | wc -l
432
$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -E '_test.go$' | wc -l
281

Then we also pull in a lot of new non-code files:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | xargs -L 1 basename | sort | uniq -c | sort -rn | head -n 15
     18 README.md
     16 .gitignore
     12 CONTRIBUTING.md
      6 .travis.yml
      6 BUILD
      6 AUTHORS
      5 CONTRIBUTORS
      4 OWNERS
      4 Makefile
      4 .gitattributes
      4 codereview.cfg
      3 README
      3 CHANGELOG.md
      3 AUTHORS.md
      2 util.go

So updated vendored code is small:

$ git diff --diff-filter=AM --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | grep -E '.go$' | xargs wc -l
   829 vendor/github.com/influxdata/influxdb/client/influxdb.go
   501 vendor/github.com/influxdata/influxdb/client/v2/client.go
    44 vendor/github.com/influxdata/influxdb/errors.go
     1 vendor/github.com/influxdata/influxdb/influxdb.go
    27 vendor/github.com/influxdata/influxdb/models/inline_fnv.go
    38 vendor/github.com/influxdata/influxdb/models/inline_strconv_parse.go
  1916 vendor/github.com/influxdata/influxdb/models/points.go
   116 vendor/github.com/influxdata/influxdb/node.go
    95 vendor/github.com/influxdata/influxdb/pkg/escape/bytes.go
    39 vendor/github.com/petar/GoLLRB/llrb/avgvar.go
    93 vendor/github.com/petar/GoLLRB/llrb/iterator.go
    46 vendor/github.com/petar/GoLLRB/llrb/llrb-stats.go
   456 vendor/github.com/petar/GoLLRB/llrb/llrb.go
    17 vendor/github.com/petar/GoLLRB/llrb/util.go
    13 vendor/github.com/shurcooL/githubql/doc.go
    11 vendor/github.com/shurcooL/graphql/doc.go
    23 vendor/golang.org/x/crypto/curve25519/doc.go
    21 vendor/golang.org/x/crypto/ssh/doc.go
   156 vendor/golang.org/x/net/context/context.go
   341 vendor/golang.org/x/oauth2/oauth2.go
    76 vendor/golang.org/x/sys/unix/syscall.go
   100 vendor/golang.org/x/tools/go/gcexportdata/gcexportdata.go
  1041 vendor/golang.org/x/tools/go/gcimporter15/gcimporter.go
   112 vendor/google.golang.org/appengine/appengine.go
   355 vendor/google.golang.org/appengine/internal/urlfetch/urlfetch_service.pb.go
   210 vendor/google.golang.org/appengine/urlfetch/urlfetch.go
    20 vendor/k8s.io/kubernetes/pkg/util/doc.go
    56 vendor/k8s.io/kubernetes/pkg/util/string_flag.go
    48 vendor/k8s.io/kubernetes/pkg/util/template.go
    72 vendor/k8s.io/kubernetes/pkg/util/trace.go
    27 vendor/k8s.io/kubernetes/pkg/util/umask_windows.go
   147 vendor/k8s.io/kubernetes/pkg/util/util.go
  7047 total

So that actually makes me feel like this is a lot smaller of a move than it seems. Seems like we are updating those dependencies?

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 2, 2017

Member

@stevekuznetsov dep and godep function differently. When godep saves things to vendor/, it only copies the *.go code itself, and the LICENSE files (omits tests and non-go files). dep on the other hand, checks out the whole git repo to vendor. dep prune then prunes back to only the packages you're actually importing and using + LICENSE files.. but it leaves the whole directory (including non-go files and tests).

Member

cblecker commented Nov 2, 2017

@stevekuznetsov dep and godep function differently. When godep saves things to vendor/, it only copies the *.go code itself, and the LICENSE files (omits tests and non-go files). dep on the other hand, checks out the whole git repo to vendor. dep prune then prunes back to only the packages you're actually importing and using + LICENSE files.. but it leaves the whole directory (including non-go files and tests).

@cblecker

This comment has been minimized.

Show comment
Hide comment
@cblecker

cblecker Nov 2, 2017

Member

/hold cancel
:shipit:

Member

cblecker commented Nov 2, 2017

/hold cancel
:shipit:

@stevekuznetsov

This comment has been minimized.

Show comment
Hide comment
@stevekuznetsov

stevekuznetsov Nov 2, 2017

Contributor

Right -- the majority of this is due to test and non-code changes, but that is still a couple transitive version bumps in the dependencies from my quick search. Should be fine but did want to point it out.

Contributor

stevekuznetsov commented Nov 2, 2017

Right -- the majority of this is due to test and non-code changes, but that is still a couple transitive version bumps in the dependencies from my quick search. Should be fine but did want to point it out.

@k8s-ci-robot k8s-ci-robot merged commit c258100 into kubernetes:master Nov 2, 2017

7 checks passed

cla/linuxfoundation cblecker authorized
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
pull-test-infra-bazel Job succeeded.
Details
pull-test-infra-gubernator Skipped
pull-test-infra-verify-bazel Job succeeded.
Details
pull-test-infra-verify-gofmt Job succeeded.
Details
pull-test-infra-verify-govet Job succeeded.
Details

@cblecker cblecker deleted the cblecker:dep branch Nov 2, 2017

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Nov 17, 2017

also just noting over here that our prune logic is going to become a) automated b) more powerful and c) more granular pretty soon: golang/dep#944

sdboyer commented Nov 17, 2017

also just noting over here that our prune logic is going to become a) automated b) more powerful and c) more granular pretty soon: golang/dep#944

@stevekuznetsov

This comment has been minimized.

Show comment
Hide comment
@stevekuznetsov

stevekuznetsov Nov 17, 2017

Contributor

@sdboyer that sounds awesome!

Contributor

stevekuznetsov commented Nov 17, 2017

@sdboyer that sounds awesome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment