Update a few dependencies #69976

ash2k · 2018-10-18T13:10:01Z

What type of PR is this?
/kind cleanup
/sig api-machinery

What this PR does / why we need it:
I'm trying to use the exact same versions of libraries that Kubernetes uses and it turned out that a few of the dependencies are significantly older than the versions I use atm. I discovered that because a few of my tests started to fail because of "regression" - bugs were fixed in newer versions but not in the versions kubernetes uses. So I decided to update the kubernetes versions.

These are the dependencies with the issues that are resolved for me in newer versions:
github.com/go-openapi/* <- new fields in some structs were added which I need
github.com/asaskevich/govalidator <- supports stricter validation of URLs
~~github.com/ghodss/yaml <- bug with pointer struct unmarshalling~~

Does this PR introduce a user-facing change?:

NONE

fedebongio · 2018-10-18T20:53:38Z

/assign @lavalamp

ash2k · 2018-10-18T20:56:30Z

/retest

ash2k · 2018-10-18T20:57:50Z

/assign sttts

lavalamp · 2018-10-18T21:05:37Z

I thought the yaml package got updated recently? Maybe that's still in flight?

I'm not convinced by this justification for the other two updates. And why is mgo getting pulled in now?

ash2k · 2018-10-19T02:10:30Z

I thought the yaml package got updated recently? Maybe that's still in flight?

#67594 This PR is not updating godeps.json though. And that is probably the reason builds are failing there. I was not aware it is there.

I'm not convinced by this justification for the other two updates.

What is the usual justification to get a dependency updated? (frankly, I was not expecting this PR to face any "resistance"...). Is that process documented somewhere?

Looks like github.com/go-openapi versions are from the middle of 2016. I could imagine there've been some bugfixes since then. If it was perfect, no changes would have been made.

github.com/asaskevich/govalidator - Jul 2016 version. There've been 170 commits since then.

And why is mgo getting pulled in now?

It is a dependency of github.com/go-openapi/strfmt. I was following the instructions to get to this state.

What worries you re. this PR? Potential regressions or something else too?

ash2k · 2018-10-23T11:28:31Z

fyi @dims

dims · 2018-10-23T12:03:56Z

@ash2k is the list of the packages in the commit message correct? here's what i see in the patch and i don't see ghodss/yaml

10 +++ b/vendor/github.com/asaskevich
22 +++ b/vendor/github.com/globalsign
133 +++ b/vendor/github.com/go-openapi

dims · 2018-10-23T12:04:27Z

related to #69776

dims · 2018-10-23T12:08:12Z

@ash2k the main concern would be "what was working before but will not work anymore", just identifying the possible scenarios, so we can figure out if we can tolerate it or need to document it or work around it.

nikhita · 2018-10-23T12:14:50Z

the main concern would be "what was working before but will not work anymore",

I think go-openapi added some changes that could be concerning for custom resource validation, but I'll double-check to be sure.

Having said that, there are a few problems with the current vendored go-openapi that I know of. I'll be sending a fix upstream for those problems, and maybe we could update the vendored go-openapi then (or I'll do a bump again later if it takes time merging the upstream PR).

ash2k · 2018-10-23T12:34:32Z

@dims

is the list of the packages in the commit message correct? here's what i see in the patch and i don't see ghodss/yaml

Turns out #69627 updated ghodss/yaml to the latest commit. I missed this fact because I was using release-1.12 branch as the latest released one and assumed that master has the same old version too. I'll amend the commit message and the issue description to avoid confusion.

the main concern would be "what was working before but will not work anymore", just identifying the possible scenarios, so we can figure out if we can tolerate it or need to document it or work around it.

I'm not sure how to proceed with this PR 🤷‍♂️ If all tests pass then... how else can we ensure it is all good?

dims · 2018-10-23T12:46:12Z

@ash2k great question :)

How about enumerating what you saw? few of my tests started to fail because of "regression" - bugs were fixed in newer versions but not in the versions kubernetes uses any details on the test and/or the PR(s) that "probably" fixed the problem

Thanks,
Dims

lavalamp · 2018-10-23T18:15:00Z

Godeps/Godeps.json

@@ -1,6 +1,6 @@
 {
 	"ImportPath": "k8s.io/kubernetes",
-	"GoVersion": "go1.10",
+	"GoVersion": "go1.11",


I think it's ok to bump this version now, but maybe @ixdy can confirm.

I think this field is automatically filled in by godep based on what version of the runtime it's using. it's pretty meaningless.

lavalamp · 2018-10-23T18:15:51Z

Godeps/Godeps.json

@@ -1351,132 +1351,132 @@
 		},
 		{
 			"ImportPath": "github.com/docker/docker/api",
-			"Comment": "docs-v1.12.0-rc4-2016-07-15-9510-ga9fbbdc8d",
+			"Comment": "docs-v1.12.0-rc4-2016-07-15-9510-ga9fbbdc8dd",


Also @ixdy do you know why the extra d is getting appended here? I have a vendor change that is doing the same thing and I don't know why.

I have no idea how godep decides how to truncate the git commit. it might be a git setting, it might be something else. godep is terrible.

I think it's just using git describe --tags.

The manpage for git describe notes

Note that the suffix you get if you type these commands today may be longer than what Linus saw above when he ran these commands, as your Git repository may have new commits whose object names begin with 975b that did not exist back then, and "-g975b" suffix alone may not be sufficient to disambiguate these commits.

so I guess this is WAI? 🤷‍♂️

I remember discussing about this with @cblecker somewhere but can't find the issue now... 🤷‍♀️

it has to do with the git version you're running locally. at a certain point, the default changed for git describe

see tools/godep#560 for more details. This PR to fix it was opened, but that happened after godep stopped active development.

+1 to not include the changes that update the comment with just one more character or one less character for the SHA

We could also just scrub all comments from the file. That's what we do with the staging-godeps script

We probably should. This is getting stupid.

+1 to scrub all comments from the file

submit a PR to address this: #70687

lavalamp · 2018-10-23T18:19:57Z

To be clear, my concern is just that we don't unnecessarily expand the dependency list. I'm not super happy that we're importing mgo/bson after this change. I'm not sure there's anything to be done about it :( But every new dependency makes things a tiny bit worse for every consumer of kubernetes.

Regressions should be caught by tests, I'm not worried about that.

ash2k · 2018-10-23T23:51:02Z

@dims

How about enumerating what you saw?

We use go-swagger to generate a client for one of our services. It generates code that uses github.com/go-openapi/runtime. Here is the diff of one of the files the generated code depends on. It needs the fields that are not present in the older version kubernetes uses. Here is the diff kube version vs wanted version:

diff --git a/vendor/github.com/go-openapi/runtime/client_operation.go b/vendor/github.com/go-openapi/runtime/client_operation.go
index fa21eacf3..421f726c5 100644
--- a/vendor/github.com/go-openapi/runtime/client_operation.go
+++ b/vendor/github.com/go-openapi/runtime/client_operation.go
@@ -14,11 +14,6 @@
 
 package runtime
 
-import (
-	"context"
-	"net/http"
-)
-
 // ClientOperation represents the context for a swagger operation to be submitted to the transport
 type ClientOperation struct {
 	ID                 string
@@ -30,8 +25,6 @@ type ClientOperation struct {
 	AuthInfo           ClientAuthInfoWriter
 	Params             ClientRequestWriter
 	Reader             ClientResponseReader
-	Context            context.Context
-	Client             *http.Client
 }
 
 // A ClientTransport implementor knows how to submit Request objects to some destination

Regarding github.com/asaskevich/govalidator - my tests with current kube version of this dependency are failing with:

--- FAIL: TestHTTPSAddressValid (0.02s)
    --- FAIL: TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.b-c.de' (0.00s)
        <autogenerated>:1: 
                Error Trace:    validation_test.go:172
                Error:          Received unexpected error:
                                Key: 'WubbaLubbaDubDub.Endpoint' Error:Field validation for 'Endpoint' failed on the 'aws_sns_topic_endpoint_url' tag
                Test:           TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.b-c.de'
    --- FAIL: TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.b--c.de/' (0.00s)
        <autogenerated>:1: 
                Error Trace:    validation_test.go:172
                Error:          Received unexpected error:
                                Key: 'WubbaLubbaDubDub.Endpoint' Error:Field validation for 'Endpoint' failed on the 'aws_sns_topic_endpoint_url' tag
                Test:           TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.b--c.de/'
    --- FAIL: TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.ba--c.de/' (0.00s)
        <autogenerated>:1: 
                Error Trace:    validation_test.go:172
                Error:          Received unexpected error:
                                Key: 'WubbaLubbaDubDub.Endpoint' Error:Field validation for 'Endpoint' failed on the 'aws_sns_topic_endpoint_url' tag
                Test:           TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.ba--c.de/'
    --- FAIL: TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.xn--c.de/' (0.00s)
        <autogenerated>:1: 
                Error Trace:    validation_test.go:172
                Error:          Received unexpected error:
                                Key: 'WubbaLubbaDubDub.Endpoint' Error:Field validation for 'Endpoint' failed on the 'aws_sns_topic_endpoint_url' tag
                Test:           TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'https://a.xn--c.de/'
    --- FAIL: TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'http://www.atlassian.com./' (0.00s)
        <autogenerated>:1: 
                Error Trace:    validation_test.go:172
                Error:          Received unexpected error:
                                Key: 'WubbaLubbaDubDub.Endpoint' Error:Field validation for 'Endpoint' failed on the 'aws_sns_topic_endpoint_url' tag
                Test:           TestHTTPSAddressValid/enusre_aws_sns_topic_endpoint_url_succeeds_on_'http://www.atlassian.com./'
FAIL

Everything is fine if I use new version. There is a whole bunch of url validation issues that have been fixed (probably not an exhaustive list) that I found: asaskevich/govalidator#193 asaskevich/govalidator#150 asaskevich/govalidator#151 asaskevich/govalidator#194 asaskevich/govalidator#269
One of them is probably the one that is causing the regression.

I've been thinking about this issue in general. Basically, here is a user story that kubernetes community probably needs to have an answer for:
As a developer, I'd like to use client-go/apimachinery/etc libraries to work with kubernetes. The recommended approach is to use dependency versions from Godeps.json so that I get a tested and verified set of dependencies that are guaranteed to work together. That makes sense. I also use one of the dependencies directly and I need a newer version to pick up a bugfix for an issue that is affecting me. What am I supposed to do in this situation?

dims · 2018-10-24T00:23:26Z

go-swagger update is for supporting code generated using latest version of the tool
govalidator update is tightening the validation for urls
mgo is pulled in because of go-openapi/strfmt indirectly

/lgtm

cblecker · 2018-10-24T00:36:08Z

/hold
Needs license review for new dependency in the tree. Looks like 2-clause BSD (https://opensource.org/licenses/BSD-2-Clause)

/assign @thockin

ash2k · 2018-10-24T01:24:53Z

(citing myself)

The recommended approach is to use dependency versions from Godeps.json so that I get a tested and verified set of dependencies that are guaranteed to work together. That makes sense.

BTW, forgot to mention - I've built a tool to generate Gopkg.toml for dep from Godeps.json that some people may find useful https://github.com/ash2k/kubegodep2dep

nikhita · 2018-10-24T06:03:28Z

Needs license review for new dependency in the tree. Looks like 2-clause BSD

fyi @justaugustus @swinslow

dims · 2018-10-24T17:00:29Z

@thockin @justaugustus @nikhita @cblecker - fwiw, we do have 2 packages with the same license already. So at least it's not a new license.

vendor/github.com/pkg/sftp/LICENSE
vendor/github.com/PuerkitoBio/purell/LICENSE

nikhita · 2018-10-26T12:21:13Z

I think go-openapi added some changes that could be concerning for custom resource validation, but I'll double-check to be sure.

Ok from the go-openapi side. 👍

lavalamp · 2018-10-31T17:49:39Z

I think I'm satisfied that this is useful. I'll approve if it is rebased. Sorry for taking a while. I think it's not adding new license types.

github.com/go-openapi/* github.com/asaskevich/govalidator

ash2k · 2018-11-01T09:01:24Z

@lavalamp @dims rebased and ready for review. Thanks.

dims · 2018-11-01T13:25:51Z

/lgtm

lavalamp · 2018-11-01T17:04:42Z

/approve

thockin · 2018-11-01T17:20:54Z

/lgtm
/approve

k8s-ci-robot · 2018-11-01T17:21:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ash2k, lavalamp, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~Godeps/OWNERS~~ [lavalamp,thockin]
~~staging/src/k8s.io/apiextensions-apiserver/Godeps/OWNERS~~ [lavalamp,thockin]
~~staging/src/k8s.io/apiserver/Godeps/OWNERS~~ [lavalamp,thockin]
~~staging/src/k8s.io/code-generator/Godeps/OWNERS~~ [lavalamp,thockin]
~~staging/src/k8s.io/kube-aggregator/Godeps/OWNERS~~ [lavalamp,thockin]
~~staging/src/k8s.io/sample-apiserver/Godeps/OWNERS~~ [lavalamp,thockin]
~~vendor/OWNERS~~ [lavalamp,thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ash2k · 2018-11-01T23:22:08Z

@cblecker mind canceling the hold?

cblecker · 2018-11-01T23:54:03Z

/hold cancel
👍

k8s-ci-robot requested review from cheftako and deads2k October 18, 2018 13:10

k8s-ci-robot assigned lavalamp Oct 18, 2018

k8s-ci-robot assigned sttts Oct 18, 2018

dims mentioned this pull request Oct 23, 2018

Updating vendored dependencies for Kubernetes 1.13 #69776

Closed

89 tasks

ash2k force-pushed the update-few-dependencies branch from 6bb1822 to 60b371d Compare October 23, 2018 12:38

lavalamp reviewed Oct 23, 2018

View reviewed changes

k8s-ci-robot assigned dims Oct 24, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 24, 2018

k8s-ci-robot assigned thockin Oct 24, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 24, 2018

cofyc mentioned this pull request Oct 24, 2018

Vendor github.com/prometheus/client_golang/prometheus/testutil package #65632

Merged

Update a few dependencies

8763223

github.com/go-openapi/* github.com/asaskevich/govalidator

ash2k force-pushed the update-few-dependencies branch from 60b371d to 8763223 Compare October 31, 2018 23:00

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 31, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 1, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 1, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 1, 2018

k8s-ci-robot merged commit a831ea2 into kubernetes:master Nov 2, 2018

ash2k deleted the update-few-dependencies branch November 2, 2018 07:32

This was referenced Nov 6, 2018

Prevent Comment field conflicts in Godeps/Godeps.json #70686

Closed

Scrub comment from Godeps.json #70687

Closed

Update a few dependencies #69976

Update a few dependencies #69976

Conversation

ash2k commented Oct 18, 2018 • edited Loading

fedebongio commented Oct 18, 2018

ash2k commented Oct 18, 2018

ash2k commented Oct 18, 2018

lavalamp commented Oct 18, 2018

ash2k commented Oct 19, 2018

ash2k commented Oct 23, 2018

dims commented Oct 23, 2018

dims commented Oct 23, 2018

dims commented Oct 23, 2018

nikhita commented Oct 23, 2018

ash2k commented Oct 23, 2018

dims commented Oct 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lavalamp commented Oct 23, 2018

ash2k commented Oct 23, 2018

dims commented Oct 24, 2018

cblecker commented Oct 24, 2018

ash2k commented Oct 24, 2018

nikhita commented Oct 24, 2018

dims commented Oct 24, 2018

nikhita commented Oct 26, 2018

lavalamp commented Oct 31, 2018

ash2k commented Nov 1, 2018

dims commented Nov 1, 2018

lavalamp commented Nov 1, 2018

thockin commented Nov 1, 2018

k8s-ci-robot commented Nov 1, 2018

ash2k commented Nov 1, 2018

cblecker commented Nov 1, 2018

ash2k commented Oct 18, 2018 •

edited

Loading