Bug 1781707: openshift-sdn, CNO: handle new kubeconfig path #420

squeed · 2019-12-10T17:27:49Z

We pull the apiserver url from the kubelet's kubeconfig. This has changed, and can be found in one of two places. Check them both.

Ultimately we need to generate this in a more sustainable way, but this is a good workaround (and easily backportable).

openshift-ci-robot · 2019-12-10T17:27:55Z

@squeed: An error was encountered adding this pull request to the external tracker bugs for bug 1781707 on the Bugzilla server at https://bugzilla.redhat.com:

JSONRPC error 100500: Insecure dependency in parameter 3 of DBI::db=HASH(0x55b6fb1ab018)->do method call while running with -T switch at /loader/0x55b6f4cee870/Bugzilla/Extension/ExternalBugs/Bug.pm line 327.
at /loader/0x55b6f4cee870/Bugzilla/Extension/ExternalBugs/Bug.pm line 327.
Bugzilla::Extension::ExternalBugs::Bug::update_ext_info('Bugzilla::Extension::ExternalBugs::Bug=HASH(0x55b6fb421540)', 1) called at /loader/0x55b6f4cee870/Bugzilla/Extension/ExternalBugs/Bug.pm line 121
Bugzilla::Extension::ExternalBugs::Bug::create('Bugzilla::Extension::ExternalBugs::Bug', 'HASH(0x55b6fb3f3568)') called at /var/www/html/bugzilla/extensions/ExternalBugs/Extension.pm line 858
Bugzilla::Extension::ExternalBugs::bug_start_of_update('Bugzilla::Extension::ExternalBugs=HASH(0x55b6fb207600)', 'HASH(0x55b6fb3a1400)') called at /var/www/html/bugzilla/Bugzilla/Hook.pm line 21
Bugzilla::Hook::process('bug_start_of_update', 'HASH(0x55b6fb3a1400)') called at /var/www/html/bugzilla/Bugzilla/Bug.pm line 1170
Bugzilla::Bug::update('Bugzilla::Bug=HASH(0x55b6fb3a0058)') called at /loader/0x55b6f4cee870/Bugzilla/Extension/ExternalBugs/WebService.pm line 80
Bugzilla::Extension::ExternalBugs::WebService::add_external_bug('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55b6fb0ad910)') called at (eval 1985) line 1
eval ' $procedure->{code}->($self, @params)
;' called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 220
JSON::RPC::Legacy::Server::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55b6fb0c14f0)') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 295
Bugzilla::WebService::Server::JSONRPC::_handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...', 'HASH(0x55b6fb0c14f0)') called at /usr/share/perl5/vendor_perl/JSON/RPC/Legacy/Server.pm line 126
JSON::RPC::Legacy::Server::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/Bugzilla/WebService/Server/JSONRPC.pm line 70
Bugzilla::WebService::Server::JSONRPC::handle('Bugzilla::WebService::Server::JSONRPC::Bugzilla::Extension::E...') called at /var/www/html/bugzilla/jsonrpc.cgi line 31
ModPerl::ROOT::Bugzilla::ModPerl::ResponseHandler::var_www_html_bugzilla_jsonrpc_2ecgi::handler('Apache2::RequestRec=SCALAR(0x55b6fb23c358)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
eval {...} called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 207
ModPerl::RegistryCooker::run('Bugzilla::ModPerl::ResponseHandler=HASH(0x55b6fb21d800)') called at /usr/lib64/perl5/vendor_perl/ModPerl/RegistryCooker.pm line 173
ModPerl::RegistryCooker::default_handler('Bugzilla::ModPerl::ResponseHandler=HASH(0x55b6fb21d800)') called at /usr/lib64/perl5/vendor_perl/ModPerl/Registry.pm line 32
ModPerl::Registry::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x55b6fb23c358)') called at /var/www/html/bugzilla/mod_perl.pl line 139
Bugzilla::ModPerl::ResponseHandler::handler('Bugzilla::ModPerl::ResponseHandler', 'Apache2::RequestRec=SCALAR(0x55b6fb23c358)') called at (eval 1985) line 0
eval {...} called at (eval 1985) line 0

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

Bug 1781707: openshift-sdn, CNO: handle new kubeconfig path

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

juanluisvaladas · 2019-12-10T20:30:03Z

/retest
/lgtm as a long as the location of the apiserver's kubeconfig doesn't change very often.
In the long run we probably want to have a standard for every component managed by the cluster operator.

juanluisvaladas · 2019-12-10T20:30:23Z

/lgtm

juanluisvaladas

I'm not very fond of having this defined in multiple places, but gets the job done.

danwinship · 2019-12-10T20:47:24Z

Have you discussed this with whatever team moved the file so they know we're depending on it?

danwinship · 2019-12-10T20:49:52Z

also maybe a good excuse to switch sdn to using KUBERNETES_SERVICE_HOST/KUBERNETES_SERVICE_PORT overrides? We ought to be able to just set those variables and then stop passing --url-only-kubeconfig, right?

openshift-bot · 2019-12-10T23:14:38Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-10T23:27:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T00:19:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T00:32:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T00:45:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T01:24:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T01:37:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T01:50:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T02:16:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T02:29:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T02:42:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T02:55:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T03:08:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T03:21:40Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T03:34:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-11T04:00:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

squeed · 2019-12-11T17:16:42Z

/hold cancel
/retest

squeed · 2019-12-12T14:03:45Z

Bah:
open /var/lib/kubelet/kubeconfig: permission denied

squeed · 2019-12-13T08:32:31Z

weird, failed with no artifacts. either fluke or we really nuked the cluster.
/test e2e-gcp-upgrade

squeed · 2019-12-16T17:15:13Z

OK, now I'm really confused: the CNO is privileged and yet it's still getting permission denied.

danwinship · 2019-12-16T18:59:28Z

selinux

squeed · 2019-12-16T19:20:22Z

selinux

Yup, but privileged: true should be sufficient. Bah.

We can't rely on the nodes having a kubeconfig where we used to expect it. The masters still have it, though, so we can just get it in the CNO and pass it to the SDN in a manifest.

squeed · 2019-12-20T08:55:35Z

Well, that's never going to pass. Looking at the manifests, the kubeconfig we expect should continue to exist on the masters. So, we can just get it from the CNO to fix the immediate in-the-wild bug.

So, undo the changes to the CNO, but still stop the SDN from reading kubeconfig.

squeed · 2019-12-20T10:28:11Z

gcp took a nap
/retest

squeed · 2019-12-20T10:34:22Z

filed SDN-740 to stop reading the kubeconfig.

squeed · 2019-12-20T13:04:13Z

still more gcp oddities
/retest

pecameron

/lgtm

openshift-ci-robot · 2019-12-20T13:37:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juanluisvaladas, pecameron, squeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [squeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-12-20T14:38:13Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2019-12-20T16:09:40Z

@squeed: All pull requests linked via external trackers have merged. Bugzilla bug 1781707 has been moved to the MODIFIED state.

In response to this:

Bug 1781707: openshift-sdn, CNO: handle new kubeconfig path

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

juanluisvaladas · 2019-12-20T16:22:47Z

/cherrypick release-4.3

openshift-cherrypick-robot · 2019-12-20T16:23:23Z

@juanluisvaladas: new pull request created: #429

In response to this:

/cherrypick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2019-12-20T16:40:26Z

Let's pick this to 4.2 too.

juanluisvaladas · 2019-12-20T19:13:17Z

Before merging? OK
/cherrypick release-4.2

openshift-cherrypick-robot · 2019-12-20T19:13:22Z

@juanluisvaladas: #420 failed to apply on top of branch "release-4.2":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	bindata/network/openshift-sdn/controller.yaml
M	bindata/network/openshift-sdn/sdn.yaml
Falling back to patching base and 3-way merge...
Auto-merging bindata/network/openshift-sdn/sdn.yaml
CONFLICT (content): Merge conflict in bindata/network/openshift-sdn/sdn.yaml
Auto-merging bindata/network/openshift-sdn/controller.yaml
Patch failed at 0001 sdn: Get apiserver URL from CNO, not from kubeconfig

In response to this:

Before merging? OK
/cherrypick release-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed requested review from juanluisvaladas and JacobTanenbaum December 10, 2019 17:27

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 10, 2019

openshift-ci-robot requested review from alexanderConstantinescu and danwinship December 10, 2019 17:28

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 10, 2019

openshift-ci-robot assigned juanluisvaladas Dec 10, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 10, 2019

juanluisvaladas reviewed Dec 10, 2019

View reviewed changes

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 11, 2019

sdn: Get apiserver URL from CNO, not from kubeconfig

9db6c73

We can't rely on the nodes having a kubeconfig where we used to expect it. The masters still have it, though, so we can just get it in the CNO and pass it to the SDN in a manifest.

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 20, 2019

pecameron reviewed Dec 20, 2019

View reviewed changes

openshift-ci-robot assigned pecameron Dec 20, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 20, 2019

openshift-merge-robot merged commit 0981500 into openshift:master Dec 20, 2019

openshift-cherrypick-robot mentioned this pull request Dec 20, 2019

[release-4.3] Bug 1785675: openshift-sdn, CNO: handle new kubeconfig path #429

Merged

juanluisvaladas mentioned this pull request Dec 23, 2019

[release-4.2] Bug 1785728: openshift-sdn, CNO: handle new kubeconfig path #431

Merged

danwinship mentioned this pull request Mar 17, 2020

Drop --url-only-kubeconfig, require --node-name / --node-ip openshift/sdn#121

Merged

squeed deleted the sdn-kubeconfig-path branch October 14, 2020 16:07

Bug 1781707: openshift-sdn, CNO: handle new kubeconfig path #420

Bug 1781707: openshift-sdn, CNO: handle new kubeconfig path #420

Conversation

squeed commented Dec 10, 2019

openshift-ci-robot commented Dec 10, 2019

juanluisvaladas commented Dec 10, 2019

juanluisvaladas commented Dec 10, 2019

juanluisvaladas left a comment

Choose a reason for hiding this comment

danwinship commented Dec 10, 2019

danwinship commented Dec 10, 2019

openshift-bot commented Dec 10, 2019

openshift-bot commented Dec 10, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

openshift-bot commented Dec 11, 2019

squeed commented Dec 11, 2019

squeed commented Dec 12, 2019

squeed commented Dec 13, 2019

squeed commented Dec 16, 2019

danwinship commented Dec 16, 2019

squeed commented Dec 16, 2019

squeed commented Dec 20, 2019

squeed commented Dec 20, 2019

squeed commented Dec 20, 2019

squeed commented Dec 20, 2019

pecameron left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 20, 2019

openshift-bot commented Dec 20, 2019

openshift-ci-robot commented Dec 20, 2019

juanluisvaladas commented Dec 20, 2019

openshift-cherrypick-robot commented Dec 20, 2019

squeed commented Dec 20, 2019

juanluisvaladas commented Dec 20, 2019

openshift-cherrypick-robot commented Dec 20, 2019