Bug 1883903:Add retries to SDN's RBAC proxy #786

juanluisvaladas · 2020-09-10T10:57:18Z

Because kube-proxy may not be initialized by the time the RBAC proxy
starts it may crashloop for a while. Doesn't have any actual impact but
the restarts show in oc get pod and people may worry about that.

danielmellado

Overall LGTM, although, could you please put up a warning message should the retry fails?

juanluisvaladas · 2020-09-10T11:05:19Z

@danielmellado Done

danwinship · 2020-09-10T11:21:11Z

bindata/network/openshift-sdn/sdn.yaml

+          until [ "$retries" -ge 20 ]
+          do


Suggested change

until [ "$retries" -ge 20 ]

do

while [[ "${retries}" -lt 20 ]]; do

all the stylistic suggestions here still apply, here and elsewhere. (single-line rather than multi-line and while rather than until because those are both more standard. [[ rather than [ and ${retries} rather than $retries because they have fewer gotchas and it matches the kube/OCP shell script style guide that was never finalized...

danwinship · 2020-09-10T11:25:42Z

bindata/network/openshift-sdn/sdn.yaml

+              "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/openshift-sdn/services/sdn" |
+                python -c 'import json,sys; print(json.load(sys.stdin)["metadata"]["creationTimestamp"])' &&
+                  break ||
+                  echo "WARN: Failed to get sdn service from API" 1>&2


If this is expected it ~~should~~ shouldn't be a "warning". And you should say you're retrying.

Also, can you use if/else rather than &&/||? (I realize the variable assignment might make that tricky, so maybe no...)

(EDIT: fixed "shouldn't")

danwinship · 2020-09-10T11:27:07Z

bindata/network/openshift-sdn/sdn.yaml

+            )
+            retries=$(( retries + 1 ))
+            sleep 15
+          done

          TS=$(date -d "${TS}" +%s)
          WARN_TS=$(( ${TS} + $(( 20 * 60)) ))


hm... github won't let me review outside the context of the diff, but below here I see:

if [[ "${CUR_TS}" -gt "WARN_TS" ]]; then echo $(date -Iseconds) WARN: sdn-metrics-certs not mounted after 20 minutes. elif [[ "${HAS_LOGGED_INFO}" -eq 0 ]] ; then echo $(date -Iseconds) INFO: sdn-metrics-certs not mounted. Waiting one hour.

"one hour" should be "20 minutes" shouldn't it?

Yes, will slide that in the PR

danielmellado · 2020-09-10T11:50:44Z

bindata/network/openshift-sdn/sdn.yaml

+              --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
+              -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+              "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/openshift-sdn/services/sdn" |
+                python -c 'import json,sys; print(json.load(sys.stdin)["metadata"]["creationTimestamp"])' &&


Just a nitpick, but are we 100% sure that the json python module would be available within the system? Maybe there's something around that installs it and I'm not aware of it, but if that's not the case, we should also catch that exception or make sure it's installed.

It's a RHEL container so we're guaranteed to have it.

juanluisvaladas · 2020-09-10T21:50:28Z

@danwinship while I was addressing it, I realized it never never exited if the retries were never successful, so besides addressing your comments I also added these new lines:

          if [ "${retries}" -ge 20 ]; then
            echo $(date -Iseconds) FATAL: Unable to get sdn service from API.
            exit 1
          fi

juanluisvaladas · 2020-09-10T21:53:08Z

/retest

danwinship · 2020-09-12T15:09:53Z

Do we need a similar patch for any of the other daemonsets?

juanluisvaladas · 2020-09-13T18:12:08Z

Yes, ovnkube-node and ovnkube-master daemonsets will need this as well, but because #778 is not merged yet I can't modify it here, so I asked @bond95 to add this code in his PR

rcarrillocruz · 2020-09-14T11:33:05Z

/lgtm

openshift-bot · 2020-09-14T11:38:20Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T11:51:43Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T12:47:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T12:56:21Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T13:22:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T13:35:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T13:48:26Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-14T14:14:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

juanluisvaladas · 2020-09-16T14:15:55Z

Still flaky
/hold

juanluisvaladas · 2020-09-18T16:54:18Z

There's something wrong, just added a -x to get more logging, don't review yet

juanluisvaladas · 2020-09-21T09:21:39Z

@danwinship I had to replace the if curl | python; then break; fi for a curl | python && break.

With the variable asignation it wasn't working as I expected.

Because kube-proxy may not be initialized by the time the RBAC proxy starts it may crashloop for a while. Doesn't have any actual impact but the restarts show in oc get pod and people may worry about that.

juanluisvaladas · 2020-09-30T11:08:01Z

/hold cancel

juanluisvaladas · 2020-09-30T11:12:07Z

There was an issue doing the break inside the subshell. Now this is fixed and ready to merge.Can you please lgtm?

rcarrillocruz · 2020-09-30T11:16:32Z

/lgtm

openshift-ci-robot · 2020-09-30T11:16:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juanluisvaladas, rcarrillocruz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rcarrillocruz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-09-30T12:54:45Z

@juanluisvaladas: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-azure	`7782ba1`	link	`/test e2e-azure`
ci/prow/e2e-vsphere	`7782ba1`	link	`/test e2e-vsphere`
ci/prow/e2e-metal-ipi	`dbdc25e`	link	`/test e2e-metal-ipi`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-09-30T12:56:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

juanluisvaladas · 2020-09-30T13:38:20Z

/retitle Bug 1883903:Add retries to SDN's RBAC proxy

openshift-ci-robot · 2020-09-30T13:38:47Z

@juanluisvaladas: This pull request references Bugzilla bug 1883903, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1883903:Add retries to SDN's RBAC proxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-09-30T13:52:49Z

@juanluisvaladas: All pull requests linked via external trackers have merged:

openshift/cluster-network-operator#786

Bugzilla bug 1883903 has been moved to the MODIFIED state.

In response to this:

Bug 1883903:Add retries to SDN's RBAC proxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from aojea and rcarrillocruz September 10, 2020 10:57

juanluisvaladas mentioned this pull request Sep 10, 2020

Bug 1884430: Move kube-rbac-proxy to general ovnkube DeamonSet #778

Merged

danielmellado suggested changes Sep 10, 2020

View reviewed changes

juanluisvaladas force-pushed the rbac-add-retries branch 2 times, most recently from 9b6c5df to 0cc2102 Compare September 10, 2020 11:05

juanluisvaladas force-pushed the rbac-add-retries branch from 0cc2102 to bc88d74 Compare September 10, 2020 11:07

danwinship reviewed Sep 10, 2020

View reviewed changes

danielmellado reviewed Sep 10, 2020

View reviewed changes

juanluisvaladas force-pushed the rbac-add-retries branch 2 times, most recently from 943a321 to f1efa45 Compare September 10, 2020 21:50

juanluisvaladas force-pushed the rbac-add-retries branch from f1efa45 to 36eedb4 Compare September 11, 2020 08:32

juanluisvaladas force-pushed the rbac-add-retries branch from 36eedb4 to 92a8c6f Compare September 14, 2020 09:24

openshift-ci-robot assigned rcarrillocruz Sep 14, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 14, 2020

juanluisvaladas changed the title ~~Add retries to SDN's RBAC proxy~~ [WIP]Add retries to SDN's RBAC proxy Sep 18, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 18, 2020

juanluisvaladas force-pushed the rbac-add-retries branch from 92a8c6f to 7a0dca9 Compare September 18, 2020 16:53

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Sep 18, 2020

juanluisvaladas force-pushed the rbac-add-retries branch 2 times, most recently from cc3bd35 to 8a95b2d Compare September 21, 2020 09:20

juanluisvaladas force-pushed the rbac-add-retries branch from 8a95b2d to 7782ba1 Compare September 21, 2020 11:02

juanluisvaladas force-pushed the rbac-add-retries branch from 7782ba1 to ae77c76 Compare September 30, 2020 09:16

Add retries to SDN's RBAC proxy

dbdc25e

Because kube-proxy may not be initialized by the time the RBAC proxy starts it may crashloop for a while. Doesn't have any actual impact but the restarts show in oc get pod and people may worry about that.

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 30, 2020

juanluisvaladas changed the title ~~[WIP]Add retries to SDN's RBAC proxy~~ Add retries to SDN's RBAC proxy Sep 30, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 30, 2020

juanluisvaladas force-pushed the rbac-add-retries branch from ae77c76 to dbdc25e Compare September 30, 2020 11:09

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 30, 2020

openshift-ci-robot changed the title ~~Add retries to SDN's RBAC proxy~~ Bug 1883903:Add retries to SDN's RBAC proxy Sep 30, 2020

openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Sep 30, 2020

openshift-merge-robot merged commit fe416a3 into openshift:master Sep 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1883903:Add retries to SDN's RBAC proxy #786

Bug 1883903:Add retries to SDN's RBAC proxy #786

juanluisvaladas commented Sep 10, 2020

danielmellado left a comment

juanluisvaladas commented Sep 10, 2020

danwinship Sep 10, 2020

danwinship Sep 12, 2020

danwinship Sep 10, 2020 •

edited

danwinship Sep 10, 2020

juanluisvaladas Sep 10, 2020

danielmellado Sep 10, 2020

juanluisvaladas Sep 10, 2020

juanluisvaladas commented Sep 10, 2020

juanluisvaladas commented Sep 10, 2020

danwinship commented Sep 12, 2020

juanluisvaladas commented Sep 13, 2020

rcarrillocruz commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

juanluisvaladas commented Sep 16, 2020

juanluisvaladas commented Sep 18, 2020

juanluisvaladas commented Sep 21, 2020

juanluisvaladas commented Sep 30, 2020

juanluisvaladas commented Sep 30, 2020

rcarrillocruz commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020 •

edited

openshift-bot commented Sep 30, 2020

juanluisvaladas commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

	until [ "$retries" -ge 20 ]
	do
	while [[ "${retries}" -lt 20 ]]; do

Bug 1883903:Add retries to SDN's RBAC proxy #786

Bug 1883903:Add retries to SDN's RBAC proxy #786

Conversation

juanluisvaladas commented Sep 10, 2020

danielmellado left a comment

Choose a reason for hiding this comment

juanluisvaladas commented Sep 10, 2020

danwinship Sep 10, 2020

Choose a reason for hiding this comment

danwinship Sep 12, 2020

Choose a reason for hiding this comment

danwinship Sep 10, 2020 • edited

Choose a reason for hiding this comment

danwinship Sep 10, 2020

Choose a reason for hiding this comment

juanluisvaladas Sep 10, 2020

Choose a reason for hiding this comment

danielmellado Sep 10, 2020

Choose a reason for hiding this comment

juanluisvaladas Sep 10, 2020

Choose a reason for hiding this comment

juanluisvaladas commented Sep 10, 2020

juanluisvaladas commented Sep 10, 2020

danwinship commented Sep 12, 2020

juanluisvaladas commented Sep 13, 2020

rcarrillocruz commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

openshift-bot commented Sep 14, 2020

juanluisvaladas commented Sep 16, 2020

juanluisvaladas commented Sep 18, 2020

juanluisvaladas commented Sep 21, 2020

juanluisvaladas commented Sep 30, 2020

juanluisvaladas commented Sep 30, 2020

rcarrillocruz commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020 • edited

openshift-bot commented Sep 30, 2020

juanluisvaladas commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

openshift-ci-robot commented Sep 30, 2020

danwinship Sep 10, 2020 •

edited

openshift-ci-robot commented Sep 30, 2020 •

edited