Bug 1894477: Fix bash in nodeip-configuration.service #2199

tnozicka · 2020-11-04T10:56:58Z

The current bash code in the service looks invalid.

# on 4.6.0-0.okd-2020-11-03-123207
[root@master-0 ~]# systemctl status nodeip-configuration
● nodeip-configuration.service - Writes IP address configuration so that kubelet and crio services select a valid node IP
     Loaded: loaded (/etc/systemd/system/nodeip-configuration.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-11-04 08:22:50 UTC; 22min ago
    Process: 980 ExecStart=/bin/bash -c     until    /usr/bin/podman run --rm    --authfile /var/lib/kubelet/config.json    --net=host    --volume /etc/systemd/system:/etc/systemd/system:z    registry.svc.ci.openshift.org/origin/4.6-2020>
   Main PID: 980 (code=exited, status=1/FAILURE)
        CPU: 4ms
lis 04 08:22:50 master-0 systemd[1]: Starting Writes IP address configuration so that kubelet and crio services select a valid node IP...
lis 04 08:22:50 master-0 bash[980]: /bin/bash: -c: line 0: syntax error near unexpected token `done'
lis 04 08:22:50 master-0 bash[980]: /bin/bash: -c: line 0: `    until    /usr/bin/podman run --rm    --authfile /var/lib/kubelet/config.json    --net=host    --volume /etc/systemd/system:/etc/systemd/system:z    registry.svc.ci.openshift>
lis 04 08:22:50 master-0 systemd[1]: nodeip-configuration.service: Main process exited, code=exited, status=1/FAILURE
lis 04 08:22:50 master-0 systemd[1]: nodeip-configuration.service: Failed with result 'exit-code'.
lis 04 08:22:50 master-0 systemd[1]: Failed to start Writes IP address configuration so that kubelet and crio services select a valid node IP.

It borked OKD but somehow got through ocp test suites - is this not covered?

Fixes #2179

/cc @danwinship @knobunc @cgwalters

vrutkovs · 2020-11-04T11:05:06Z

/test e2e-vsphere-upi

openshift-ci-robot · 2020-11-04T11:06:06Z

@tnozicka: This pull request references Bugzilla bug 1894477, which is invalid:

expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1894477: Fix bash in nodeip-configuration.service

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tnozicka · 2020-11-04T11:11:59Z

/bugzilla refresh

openshift-ci-robot · 2020-11-04T11:12:09Z

@tnozicka: This pull request references Bugzilla bug 1894477, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.0) matches configured target release for branch (4.7.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tnozicka · 2020-11-04T14:22:35Z

/retest

LorbusChris · 2020-11-04T20:18:44Z

/test e2e-vsphere-upi
/test e2e-gcp-op

tnozicka · 2020-11-05T06:50:10Z

e2e-vsphere-upi seem consistently failing with infra login but it doesn't actually test the scenario because it has it's own platform, not None - so we are good to ship without it

runcom · 2020-11-05T08:51:04Z

/approve

sinnykumari · 2020-11-05T10:20:36Z

/retest

danwinship · 2020-11-05T12:14:17Z

Maybe a bash version thing? I get the same behavior with both versions with "GNU bash, version 5.0.17(1)-release (x86_64-redhat-linux-gnu)" (F32):

danw@p50:~> /bin/bash -c " \
>     until \
>     /usr/bin/podman run --rm \
>     --authfile /var/lib/kubelet/config.json \
>     --net=host \
>     --volume /etc/systemd/system:/etc/systemd/system:z \
>     {{ .Images.baremetalRuntimeCfgImage }} \
>     node-ip \
>     set --retry-on-failure \
>     ; \
>     do \
>     sleep 5; \
>     done"
Error: error checking authfile path /var/lib/kubelet/config.json: stat /var/lib/kubelet/config.json: no such file or directory
Error: error checking authfile path /var/lib/kubelet/config.json: stat /var/lib/kubelet/config.json: no such file or directory
^C
danw@p50:~> /bin/bash -c " \
    until \
    /usr/bin/podman run --rm \
    --authfile /var/lib/kubelet/config.json \
    --net=host \
    --volume /etc/systemd/system:/etc/systemd/system:z \
    {{ .Images.baremetalRuntimeCfgImage }} \
    node-ip \
    set --retry-on-failure; \
    do \
    sleep 5; \
    done"
Error: error checking authfile path /var/lib/kubelet/config.json: stat /var/lib/kubelet/config.json: no such file or directory
Error: error checking authfile path /var/lib/kubelet/config.json: stat /var/lib/kubelet/config.json: no such file or directory
^C

/lgtm
but I don't know why it should be needed?

openshift-ci-robot · 2020-11-05T12:14:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, runcom, tnozicka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-11-05T12:25:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

danwinship · 2020-11-05T12:27:09Z

It borked OKD but somehow got through ocp test suites - is this not covered?

I don't know if MCO/OCP has any test suites that cover this but the reporter had verified that it worked: https://bugzilla.redhat.com/show_bug.cgi?id=1872632#c70

openshift-bot · 2020-11-05T12:38:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-05T12:52:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-05T13:05:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

tnozicka · 2020-11-09T17:53:44Z

I think Pending — Not mergeable. Retesting 7 jobs. means it is ignoring the e2e-gcp-op job, but other tests are flaking on the batch merge. Let's see how it goes. Otherwise someone from top level OWNERS can try /override e2e-gcp-op.

openshift-bot · 2020-11-09T18:40:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-09T18:53:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-09T21:16:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-09T21:29:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-09T23:39:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-09T23:52:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T02:02:38Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T02:15:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T04:12:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T07:40:25Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T08:58:21Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T09:11:25Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T10:29:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T12:00:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T14:10:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T16:07:24Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T17:02:04Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T17:41:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T19:51:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-10T22:53:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-11T00:50:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-11T01:28:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-merge-robot · 2020-11-11T02:17:23Z

@tnozicka: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-vsphere-upi	`ec3e50d`	link	`/test e2e-vsphere-upi`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-11-11T02:21:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-11-11T02:47:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-11-11T03:44:26Z

@tnozicka: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#2199

Bugzilla bug 1894477 has been moved to the MODIFIED state.

In response to this:

Bug 1894477: Fix bash in nodeip-configuration.service

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Fix bash in nodeip-configuration.service

ec3e50d

openshift-ci-robot requested review from cgwalters, danwinship and knobunc November 4, 2020 10:56

tnozicka changed the title ~~Fix bash in nodeip-configuration.service~~ Bug 1894477: Fix bash in nodeip-configuration.service Nov 4, 2020

openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Nov 4, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 4, 2020

tnozicka mentioned this pull request Nov 4, 2020

[release-4.6] Bug 1894483: Fix bash in nodeip-configuration.service #2200

Merged

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Nov 4, 2020

vrutkovs mentioned this pull request Nov 4, 2020

[Tracking] OKD 4.6 Stable okd-project/okd#348

Closed

6 tasks

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 5, 2020

openshift-ci-robot assigned danwinship Nov 5, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 5, 2020

openshift-merge-robot merged commit aa54de3 into master Nov 11, 2020

tnozicka deleted the fix-nodeip-service branch November 11, 2020 06:55

vrutkovs mentioned this pull request Nov 11, 2020

UPI OKD4.6 bash syntax error in nodeip-configuration.service on FCOS #2179

Closed

Bug 1894477: Fix bash in nodeip-configuration.service #2199

Bug 1894477: Fix bash in nodeip-configuration.service #2199

Conversation

tnozicka commented Nov 4, 2020

vrutkovs commented Nov 4, 2020

openshift-ci-robot commented Nov 4, 2020

tnozicka commented Nov 4, 2020

openshift-ci-robot commented Nov 4, 2020

tnozicka commented Nov 4, 2020

LorbusChris commented Nov 4, 2020

tnozicka commented Nov 5, 2020

runcom commented Nov 5, 2020

sinnykumari commented Nov 5, 2020

danwinship commented Nov 5, 2020

openshift-ci-robot commented Nov 5, 2020

openshift-bot commented Nov 5, 2020

danwinship commented Nov 5, 2020

openshift-bot commented Nov 5, 2020

openshift-bot commented Nov 5, 2020

openshift-bot commented Nov 5, 2020

tnozicka commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 9, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 10, 2020

openshift-bot commented Nov 11, 2020

openshift-bot commented Nov 11, 2020

openshift-merge-robot commented Nov 11, 2020 • edited

openshift-bot commented Nov 11, 2020

openshift-bot commented Nov 11, 2020

openshift-ci-robot commented Nov 11, 2020

openshift-merge-robot commented Nov 11, 2020 •

edited