Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1824983: explicitly delete pidfiles when exiting #661

Merged
merged 1 commit into from Jun 9, 2020

Conversation

JacobTanenbaum
Copy link
Contributor

saw this error message in the logs

Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:ovsdb-server: /var/run/openvswitch/ovsdb-server.pid: pidfile check failed (No such process)

when it looked like ovs was restarting causing the sdn pods to spin. explicitly delete

/var/run/openvswitch/ovs-vswitchd.pid and
/var/run/openvswitch/ovsdb-server.pid:

on ovs pod exit

saw this error message in the logs

Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:ovsdb-server: /var/run/openvswitch/ovsdb-server.pid: pidfile check failed (No such process)

when it looked like ovs was restarting causing the sdn pods to spin. explicitly delete

/var/run/openvswitch/ovs-vswitchd.pid and
/var/run/openvswitch/ovsdb-server.pid:

on ovs pod exit
@JacobTanenbaum
Copy link
Contributor Author

/retest

1 similar comment
@JacobTanenbaum
Copy link
Contributor Author

/retest

@abhat
Copy link
Contributor

abhat commented Jun 5, 2020

saw this error message in the logs

Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:ovsdb-server: /var/run/openvswitch/ovsdb-server.pid: pidfile check failed (No such process)

when it looked like ovs was restarting causing the sdn pods to spin. explicitly delete

/var/run/openvswitch/ovs-vswitchd.pid and
/var/run/openvswitch/ovsdb-server.pid:

on ovs pod exit

So this fix seems to be counter-intuitive to the error message. The error message is saying that the pod termination failed because the pid file wasn't found. And we are trying to fix that by adding a EXIT handler that deletes the pid? Can you please explain the need for the change?

@JacobTanenbaum
Copy link
Contributor Author

JacobTanenbaum commented Jun 5, 2020

@abhat it is more important that it runs on the previous iteration of the ovs pod. We want to make sure that we clean up the pidfile before another container tries to start since /var/run is mounted from the host when the container exits if the file remains it could cause problems like this

@abhat
Copy link
Contributor

abhat commented Jun 8, 2020

/lgtm

@abhat
Copy link
Contributor

abhat commented Jun 8, 2020

/approve

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, JacobTanenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2020
@abhat
Copy link
Contributor

abhat commented Jun 8, 2020

Is there a bz to tag this with @JacobTanenbaum?

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

2 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@JacobTanenbaum JacobTanenbaum changed the title explicitly delete pidfiles when exiting 1824983: explicitly delete pidfiles when exiting Jun 8, 2020
@JacobTanenbaum JacobTanenbaum changed the title 1824983: explicitly delete pidfiles when exiting Bug 1824983: explicitly delete pidfiles when exiting Jun 8, 2020
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Jun 8, 2020
@openshift-ci-robot
Copy link
Contributor

@JacobTanenbaum: This pull request references Bugzilla bug 1824983, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1824983: explicitly delete pidfiles when exiting

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 8, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

7 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 9, 2020

@JacobTanenbaum: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-vsphere d5b1b9b link /test e2e-vsphere

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit a0a19b6 into openshift:master Jun 9, 2020
@openshift-ci-robot
Copy link
Contributor

@JacobTanenbaum: All pull requests linked via external trackers have merged: openshift/cluster-network-operator#661. Bugzilla bug 1824983 has been moved to the MODIFIED state.

In response to this:

Bug 1824983: explicitly delete pidfiles when exiting

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants