[release-4.13] OCPBUGS-19894: remove prestop hooks for northd, sbdbd and nbdb #2001

jluhrsen · 2023-09-13T15:23:23Z

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart.

removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state.

[0] cri-o/cri-o#7168
[1]

cluster-network-operator/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml

Lines 74 to 82 in c55f191

    
                     quit() { 
        
                       echo "$(date -Iseconds) - stopping ovn-northd" 
        
                       OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd 
        
                       echo "$(date -Iseconds) - ovn-northd stopped" 
        
                       rm -f /var/run/ovn/ovn-northd.pid 
        
                       exit 0 
        
                     } 
        
                     # end of quit 
        
                     trap quit TERM INT

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart. removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state. [0] cri-o/cri-o#7168 [1] https://github.com/openshift/cluster-network-operator/blob/c55f19132864a9d8fe347ad6a8280eaa2f60e839/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml#L74-L82 Signed-off-by: Jamo Luhrsen <jluhrsen@gmail.com>

openshift-ci-robot · 2023-09-19T17:24:42Z

@jluhrsen: This pull request references Jira Issue OCPBUGS-17391, which is invalid:

expected the bug to target the "4.13.z" version, but it targets "4.14.0" instead
expected Jira Issue OCPBUGS-17391 to depend on a bug targeting a version in 4.14.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart.

removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state.

[0] cri-o/cri-o#7168
[1]

cluster-network-operator/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml

Lines 74 to 82 in c55f191

quit() {

echo "$(date -Iseconds) - stopping ovn-northd"

OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd

echo "$(date -Iseconds) - ovn-northd stopped"

rm -f /var/run/ovn/ovn-northd.pid

exit 0

}

# end of quit

trap quit TERM INT

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-09-28T12:57:31Z

@jluhrsen: This pull request references Jira Issue OCPBUGS-19894, which is invalid:

expected dependent Jira Issue OCPBUGS-19808 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart.

removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state.

[0] cri-o/cri-o#7168
[1]

cluster-network-operator/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml

Lines 74 to 82 in c55f191

quit() {

echo "$(date -Iseconds) - stopping ovn-northd"

OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd

echo "$(date -Iseconds) - ovn-northd stopped"

rm -f /var/run/ovn/ovn-northd.pid

exit 0

}

# end of quit

trap quit TERM INT

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jluhrsen · 2023-10-02T21:08:20Z

/retest

jluhrsen · 2023-10-02T21:08:45Z

/jira refresh

openshift-ci-robot · 2023-10-02T21:08:48Z

@jluhrsen: This pull request references Jira Issue OCPBUGS-19894, which is invalid:

expected dependent Jira Issue OCPBUGS-19808 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2023-10-02T23:08:11Z

@jluhrsen: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

jluhrsen · 2023-10-03T14:56:45Z

/jira refresh

openshift-ci-robot · 2023-10-03T14:56:52Z

@jluhrsen: This pull request references Jira Issue OCPBUGS-19894, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-19808 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-19808 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-10-03T14:56:54Z

@jluhrsen: This pull request references Jira Issue OCPBUGS-19894, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.13.z) matches configured target version for branch (4.13.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-19808 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-19808 targets the "4.14.0" version, which is one of the valid target versions: 4.14.0
bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart.

removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state.

[0] cri-o/cri-o#7168
[1]

cluster-network-operator/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml

Lines 74 to 82 in c55f191

quit() {

echo "$(date -Iseconds) - stopping ovn-northd"

OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd

echo "$(date -Iseconds) - ovn-northd stopped"

rm -f /var/run/ovn/ovn-northd.pid

exit 0

}

# end of quit

trap quit TERM INT

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcaamano · 2023-10-03T16:22:10Z

/lgtm
/approve
/label backport-risk-assessed

openshift-ci · 2023-10-03T16:23:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcaamano, jluhrsen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jcaamano]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

asood-rh · 2023-10-03T16:43:25Z

/label cherry-pick-approved

openshift-ci-robot · 2023-10-03T19:11:01Z

/retest-required

Remaining retests: 0 against base HEAD 777eb7b and 2 for PR HEAD 55ae57d in total

jluhrsen · 2023-10-04T04:11:01Z

/retest

jluhrsen · 2023-10-04T15:03:29Z

/retest

jluhrsen · 2023-10-04T16:43:59Z

/retest

jluhrsen · 2023-10-04T20:41:35Z

/retest

flavio-fernandes · 2023-10-05T11:53:03Z

/retest-required

jluhrsen · 2023-10-05T15:38:57Z

/retest

flavio-fernandes · 2023-10-05T22:08:44Z

/retest-required

jluhrsen · 2023-10-06T02:04:57Z

/retest

jluhrsen · 2023-10-06T03:32:49Z

/retest

jluhrsen · 2023-10-06T05:13:25Z

/retest

flavio-fernandes · 2023-10-06T13:06:03Z

/retest-required

sdodson · 2023-10-06T15:47:02Z

/override ci/prow/e2e-metal-ipi-ovn-ipv6
This job is known to be broken at the moment

openshift-ci · 2023-10-06T15:47:19Z

@sdodson: sdodson unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight.

In response to this:

/override ci/prow/e2e-metal-ipi-ovn-ipv6
This job is known to be broken at the moment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dcbw · 2023-10-06T16:24:22Z

/override ci/prow/e2e-metal-ipi-ovn-ipv6

dougbtv · 2023-10-06T16:24:28Z

/override ci/prow/e2e-metal-ipi-ovn-ipv6

openshift-ci · 2023-10-06T16:24:59Z

@dcbw: Overrode contexts on behalf of dcbw: ci/prow/e2e-metal-ipi-ovn-ipv6

In response to this:

/override ci/prow/e2e-metal-ipi-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-10-06T16:25:11Z

@dougbtv: Overrode contexts on behalf of dougbtv: ci/prow/e2e-metal-ipi-ovn-ipv6

In response to this:

/override ci/prow/e2e-metal-ipi-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-10-06T16:25:14Z

@jluhrsen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ipi-ovn-ipv6	`55ae57d`	link	true	`/test e2e-metal-ipi-ovn-ipv6`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-10-06T16:27:41Z

@jluhrsen: Jira Issue OCPBUGS-19894: All pull requests linked via external trackers have merged:

openshift/cluster-network-operator#2001

Jira Issue OCPBUGS-19894 has been moved to the MODIFIED state.

In response to this:

when the prestop hook executes the ovn-ctl stop... command this causes the containers main script to exit which also propagates to the main container (PID 1) process which exits causing the container to exit. This is not neccessarily a problem except that with recent changes in CRI-O [0] there is some problem with how kubelet views this case. the containers are Exited for crio but seen as Ready from kubelet's perspective. Because of this, the pod gets stuck in Terminating state until it times out (~3m) and then containers will restart.

removing the prestop hook let's the SIGTERM reach the configured trap (example [1]) run which effectively does the same thing as what the prestop hook would have done. This prevents the pod from being stuck in Terminating state.

[0] cri-o/cri-o#7168
[1]

cluster-network-operator/bindata/network/ovn-kubernetes/self-hosted/single-zone-interconnect/ovnkube-master.yaml

Lines 74 to 82 in c55f191

quit() {

echo "$(date -Iseconds) - stopping ovn-northd"

OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd

echo "$(date -Iseconds) - ovn-northd stopped"

rm -f /var/run/ovn/ovn-northd.pid

exit 0

}

# end of quit

trap quit TERM INT

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jluhrsen · 2023-10-06T16:29:49Z

/cherry-pick release-4.12

openshift-cherrypick-robot · 2023-10-06T16:30:35Z

@jluhrsen: new pull request created: #2055

In response to this:

/cherry-pick release-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2023-10-07T09:15:51Z

Fix included in accepted release 4.13.0-0.nightly-2023-10-07-050717

openshift-ci bot requested review from abhat and danwinship September 13, 2023 15:25

jluhrsen changed the title ~~JIRA: https://issues.redhat.com/browse/OCPBUGS-17391~~ OCPBUGS-17391: remove prestop hooks for northd, sbdbd and nbdb Sep 19, 2023

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 19, 2023

jluhrsen mentioned this pull request Sep 19, 2023

OCPBUGS-17391: remove prestop hooks for northd, sbdbd and nbdb #1978

Merged

dcbw changed the title ~~OCPBUGS-17391: remove prestop hooks for northd, sbdbd and nbdb~~ [release-4.13] OCPBUGS-17391: remove prestop hooks for northd, sbdbd and nbdb Sep 28, 2023

dcbw changed the title ~~[release-4.13] OCPBUGS-17391: remove prestop hooks for northd, sbdbd and nbdb~~ [release-4.13] OCPBUGS-19894: remove prestop hooks for northd, sbdbd and nbdb Sep 28, 2023

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Oct 3, 2023

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 3, 2023

openshift-ci bot requested a review from anuragthehatter October 3, 2023 14:56

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Oct 3, 2023

openshift-ci bot assigned anuragthehatter, asood-rh, eurijon, huiran0826, jechen0648, mffiedler and qiowang721 Oct 3, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 3, 2023

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Oct 3, 2023

openshift-ci bot merged commit 961ca3d into openshift:release-4.13 Oct 6, 2023
31 checks passed

openshift-cherrypick-robot mentioned this pull request Oct 6, 2023

[release-4.12] OCPBUGS-20197: remove prestop hooks for northd, sbdbd and nbdb #2055

Merged

jluhrsen deleted the no-prestop-hook-4.13 branch April 19, 2024 22:11

	quit() {
	echo "$(date -Iseconds) - stopping ovn-northd"
	OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd
	echo "$(date -Iseconds) - ovn-northd stopped"
	rm -f /var/run/ovn/ovn-northd.pid
	exit 0
	}
	# end of quit
	trap quit TERM INT

[release-4.13] OCPBUGS-19894: remove prestop hooks for northd, sbdbd and nbdb #2001

[release-4.13] OCPBUGS-19894: remove prestop hooks for northd, sbdbd and nbdb #2001

Conversation

jluhrsen commented Sep 13, 2023

openshift-ci-robot commented Sep 19, 2023

openshift-ci-robot commented Sep 28, 2023

jluhrsen commented Oct 2, 2023

jluhrsen commented Oct 2, 2023

openshift-ci-robot commented Oct 2, 2023

openshift-merge-robot commented Oct 2, 2023

jluhrsen commented Oct 3, 2023

openshift-ci-robot commented Oct 3, 2023

openshift-ci-robot commented Oct 3, 2023

jcaamano commented Oct 3, 2023

openshift-ci bot commented Oct 3, 2023

asood-rh commented Oct 3, 2023

openshift-ci-robot commented Oct 3, 2023

jluhrsen commented Oct 4, 2023

jluhrsen commented Oct 4, 2023

jluhrsen commented Oct 4, 2023

jluhrsen commented Oct 4, 2023

flavio-fernandes commented Oct 5, 2023

jluhrsen commented Oct 5, 2023

flavio-fernandes commented Oct 5, 2023

jluhrsen commented Oct 6, 2023

jluhrsen commented Oct 6, 2023

jluhrsen commented Oct 6, 2023

flavio-fernandes commented Oct 6, 2023

sdodson commented Oct 6, 2023

openshift-ci bot commented Oct 6, 2023

dcbw commented Oct 6, 2023

dougbtv commented Oct 6, 2023

openshift-ci bot commented Oct 6, 2023

openshift-ci bot commented Oct 6, 2023

openshift-ci bot commented Oct 6, 2023

openshift-ci-robot commented Oct 6, 2023

jluhrsen commented Oct 6, 2023

openshift-cherrypick-robot commented Oct 6, 2023

openshift-merge-robot commented Oct 7, 2023