CNF-12656: Handle SIGTERM for stateroot setup job

[pixelsoccupied]

This PR aims to handle the case when Stateroot setup job may be in-progress and an unexpected SIGTERM is received (e.g moving to Idle while Prep in progress).

The solution is broken into to two parts:

k8s:

• The max wait time before a SIGKILL is sent is determined with TerminationGracePeriodSeconds. This is now set to 60 secs (default 30)
• PreStop lifecycle handler was investigated but not useful in our case. Generally it's used to delay the container from getting SIGTERM and in the meantime k8s does other housekeeping tasks (e.g de-registering Ingress) to avoid any race conditions.

Handler in the code:

• Every app is expected to have it's own signal handler. In this case if seed image is present we allow the rest of the stateroot setup to go through by sleeping at most time set with TerminationGracePeriodSeconds before shutting down.

/cc @donpenney @jc-rh

[openshift-ci-robot]

@pixelsoccupied: This pull request references CNF-12656 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR aims to handle the case when Stateroot setup job may be in-progress and an unexpected SIGTERM is received (e.g moving to Idle while Prep in progress).

The solution is broken into to two parts:

k8s:

• The max wait time before a SIGKILL is sent is determined with TerminationGracePeriodSeconds. This is now set to 60 secs (default 30)
• PreStop lifecycle handler was investigated but not useful in our case. Generally it's used to delay the container from getting SIGTERM and in the meantime k8s does other housekeeping tasks (e.g de-registering Ingress) to avoid any race conditions.

Handler in the code:

• Every app is expected to have it's own signal handler. In this case are simply canceling context and sleep at most time set with TerminationGracePeriodSeconds before shutting down.

/cc @donpenney @jc-rh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@pixelsoccupied: This pull request references CNF-12656 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR aims to handle the case when Stateroot setup job may be in-progress and an unexpected SIGTERM is received (e.g moving to Idle while Prep in progress).

The solution is broken into to two parts:

k8s:

• The max wait time before a SIGKILL is sent is determined with TerminationGracePeriodSeconds. This is now set to 60 secs (default 30)
• PreStop lifecycle handler was investigated but not useful in our case. Generally it's used to delay the container from getting SIGTERM and in the meantime k8s does other housekeeping tasks (e.g de-registering Ingress) to avoid any race conditions.

Handler in the code:

• Every app is expected to have it's own signal handler. In this case if seed image is present we allow the rest of the stateroot setup to go through by sleeping at most time set with TerminationGracePeriodSeconds before shutting down.

/cc @donpenney @jc-rh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[pixelsoccupied]

/hold
will update the graceperiod to 30 mins + document this behaviour

[pixelsoccupied]

/unhold

[pixelsoccupied]

/retest

[pixelsoccupied]

/test ibu-e2e-flow

[donpenney]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jc-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jc-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment