Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return non-zero code on logexporter failures #95563

Merged
merged 1 commit into from Oct 23, 2020
Merged

Return non-zero code on logexporter failures #95563

merged 1 commit into from Oct 23, 2020

Conversation

tosi3k
Copy link
Member

@tosi3k tosi3k commented Oct 14, 2020

What type of PR is this?
/kind bug

What this PR does / why we need it:
When logexporter daemonset fails to be created, it is not reported in our CI tests as a failure.

This will make the log-dump.sh process return a non-zero code whenever logexporter daemonset creation fails or less than half of the nodes got logexported successfully.

Which issue(s) this PR fixes:
No issue is opened for that.

Special notes for your reviewer:
/sig scalability
/assign @jkaniuk @wojtek-t

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 14, 2020
@k8s-ci-robot
Copy link
Contributor

@tosi3k: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Oct 14, 2020
@jkaniuk
Copy link
Contributor

jkaniuk commented Oct 15, 2020 •
edited

I would like to hide it behind some flag so as to not enable it by default everywhere.

Separately I think it would be better to have that threshold configurable.

Maybe we could merge above and have a flag that default to disabling, but control the threshold?
Like NEEDED_PERCENT_OF_LOG_DUMPED_CORRECTLY, default=0 - bad naming, just to illustrate.

Percent (or fractional ratio) is better than node count as the flag will land in some presets.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 15, 2020
@tosi3k
Copy link
Member Author

tosi3k commented Oct 15, 2020 •
edited

I would like to hide it behind some flag so as to not enable it by default everywhere.

Separately I think it would be better to have that threshold configurable.

Maybe we could merge above and have a flag that default to disabling, but control the threshold?
Like NEEDED_PERCENT_OF_LOG_DUMPED_CORRECTLY, default=0 - bad naming, just to illustrate.

Added LOG_DUMP_SUCCESS_PERCENTAGE env var for that. Not sure about its naming, though, but I couldn't come up with anything better :/.

Edit: renamed it to LOG_DUMP_EXPECTED_SUCCESS_PERCENTAGE - kinda long but exactly describes what its purpose is.

@tosi3k
Copy link
Member Author

tosi3k commented Oct 16, 2020

/retest

@tosi3k
Copy link
Member Author

tosi3k commented Oct 16, 2020

/retest

jkaniuk
jkaniuk reviewed Oct 20, 2020
View reviewed changes
cluster/log-dump/log-dump.sh Outdated
# TODO: Get rid of all the sourcing of bash dependencies eventually.
function setup() {
LOG_DUMP_EXIT_CODE=0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would lower case and move variable initialization outside of the function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

jkaniuk
jkaniuk reviewed Oct 20, 2020
View reviewed changes
cluster/log-dump/log-dump.sh Outdated
@@ -63,6 +63,13 @@ readonly windows_node_otherfiles="C:\\Windows\\MEMORY.dmp"
# file descriptors for large clusters.
readonly max_dump_processes=25

# Default exit code for the log dump process.
log_dump_exit_code=0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as this is rather a marker that log exporting failed, than an exit code that is always used, could you reflect that in variable name and use?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Done.

cluster/log-dump/log-dump.sh Outdated
@@ -675,6 +689,9 @@ function main() {
fi

detect_node_failures
if [[ ${log_dump_expected_success_percentage} -gt 0 ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after switch from error code to flag indicating failure:

if log exporting failed; then
  if [[ ${log_dump_expected_success_percentage} -gt 0 ]]; then
     return 1
   fi
fi

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or combine both in one condition if more readable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combined both conditions into a single one. Done.

jkaniuk
jkaniuk reviewed Oct 21, 2020
View reviewed changes
cluster/log-dump/log-dump.sh Outdated
@@ -675,6 +690,9 @@ function main() {
fi

detect_node_failures
if [[ ${log_dump_expected_success_percentage} -gt 0 && ${logexporter_failed} -ne 0 ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would reverse conditions to be more readable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh, the current order sounds more natural to me (if failing_on_unsuccessful_log_dumping_enabled && logexporter_failed) but I might be slightly biased here because I wrote that logic. Nevertheless, I'll switch the conditions if their sequence sounds more natural.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@jkaniuk
Copy link
Contributor

jkaniuk commented Oct 21, 2020

/lgtm

But please verify if it is working in some manual test
/hold

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Oct 21, 2020
@jkaniuk
Copy link
Contributor

jkaniuk commented Oct 21, 2020

@wojtek-t

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 21, 2020
@tosi3k
Copy link
Member Author

tosi3k commented Oct 21, 2020

But please verify if it is working in some manual test

Tested it using a broken logexporter-daemonset.yaml config that prevents the successful execution of logexporter pods:

diff --git a/cluster/log-dump/logexporter-daemonset.yaml b/cluster/log-dump/logexporter-daemonset.yaml
index 8279ae62f68..97e233a7ccf 100644
--- a/cluster/log-dump/logexporter-daemonset.yaml
+++ b/cluster/log-dump/logexporter-daemonset.yaml
@@ -53,6 +53,7 @@ spec:
         - --extra-log-files={{.ExtraLogFiles}}
         - --extra-systemd-services={{.ExtraSystemdServices}}
         - --sleep-duration=24h
+        - --unknown-flag=unknown-value
         volumeMounts:
         - mountPath: /etc/service-account
           name: service

When LOG_DUMP_EXPECTED_SUCCESS_PERCENTAGE was set to 50, the log dumping was reported as failing in the Prow GCS viewer as expected. When it wasn't set at all, the log dumping process returned a zero exit code (meaning the old behavior persists), so everything works properly.

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 21, 2020
wojtek-t
wojtek-t reviewed Oct 21, 2020
View reviewed changes
Copy link
Member

@wojtek-t wojtek-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small comment - other than that LGTM.

cluster/log-dump/log-dump.sh Outdated
@@ -611,6 +621,11 @@ function dump_nodes_with_logexporter() {
done
fi

# If less than a certain ratio of the nodes got logexported, report an error.
if [[ $((${#NODE_NAMES[@]} - ${#failed_nodes[@]})) -lt $((${#NODE_NAMES[@]} * log_dump_expected_success_percentage / 100)) ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dividing is always tricky - let's instead do:

100 * (node_names - failed_node) < node_names * percentage

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dividing is always tricky

Especially tricky in bash which doesn't know anything about floating-point numbers ;) thanks for the remark, done.

@wojtek-t
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 21, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tosi3k, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 21, 2020
@spiffxp
Copy link
Member

spiffxp commented Oct 21, 2020 •
edited

/test pull-kubernetes-e2e-gce-ubuntu-containerd
flake #91236 - https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/95563/pull-kubernetes-e2e-gce-ubuntu-containerd/1318978540581949440

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

6 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit c446332 into kubernetes:master Oct 23, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Oct 23, 2020
@tosi3k tosi3k mentioned this pull request Oct 23, 2020
Merged
@tosi3k tosi3k deleted the log-dump-verification branch February 9, 2021 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkaniuk jkaniuk jkaniuk left review comments

@wojtek-t wojtek-t wojtek-t left review comments

@spiffxp spiffxp Awaiting requested review from spiffxp

Assignees

@jkaniuk jkaniuk

@wojtek-t wojtek-t

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@tosi3k @k8s-ci-robot @jkaniuk @wojtek-t @spiffxp @fejta-bot