Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configure StackdriverLogging Windows service to restart on failure. #93765

Merged
merged 1 commit into from Nov 13, 2020
Merged

Configure StackdriverLogging Windows service to restart on failure. #93765

merged 1 commit into from Nov 13, 2020

Conversation

jeremyje
Copy link
Contributor

@jeremyje jeremyje commented Aug 6, 2020 •
edited

What type of PR is this?
/kind bug

What this PR does / why we need it:
This change configures the StackdriverLogging Windows service to automatically restart if it fails. This improves reliability of emitting Stackdriver log output when the host has disruptions that crash the logging agent. The service is configured to restart 1 second after first failure and then 10 seconds for subsequent failures.

Currently, there's a race between GCE/GKE metadata server being reachable and this service starting up. If the metadata server is unavailable it'll crash and never emit logs until manually started.

This change also improves the manual e2e testing documentation.

Which issue(s) this PR fixes:

Fixes #94138

Special notes for your reviewer:

Does this PR introduce a user-facing change?:
NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Aug 6, 2020
@k8s-ci-robot
Copy link
Contributor

Welcome @jeremyje!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 6, 2020
@k8s-ci-robot
Copy link
Contributor

Hi @jeremyje. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added area/provider/gcp Issues or PRs related to gcp provider sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 6, 2020
@jeremyje
Copy link
Contributor Author

jeremyje commented Aug 6, 2020

/test pull-kubernetes-e2e-windows-gce

@k8s-ci-robot
Copy link
Contributor

@jeremyje: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-kubernetes-e2e-windows-gce

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 7, 2020
@jeremyje
Copy link
Contributor Author

jeremyje commented Aug 7, 2020

/assign @pjh

@pjh
Copy link
Contributor

pjh commented Aug 7, 2020

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 7, 2020
@pjh
Copy link
Contributor

pjh commented Aug 7, 2020

/priority important-soon

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Aug 7, 2020
@pjh
Copy link
Contributor

pjh commented Aug 7, 2020

/test pull-kubernetes-e2e-windows-gce

@jeremyje
Copy link
Contributor Author

/test pull-kubernetes-e2e-windows-gce
/test pull-kubernetes-e2e-gce-ubuntu-containerd
/test pull-kubernetes-integration

@jeremyje
Copy link
Contributor Author

/retest

@jeremyje
Copy link
Contributor Author

I've created #94138 where would I propose this in the sig-windows chat room?

@pjh
Copy link
Contributor

pjh commented Aug 20, 2020

I found the chery-pick guide: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md

Someone had sent it to me in the past so I had it bookmarked. It is not referenced anywhere under the actual contributor guide, https://github.com/kubernetes/community/tree/master/contributors/guide :(

@justaugustus justaugustus added this to the v1.20-phase-bug milestone Aug 26, 2020
@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 28, 2020
@justaugustus justaugustus modified the milestones: v1.20-phase-bug, v1.20 Sep 8, 2020
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 12, 2020
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 12, 2020
@jeremyje
Copy link
Contributor Author

@pjh Given the issues with upgrading to Fluent Bit I think we should submit this change anyway just in case the upgrade does not happen within the v1.20 cut off.

@jeremyje
Copy link
Contributor Author

jeremyje commented Nov 12, 2020 via email

@pjh
Copy link
Contributor

pjh commented Nov 12, 2020

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 12, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jeremyje, pjh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jeremyje
Copy link
Contributor Author

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 12, 2020
@jeremyje
Copy link
Contributor Author

/retest

2 similar comments
@jeremyje
Copy link
Contributor Author

/retest

@jeremyje
Copy link
Contributor Author

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 13, 2020 •
edited

@jeremyje: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-windows-gce ef5da75b7fc556d2b337c8f59b5fd593a8f0bfb4 link /test pull-kubernetes-e2e-windows-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 84d15ca into kubernetes:master Nov 13, 2020
@jeremyje jeremyje deleted the restartsd branch November 13, 2020 09:26
@jeremyje jeremyje mentioned this pull request Dec 1, 2020
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pjh pjh pjh left review comments

@mtaufen mtaufen Awaiting requested review from mtaufen

@yliaog yliaog Awaiting requested review from yliaog

Assignees

@pjh pjh

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/gcp Issues or PRs related to gcp provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

GCE Windows StackdriverLogging Service sometimes fails and never restarts on node boot.
6 participants
@jeremyje @k8s-ci-robot @pjh @yliaog @fejta-bot @justaugustus