KEP 2258: add node log query by LorbusChris · Pull Request #96120 · kubernetes/kubernetes

LorbusChris · 2020-11-02T19:05:05Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Provide an administrator a streaming view of journal logs on Linux and WinEvent logs on Windows
without them having to implement a client side reader. Only available to cluster admins.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
Upstreaming from OpenShift:

Make viewing node logs from the CLI easy openshift/origin#21701
Bug 1862643: UPSTREAM: 96120: kubelet: Expose a simple Get-WinEvent shim on the kubelet logs endpoint openshift/kubernetes#383

Does this PR introduce a user-facing change?:

Adds feature gate `NodeLogQuery` which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

KEP: KEP 2258: Node service log viewer enhancements#2271
KEP Discussion: Node log query enhancements#2258
Usage: KEP-2258: Node log query documentation website#40003

k8s-ci-robot · 2020-11-02T19:05:14Z

Hi @LorbusChris. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LorbusChris · 2020-11-03T09:37:34Z

/cc @smarterclayton @sttts @aravindhp

k8s-ci-robot · 2020-11-03T09:37:36Z

@LorbusChris: GitHub didn't allow me to request PR reviews from the following users: aravindhp.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @smarterclayton @sttts @aravindhp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LorbusChris · 2020-11-05T18:31:18Z

Somebody please give this an ok-to-test

rphillips · 2020-11-05T20:24:40Z

/ok-to-test
/lgtm

rphillips · 2020-11-05T20:26:35Z

/assign @sjenning

mrunalp · 2023-03-13T20:08:18Z

I think logDir could be renamed to something more specific like nodeLogDir.

mrunalp · 2023-03-13T20:08:35Z

Could we use the constant defined for /var/log?

mrunalp · 2023-03-13T20:29:22Z

Where does the 100 limit come from? Probably worth adding a comment. The systemd limit for a unit name is 256 characters https://www.freedesktop.org/software/systemd/man/systemd.unit.html#:~:text=The%20total%20length%20of%20the,service%20%22%2C%20%22%20.

I changed the limit to 256.

mrunalp · 2023-03-13T20:33:06Z

Need a comment for the limit.

Given the previous comment about the length, I change this to check for number of services with a constant of 4.

mrunalp · 2023-03-13T20:33:27Z

constant for 100000

mrunalp · 2023-03-13T20:35:37Z

Can use something like https://github.com/cyphar/filepath-securejoin to prevent filepath parent walk attacks.

mrunalp · 2023-03-13T20:40:19Z

Consider using https://github.com/cyphar/filepath-securejoin

mrunalp · 2023-03-13T20:40:51Z

Why is this 7?

Good catch. I missed updating this when reducing the heuristics. Fixed.

mrunalp · 2023-03-13T20:41:34Z

nit: was for found -> was found

aravindhp

@mrunalp thank you for the review. I have addressed your comments. Please take a look.

aravindhp · 2023-03-13T21:57:52Z

aravindhp · 2023-03-13T22:02:27Z

aravindhp · 2023-03-13T22:03:26Z

aravindhp · 2023-03-13T22:03:40Z

I changed the limit to 256.

aravindhp · 2023-03-13T22:04:18Z

Given the previous comment about the length, I change this to check for number of services with a constant of 4.

aravindhp · 2023-03-13T22:04:25Z

aravindhp · 2023-03-13T22:04:31Z

aravindhp · 2023-03-13T22:04:56Z

Good catch. I missed updating this when reducing the heuristics. Fixed.

aravindhp · 2023-03-13T22:05:03Z

mrunalp · 2023-03-13T22:35:52Z

100 -> 256? Maybe a constant for it :)

mrunalp · 2023-03-13T22:38:18Z

journalctl won't return an error if we try to fetch logs for a non-existent service, hence we search for it in the list of services known to journalctl

aravindhp

@mrunalp I have addressed your comments. PTAL.

aravindhp · 2023-03-13T23:23:21Z

aravindhp · 2023-03-13T23:23:58Z

Enable the query endpoint only if this gate is enabled in addition to the EnableNodeLogQuery kubelet config option.

Added EnableNodeLogQuery field to kubelet/apis/config/types.go and staging/src/k8s.io/kubelet/config/v1beta1/types.go, then executed. `hack/update-codegen.sh`. This new field will default to off and will need to be explicitly enabled in addition to the NodeLogQuery gate to use the feature.

…dpoint Provide an administrator a streaming view of journal logs on Linux systems using journalctl, and event logs on Windows systems using the Get-WinEvent PowerShell cmdlet without them having to implement a client side reader. Only available to cluster admins. The implementation for journald on Linux was originally done by Clayton Coleman. Introduce a heuristics approach to query logs The logs query for node objects will follow a heuristics approach when asked to query for logs from a service. If asked to get the logs from a service foobar, it will first check if foobar logs to the native OS service log provider. If unable to get logs from these, it will attempt to get logs from /var/foobar, /var/log/foobar.log or /var/log/foobar/foobar.log in that order. The logs sub-command can also directly serve a file if the query looks like a file. Co-authored-by: Clayton Coleman <ccoleman@redhat.com> Co-authored-by: Christian Glombek <cglombek@redhat.com>

k8s-ci-robot · 2023-03-14T17:33:11Z

@LorbusChris: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2	c315b5bca759bbc554a383e16e83c2b8f15e1ebf	link	false	`/test pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2`
pull-kubernetes-e2e-capz-windows-containerd	`d12696c`	link	false	`/test pull-kubernetes-e2e-capz-windows-containerd`
pull-kubernetes-e2e-gce-cos-alpha-features	`d12696c`	link	false	`/test pull-kubernetes-e2e-gce-cos-alpha-features`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

mrunalp · 2023-03-14T17:52:58Z

/lgtm
/approve
We discussed in sig-node today and are okay merging this.

k8s-ci-robot · 2023-03-14T17:53:07Z

LGTM label has been added.

Details

Git tree hash: 6358ad89a82d840117d4e133309f6681e000face

liggitt · 2023-03-14T19:55:51Z

/approve
kubelet config field / validation / defaults lgtm
approval of kubelet implementation from @mrunalp

k8s-ci-robot · 2023-03-14T19:56:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, LorbusChris, marosset, mrunalp, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [liggitt,soltysh]
~~pkg/features/OWNERS~~ [liggitt,marosset,soltysh]
~~pkg/generated/openapi/OWNERS~~ [liggitt]
~~pkg/kubelet/OWNERS~~ [liggitt,mrunalp]
~~pkg/kubelet/apis/config/OWNERS~~ [liggitt]
~~staging/src/k8s.io/kubelet/config/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aravindhp · 2023-03-14T20:22:28Z

/retest-required

marosset · 2023-03-14T20:29:31Z

/test pull-kubernetes-e2e-capz-windows-containerd

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 2, 2020

k8s-ci-robot requested review from andrewsykim and derekwaynecarr November 2, 2020 19:05

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 2, 2020

k8s-ci-robot requested review from smarterclayton and sttts November 3, 2020 09:37

LorbusChris changed the title ~~Expose a system journal shim on the kubelet logs endpoint on Linux and Windows~~ kubelet: Expose a system journal shim on the logs endpoint on Linux and Windows Nov 3, 2020

LorbusChris mentioned this pull request Nov 5, 2020

Bug 1862643: UPSTREAM: 96120: kubelet: Expose a simple Get-WinEvent shim on the kubelet logs endpoint openshift/kubernetes#383

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 5, 2020

k8s-ci-robot assigned rphillips Nov 5, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2020

k8s-ci-robot assigned sjenning Nov 5, 2020

LorbusChris force-pushed the kubelet-journal-logs branch from 725c3f3 to 9827bca Compare November 10, 2020 15:57

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 10, 2020

marosset approved these changes Mar 13, 2023

View reviewed changes

mrunalp reviewed Mar 13, 2023

View reviewed changes

aravindhp reviewed Mar 13, 2023

View reviewed changes

mrunalp reviewed Mar 13, 2023

View reviewed changes

aravindhp reviewed Mar 13, 2023

View reviewed changes

aravindhp and others added 4 commits March 14, 2023 08:45

feature: Add NodeLogQuery feature gate

d52cbd4

Enable the query endpoint only if this gate is enabled in addition to the EnableNodeLogQuery kubelet config option.

kubelet: Add validation for EnableNodeLogQuery

26279a5

aravindhp mentioned this pull request Mar 14, 2023

KEP-2258: Node log query documentation kubernetes/website#40003

Merged

openshift-ci-robot mentioned this pull request Mar 28, 2023

WIP: inject detach error openshift/kubernetes#1523

Closed

fabi200123 mentioned this pull request Mar 30, 2023

Add e2e tests for feature NodeLogQuery #117011

Merged

cartermckinnon mentioned this pull request Jul 19, 2023

REQUEST: New membership for cartermckinnon kubernetes/org#4338

Closed

9 tasks

jrvaldes mentioned this pull request Jun 3, 2026

REQUEST: New membership for jrvaldes kubernetes/org#6430

Closed

10 tasks

Conversation

LorbusChris commented Nov 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Nov 2, 2020

Uh oh!

LorbusChris commented Nov 3, 2020

Uh oh!

k8s-ci-robot commented Nov 3, 2020

Uh oh!

LorbusChris commented Nov 5, 2020

Uh oh!

rphillips commented Nov 5, 2020

Uh oh!

rphillips commented Nov 5, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aravindhp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aravindhp left a comment

LorbusChris commented Nov 2, 2020 •

edited

Loading