KEP 2258: add node log query #96120

LorbusChris · 2020-11-02T19:05:05Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Provide an administrator a streaming view of journal logs on Linux and WinEvent logs on Windows
without them having to implement a client side reader. Only available to cluster admins.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
Upstreaming from OpenShift:

Make viewing node logs from the CLI easy openshift/origin#21701
Bug 1862643: UPSTREAM: 96120: kubelet: Expose a simple Get-WinEvent shim on the kubelet logs endpoint openshift/kubernetes#383

Does this PR introduce a user-facing change?:

Adds feature gate `NodeLogQuery` which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

KEP: KEP 2258: Node service log viewer enhancements#2271
KEP Discussion: Node log query enhancements#2258
Usage: KEP-2258: Node log query documentation website#40003

k8s-ci-robot · 2020-11-02T19:05:14Z

Hi @LorbusChris. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LorbusChris · 2020-11-03T09:37:34Z

/cc @smarterclayton @sttts @aravindhp

k8s-ci-robot · 2020-11-03T09:37:36Z

@LorbusChris: GitHub didn't allow me to request PR reviews from the following users: aravindhp.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @smarterclayton @sttts @aravindhp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LorbusChris · 2020-11-05T18:31:18Z

Somebody please give this an ok-to-test

rphillips · 2020-11-05T20:24:40Z

/ok-to-test
/lgtm

rphillips · 2020-11-05T20:26:35Z

/assign @sjenning

mrunalp · 2023-03-13T22:35:52Z

pkg/kubelet/kubelet_server_journal.go

+func safeString(s string) error {
+	// Max length of a service name is 256 across supported OSes
+	if len(s) > 256 {
+		return fmt.Errorf("length must be less than 100")


100 -> 256? Maybe a constant for it :)

mrunalp · 2023-03-13T22:38:18Z

pkg/kubelet/kubelet_server_journal_linux.go

+		// Returning false to allow checking if the service is logging to a file
+		return false
+	}
+	// journalctl will not return an error even if the service is not using journald and hence we search the output for


journalctl won't return an error if we try to fetch logs for a non-existent service, hence we search for it in the list of services known to journalctl

aravindhp

@mrunalp I have addressed your comments. PTAL.

aravindhp · 2023-03-13T23:23:21Z

pkg/kubelet/kubelet_server_journal.go

+func safeString(s string) error {
+	// Max length of a service name is 256 across supported OSes
+	if len(s) > 256 {
+		return fmt.Errorf("length must be less than 100")


aravindhp · 2023-03-13T23:23:58Z

pkg/kubelet/kubelet_server_journal_linux.go

+		// Returning false to allow checking if the service is logging to a file
+		return false
+	}
+	// journalctl will not return an error even if the service is not using journald and hence we search the output for


Enable the query endpoint only if this gate is enabled in addition to the EnableNodeLogQuery kubelet config option.

Added EnableNodeLogQuery field to kubelet/apis/config/types.go and staging/src/k8s.io/kubelet/config/v1beta1/types.go, then executed. `hack/update-codegen.sh`. This new field will default to off and will need to be explicitly enabled in addition to the NodeLogQuery gate to use the feature.

…dpoint Provide an administrator a streaming view of journal logs on Linux systems using journalctl, and event logs on Windows systems using the Get-WinEvent PowerShell cmdlet without them having to implement a client side reader. Only available to cluster admins. The implementation for journald on Linux was originally done by Clayton Coleman. Introduce a heuristics approach to query logs The logs query for node objects will follow a heuristics approach when asked to query for logs from a service. If asked to get the logs from a service foobar, it will first check if foobar logs to the native OS service log provider. If unable to get logs from these, it will attempt to get logs from /var/foobar, /var/log/foobar.log or /var/log/foobar/foobar.log in that order. The logs sub-command can also directly serve a file if the query looks like a file. Co-authored-by: Clayton Coleman <ccoleman@redhat.com> Co-authored-by: Christian Glombek <cglombek@redhat.com>

k8s-ci-robot · 2023-03-14T17:33:11Z

@LorbusChris: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2	c315b5bca759bbc554a383e16e83c2b8f15e1ebf	link	false	`/test pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2`
pull-kubernetes-e2e-capz-windows-containerd	`d12696c`	link	false	`/test pull-kubernetes-e2e-capz-windows-containerd`
pull-kubernetes-e2e-gce-cos-alpha-features	`d12696c`	link	false	`/test pull-kubernetes-e2e-gce-cos-alpha-features`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

mrunalp · 2023-03-14T17:52:58Z

/lgtm
/approve
We discussed in sig-node today and are okay merging this.

k8s-ci-robot · 2023-03-14T17:53:07Z

LGTM label has been added.

Git tree hash: 6358ad89a82d840117d4e133309f6681e000face

liggitt · 2023-03-14T19:55:51Z

/approve
kubelet config field / validation / defaults lgtm
approval of kubelet implementation from @mrunalp

k8s-ci-robot · 2023-03-14T19:56:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, LorbusChris, marosset, mrunalp, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [liggitt,soltysh]
~~pkg/features/OWNERS~~ [liggitt,marosset,soltysh]
~~pkg/generated/openapi/OWNERS~~ [liggitt]
~~pkg/kubelet/OWNERS~~ [liggitt,mrunalp]
~~pkg/kubelet/apis/config/OWNERS~~ [liggitt]
~~staging/src/k8s.io/kubelet/config/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aravindhp · 2023-03-14T20:22:28Z

/retest-required

marosset · 2023-03-14T20:29:31Z

/test pull-kubernetes-e2e-capz-windows-containerd

Enhancement: https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/2258-node-service-log-viewer Feature PR: kubernetes/kubernetes#96120

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 2, 2020

k8s-ci-robot requested review from andrewsykim and derekwaynecarr November 2, 2020 19:05

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 2, 2020

k8s-ci-robot requested review from smarterclayton and sttts November 3, 2020 09:37

LorbusChris changed the title ~~Expose a system journal shim on the kubelet logs endpoint on Linux and Windows~~ kubelet: Expose a system journal shim on the logs endpoint on Linux and Windows Nov 3, 2020

LorbusChris mentioned this pull request Nov 5, 2020

Bug 1862643: UPSTREAM: 96120: kubelet: Expose a simple Get-WinEvent shim on the kubelet logs endpoint openshift/kubernetes#383

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 5, 2020

k8s-ci-robot assigned rphillips Nov 5, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2020

k8s-ci-robot assigned sjenning Nov 5, 2020

LorbusChris force-pushed the kubelet-journal-logs branch from 725c3f3 to 9827bca Compare November 10, 2020 15:57

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 10, 2020

mrunalp reviewed Mar 13, 2023

View reviewed changes

aravindhp force-pushed the kubelet-journal-logs branch from 5cd43c6 to 93e76b1 Compare March 13, 2023 23:22

aravindhp reviewed Mar 13, 2023

View reviewed changes

aravindhp and others added 4 commits March 14, 2023 08:45

feature: Add NodeLogQuery feature gate

d52cbd4

Enable the query endpoint only if this gate is enabled in addition to the EnableNodeLogQuery kubelet config option.

kubelet: Add validation for EnableNodeLogQuery

26279a5

aravindhp force-pushed the kubelet-journal-logs branch from 93e76b1 to d12696c Compare March 14, 2023 15:57

k8s-ci-robot assigned mrunalp Mar 14, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 14, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 14, 2023

aravindhp mentioned this pull request Mar 14, 2023

KEP-2258: Node log query documentation kubernetes/website#40003

Merged

k8s-ci-robot merged commit 89a9c0c into kubernetes:master Mar 14, 2023

openshift-ci-robot mentioned this pull request Mar 28, 2023

WIP: inject detach error openshift/kubernetes#1523

Closed

fabi200123 mentioned this pull request Mar 30, 2023

Add e2e tests for feature NodeLogQuery #117011

Merged

aravindhp added a commit to aravindhp/website that referenced this pull request Apr 3, 2023

Add blog about Node Log Query alpha

86aec53

Enhancement: https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/2258-node-service-log-viewer Feature PR: kubernetes/kubernetes#96120

aravindhp added a commit to aravindhp/website that referenced this pull request Apr 4, 2023

Add blog about Node Log Query alpha

ba0a13c

Enhancement: https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/2258-node-service-log-viewer Feature PR: kubernetes/kubernetes#96120

aravindhp added a commit to aravindhp/website that referenced this pull request Apr 11, 2023

Add blog about Node Log Query alpha

4a658ee

Enhancement: https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/2258-node-service-log-viewer Feature PR: kubernetes/kubernetes#96120

DonatoHorn pushed a commit to DonatoHorn/website that referenced this pull request Jun 25, 2023

Add blog about Node Log Query alpha

a8c33bd

Enhancement: https://github.com/kubernetes/enhancements/tree/master/keps/sig-windows/2258-node-service-log-viewer Feature PR: kubernetes/kubernetes#96120

cartermckinnon mentioned this pull request Jul 19, 2023

REQUEST: New membership for cartermckinnon kubernetes/org#4338

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP 2258: add node log query #96120

KEP 2258: add node log query #96120

LorbusChris commented Nov 2, 2020 •

edited

Loading

k8s-ci-robot commented Nov 2, 2020

LorbusChris commented Nov 3, 2020

k8s-ci-robot commented Nov 3, 2020

LorbusChris commented Nov 5, 2020

rphillips commented Nov 5, 2020

rphillips commented Nov 5, 2020

mrunalp Mar 13, 2023

aravindhp Mar 13, 2023

mrunalp Mar 13, 2023

aravindhp Mar 13, 2023

aravindhp left a comment

aravindhp Mar 13, 2023

aravindhp Mar 13, 2023

k8s-ci-robot commented Mar 14, 2023

mrunalp commented Mar 14, 2023

k8s-ci-robot commented Mar 14, 2023

liggitt commented Mar 14, 2023

k8s-ci-robot commented Mar 14, 2023

aravindhp commented Mar 14, 2023

marosset commented Mar 14, 2023

KEP 2258: add node log query #96120

KEP 2258: add node log query #96120

Conversation

LorbusChris commented Nov 2, 2020 • edited Loading

k8s-ci-robot commented Nov 2, 2020

LorbusChris commented Nov 3, 2020

k8s-ci-robot commented Nov 3, 2020

LorbusChris commented Nov 5, 2020

rphillips commented Nov 5, 2020

rphillips commented Nov 5, 2020

mrunalp Mar 13, 2023

Choose a reason for hiding this comment

aravindhp Mar 13, 2023

Choose a reason for hiding this comment

mrunalp Mar 13, 2023

Choose a reason for hiding this comment

aravindhp Mar 13, 2023

Choose a reason for hiding this comment

aravindhp left a comment

Choose a reason for hiding this comment

aravindhp Mar 13, 2023

Choose a reason for hiding this comment

aravindhp Mar 13, 2023

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 14, 2023

mrunalp commented Mar 14, 2023

k8s-ci-robot commented Mar 14, 2023

liggitt commented Mar 14, 2023

k8s-ci-robot commented Mar 14, 2023

aravindhp commented Mar 14, 2023

marosset commented Mar 14, 2023

LorbusChris commented Nov 2, 2020 •

edited

Loading