contextual logging #3077

pohly · 2021-12-06T10:32:26Z

Enhancement Description

Current configuration

https://github.com/kubernetes/kubernetes/blob/master/hack/logcheck.conf

Status

The following table counts log calls that need to be converted. The numbers for contextual logging include those for structured logging.

At this point, controllers could get converted to contextual logging or one of the components that was already converted to structured logging. If you want to pick one, ping @pohly on the #wg-structured-logging Slack channel. See structured and contextual logging migration instructions for guidance.

Besides migrating log calls, we also might have to migrate from APIs which don't support contextual logging to APIs which do:

k8s.io/client-go/tools/[events|record]: add WithContext
- cronjob controller: reconsider log verbosity kubernetes#115574
HandlePanic/HandleError -> HandlePanicWithContext, HandleErrorWithContext

From 2022-10-27 ~= Kubernetes 1.26

The focus was on converting kube-controller-manager. Of 1944 unstructured and/or non-contexual logging calls in pkg/controller and cmd/kube-controller-manager, 82% were converted to structured, contextual logging in Kubernetes 1.27.

Component	Non-Structured Logging	Non-Contextual Logging	Owner
pkg/controller/bootstrap	15	28	@mengjiao-liu, kubernetes/kubernetes#113464
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/clusterroleaggregation	2	2	@mengjiao-liu, kubernetes/kubernetes#113910
pkg/controller/cronjob	1	44	@mengjiao-liu, kubernetes/kubernetes#113428
pkg/controller/daemon	45	85	@249043822, kubernetes/kubernetes#113622
pkg/controller/deployment	23	79	@249043822, kubernetes/kubernetes#113525
pkg/controller/disruption	29	56	@Namanl2001, kubernetes/kubernetes#116021
pkg/controller/endpoint	12	24	lunhuijie (Slack)
pkg/controller/endpointslice	22	36	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	55	105	@ncdc, kubernetes/kubernetes#113471
pkg/controller/job	12	36	was: @sanwishe, kubernetes/kubernetes#113576, now: @mengjiao-liu
pkg/controller/namespace	30	55	@yangjunmyfm192085, kubernetes/kubernetes#113443
pkg/controller/nodeipam	135	210	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/nodelifecycle	60	106	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/podautoscaler	9	13	@freddie400, kubernetes/kubernetes#114687
pkg/controller/podgc	10	24	@pravarag, kubernetes/kubernetes#114689
pkg/controller/replicaset	20	49	@Namanl2001, kubernetes/kubernetes#114871
pkg/controller/resourcequota	24	37	@ncdc, kubernetes/kubernetes#113315
pkg/controller/serviceaccount	22	31	@Namanl2001, kubernetes/kubernetes#114918
pkg/controller/statefulset	19	59	@249043822, kubernetes/kubernetes#113840
pkg/controller/storageversiongc	4	6	@songxiao-wang87, kubernetes/kubernetes#113986
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/ttl	4	8	wxs (Slack) = @songxiao-wang87, kubernetes/kubernetes#113916
pkg/controller/ttlafterfinished	9	15	@obaranov1, kubernetes/kubernetes#115332
pkg/controller/util	0	19	@fatsheep9146, kubernetes/kubernetes#115049
pkg/controller/volume	351	673	@yangjunmyfm192085, kubernetes/kubernetes#113584
pkg/kubelet	1	1805	@fmuyassarov
pkg/scheduler	0	348	@knelasevero, kubernetes/kubernetes#111155
staging/src/k8s.io/apiextensions-apiserver	57	81
staging/src/k8s.io/apimachinery	73	114	@yanjing1104
staging/src/k8s.io/apiserver	262	543
staging/src/k8s.io/client-go	161	267
staging/src/k8s.io/cloud-provider	108	146
staging/src/k8s.io/cluster-bootstrap	2	4
staging/src/k8s.io/code-generator	108	168
staging/src/k8s.io/component-base	32	63
staging/src/k8s.io/component-helpers	7	8
staging/src/k8s.io/controller-manager	10	10
staging/src/k8s.io/csi-translation-lib	3	4
staging/src/k8s.io/kube-aggregator	52	76
staging/src/k8s.io/kube-controller-manager	0	0
staging/src/k8s.io/kubectl	89	147	@yanjing1104
staging/src/k8s.io/legacy-cloud-providers	1445	2238
staging/src/k8s.io/mount-utils	54	92
staging/src/k8s.io/pod-security-admission	1	34	@Namanl2001, kubernetes/kubernetes#114471
staging/src/k8s.io/sample-controller	16	22	@pchan, kubernetes/kubernetes#113879

From 2023-03-17 = Kubernetes v1.27.0-beta.0

All of kube-controller-manager got converted.

Tables created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- | ------" && \
for i in $(find pkg/controller/* pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet  -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	7	8	on hold
cmd/kubelet	0	47	@fmuyassarov
cmd/kube-proxy	0	46	on hold
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/deployment	2	5	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/disruption	29	54	~~@obaranov1, kubernetes/kubernetes#116021~~, @mengjiao-liu, kubernetes/kubernetes#119147
pkg/controller/endpoint	12	24	@my-git9, kubernetes/kubernetes#116755
pkg/controller/endpointslice	20	35	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	3	3	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/job	12	35	@sanwishe, kubernetes/kubernetes#113576 (needs new owner?)
pkg/controller/nodeipam	8	13	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/podgc	10	24	~~@pravarag, kubernetes/kubernetes#114689~~, @pohly, kubernetes/kubernetes#119250
pkg/controller/replicaset	9	18	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/statefulset	3	5	@kerthcet, kubernetes/kubernetes#118071
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/util	0	4	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/volume	5	20	@fatsheep9146, kubernetes/kubernetes#116930
pkg/kubelet	2	1923	@fmuyassarov, kubernetes/kubernetes#114352
pkg/scheduler	2	349	@mengjiao-liu, kubernetes/kubernetes#91633

From 2023-09-18 =~ Kubernetes v1.28

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	on hold
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
cmd/kube-proxy	0	41	on hold
pkg/kubelet	2	1942	@fmuyassarov (?)
pkg/scheduler	1	137	@mengjiao-liu, https://github.com/kubernetes/kubernetes/pulls/mengjiao-liu
staging/src/k8s.io/apiserver	?	?	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21	on hold
staging/src/k8s.io/client-go/examples	14	14	on hold
staging/src/k8s.io/client-go/metadata	2	4	on hold
staging/src/k8s.io/client-go/plugin	5	8	on hold
staging/src/k8s.io/client-go/rest	16	37	on hold
staging/src/k8s.io/client-go/restmapper	3	6	on hold
staging/src/k8s.io/client-go/tools	104	171	@pohly, kubernetes/kubernetes#120729
staging/src/k8s.io/client-go/transport	17	31	on hold
staging/src/k8s.io/client-go/util	12	19	on hold

Table created manually and with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- |  ------- | ------" && \
for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

From 2023-11-20 =~ Kubernetes v1.29

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	@tallclair
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
pkg/kubelet	2	1983	@fmuyassarov
cmd/kube-proxy	0	42	@ fatsheep9146
pkg/proxy	0	360	@ fatsheep9146
staging/src/k8s.io/apiserver	285	655	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21
staging/src/k8s.io/client-go/examples	14	14
staging/src/k8s.io/client-go/metadata	2	4
staging/src/k8s.io/client-go/plugin	5	8
staging/src/k8s.io/client-go/rest	16	37
staging/src/k8s.io/client-go/restmapper	3	6
staging/src/k8s.io/client-go/tools	83	143
staging/src/k8s.io/client-go/transport	17	31
staging/src/k8s.io/client-go/util	12	19

Table created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && echo "------ | ------- |  ------- | ------" && for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* staging/src/k8s.io/apiserver -maxdepth 0 -type d | sort); do      echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; done

The text was updated successfully, but these errors were encountered:

pohly · 2021-12-08T12:37:28Z

/sig instrumentation
/wg structured-logging

hosseinsalahi · 2022-01-21T10:03:34Z

Hello @pohly

v1.24 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,

Here’s where this enhancement currently stands:

Updated KEP file using the latest template has been merged into the k/enhancements repo.
KEP status is marked as implementable for this release
KEP has a test plan section filled out.
KEP has up to date graduation criteria.
KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as tracked. Please keep the issue description and the targeted stage up-to-date for release v1.24.
Thanks!

pohly · 2022-02-03T06:44:26Z

@encodeflush : the KEP PR was merged, all criteria for alpha in 1.24 should be met now.

chrisnegus · 2022-02-11T18:51:41Z

Hi @pohly 👋 1.24 Docs shadow here.

This enhancement is marked as 'Needs Docs' for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

valaparthvi · 2022-03-21T04:13:22Z

Hi @pohly 👋 1.24 Release Comms team here.

We have an opt-in process for the feature blog delivery. If you would like to publish a feature blog for this issue in this cycle, then please opt in on this tracking sheet.

The deadline for submissions and the feature blog freeze is scheduled for 01:00 UTC Wednesday 23rd March 2022 / 18:00 PDT Tuesday 22nd March 2022. Other important dates for delivery and review are listed here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24#timeline.

For reference, here is the blog for 1.23.

Please feel free to reach out any time to me or on the #release-comms channel with questions or comments.

Thanks!

hosseinsalahi · 2022-03-21T17:13:46Z

Hello @pohly

I'm just checking in once more as we approach the 1.24 Code Freeze on 18:00 PDT, Tuesday, March 29th 2022

Please ensure the following items are completed:

All PRs to the Kubernetes repo that are related to your enhancement are merged by the code freeze deadline.
Have a documentation placeholder PR open by 18:00 PDT, Thursday, March 31, 2022.
- features: add ContextualLogging website#32452
- logging: include information about contextual logging community#6560

For note, the status of this enhancement is currently marked as tracked.

Thank you!

pohly · 2022-03-23T16:16:49Z

/assign

pohly · 2022-03-23T16:48:29Z

I have added two doc PRs to the description.

Priyankasaggu11929 · 2022-05-10T04:54:32Z

/milestone clear

logicalhan · 2022-05-12T16:38:58Z

@pohly can we close this?

logicalhan · 2022-05-12T16:40:00Z

/assign @serathius

atiratree · 2023-02-15T23:22:00Z

KCM / CCM controller aliases should help with consistent component names (eventually once we do the wiring)

kubernetes/kubernetes#115813

gurpreet-legend · 2023-03-23T10:32:13Z

I would really like to work on the component pkg/controller/deployment, can someone assign this to me?

pohly · 2023-03-23T10:53:20Z

Can you also do pkg/controller/endpoint, pkg/controller/garbagecollector, pkg/controller/nodeipam, pkg/controller/replicaset, pkg/controller/statefulset, pkg/controller/util, pkg/controller/volume?

That can be a single cleanup PR because those components were already converted earlier.

gurpreet-legend · 2023-03-23T11:34:34Z

Can you also do pkg/controller/endpoint, pkg/controller/garbagecollector, pkg/controller/nodeipam, pkg/controller/replicaset, pkg/controller/statefulset, pkg/controller/util, pkg/controller/volume?

That can be a single cleanup PR because those components were already converted earlier.

@pohly, will be happy to work on those as well :)

pohly · 2023-03-23T11:41:46Z

@gurpreet-legend: I added you to the table.

It looks like pkg/controller/endpoint hasn't been touched at all yet. Then do that in one PR and the cleanup changes in another.

sadityakumar9211 · 2023-03-23T12:21:33Z

I am really interested in working on cmd/kube-proxy component. Can someone please assign this to me?

GLVSKiriti · 2023-03-23T12:56:17Z

I would like to work on this component: cmd/kube-apiserver .Can someone assign this to me.

esthejas · 2023-03-23T13:50:09Z

I would really like to work on this component cmd/kube-apiserver.Can someone please assign this to me

gurpreet-legend · 2023-03-23T14:53:24Z

@gurpreet-legend: I added you to the table.

It looks like pkg/controller/endpoint hasn't been touched at all yet. Then do that in one PR and the cleanup changes in another.

Sure, will work on that first and then will work on the other components :)

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Dec 6, 2021

pohly mentioned this issue Dec 6, 2021

KEP-3077: contextual logging #3078

Merged

k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 8, 2021

ehashman mentioned this issue Jan 14, 2022

New unique logging meta-data into key Kubernetes logs #1668

Closed

gracenng added the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label Jan 17, 2022

gracenng added this to the v1.24 milestone Jan 17, 2022

This was referenced Feb 14, 2022

KEP 3077: contextual logging: simplify scope, avoid transition #3222

Merged

contextual logging kubernetes/klog#296

Merged

logcheck: contextual logging + enhanced checks kubernetes/klog#297

Merged

k8s-ci-robot assigned pohly Mar 23, 2022

This was referenced Mar 24, 2022

contextual logging kubernetes/kubernetes#108995

Merged

contextual logging blog post kubernetes/website#32656

Merged

mehabhalodiya mentioned this issue Apr 4, 2022

features: add ContextualLogging kubernetes/website#32452

Merged

github-actions bot mentioned this issue May 6, 2022

Blog: Kubernetes 1.24: Stargazer rajula96reddy/rss-issue-action-test#18

Open

k8s-ci-robot removed this from the v1.24 milestone May 10, 2022

Priyankasaggu11929 added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels May 10, 2022

This was referenced Feb 19, 2023

migrating component-helpers kubernetes/kubernetes#115875

Closed

[WIP] migrating csi-translation-lib kubernetes/kubernetes#115885

Open

[WIP] migrate-disruption kubernetes/kubernetes#115886

Open

oxxenix mentioned this issue Feb 23, 2023

Migrate /pkg/controller/disruption to structured and contextual logging kubernetes/kubernetes#116021

Closed

knelasevero mentioned this issue Mar 13, 2023

scheduler: contextual logging - WithName and WithValue kubernetes/kubernetes#111155

Closed

This was referenced Mar 24, 2023

Migrated pkg/controller/job to contextual logging kubernetes/kubernetes#116910

Merged

contextual logging cleanup kubernetes/kubernetes#116930

Merged

Atharva-Shinde removed the tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team label May 14, 2023

mengjiao-liu mentioned this issue Jul 7, 2023

Migrate /pkg/controller/disruption to structured and contextual logging kubernetes/kubernetes#119147

Merged

pohly mentioned this issue Jul 12, 2023

kube-controller-manager: finish conversion to contextual logging kubernetes/kubernetes#119250

Merged

dashpole mentioned this issue Jul 31, 2023

Trace and span IDs in apiserver logs using contextual logging kubernetes/kubernetes#119699

Open

mengjiao-liu mentioned this issue Aug 14, 2023

log: get logger from context if existed kubernetes/kubernetes#119916

Closed

pohly mentioned this issue Sep 18, 2023

KEP-3077: contextual logging: promotion to beta in 1.30 #4219

Open

mengjiao-liu mentioned this issue Oct 19, 2023

feature blog: contextual logging update for 1.29 kubernetes/website#43579

Open

pohly mentioned this issue Oct 23, 2023

output: handle WithName like zapr does kubernetes/klog#391

Merged

mengjiao-liu mentioned this issue Nov 14, 2023

[WIP]Add blog for Contextual logging in Kubernetes v1.29 kubernetes/website#43925

Closed

This was referenced Nov 21, 2023

WIP: convert client-go/tools/cache to contextual logging kubernetes/kubernetes#121989

Open

contextual logging: move key/value args into WithValues kubernetes/kubernetes#121998

Open

contextual logging #3077

contextual logging #3077

pohly commented Dec 6, 2021 •

edited

pohly commented Dec 8, 2021

hosseinsalahi commented Jan 21, 2022 •

edited

pohly commented Feb 3, 2022

chrisnegus commented Feb 11, 2022

valaparthvi commented Mar 21, 2022 •

edited

hosseinsalahi commented Mar 21, 2022 •

edited

pohly commented Mar 23, 2022

pohly commented Mar 23, 2022

Priyankasaggu11929 commented May 10, 2022

logicalhan commented May 12, 2022

logicalhan commented May 12, 2022

atiratree commented Feb 15, 2023

gurpreet-legend commented Mar 23, 2023

pohly commented Mar 23, 2023

gurpreet-legend commented Mar 23, 2023

pohly commented Mar 23, 2023

sadityakumar9211 commented Mar 23, 2023

GLVSKiriti commented Mar 23, 2023 •

edited

esthejas commented Mar 23, 2023

gurpreet-legend commented Mar 23, 2023

contextual logging #3077

contextual logging #3077

Comments

pohly commented Dec 6, 2021 • edited

Enhancement Description

Current configuration

Status

From 2022-10-27 ~= Kubernetes 1.26

From 2023-03-17 = Kubernetes v1.27.0-beta.0

From 2023-09-18 =~ Kubernetes v1.28

From 2023-11-20 =~ Kubernetes v1.29

pohly commented Dec 8, 2021

hosseinsalahi commented Jan 21, 2022 • edited

pohly commented Feb 3, 2022

chrisnegus commented Feb 11, 2022

valaparthvi commented Mar 21, 2022 • edited

hosseinsalahi commented Mar 21, 2022 • edited

pohly commented Mar 23, 2022

pohly commented Mar 23, 2022

Priyankasaggu11929 commented May 10, 2022

logicalhan commented May 12, 2022

logicalhan commented May 12, 2022

atiratree commented Feb 15, 2023

gurpreet-legend commented Mar 23, 2023

pohly commented Mar 23, 2023

gurpreet-legend commented Mar 23, 2023

pohly commented Mar 23, 2023

sadityakumar9211 commented Mar 23, 2023

GLVSKiriti commented Mar 23, 2023 • edited

esthejas commented Mar 23, 2023

gurpreet-legend commented Mar 23, 2023

pohly commented Dec 6, 2021 •

edited

hosseinsalahi commented Jan 21, 2022 •

edited

valaparthvi commented Mar 21, 2022 •

edited

hosseinsalahi commented Mar 21, 2022 •

edited

GLVSKiriti commented Mar 23, 2023 •

edited