Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured logging #1602

Open
39 tasks done
serathius opened this issue Mar 6, 2020 · 117 comments
Open
39 tasks done

Structured logging #1602

serathius opened this issue Mar 6, 2020 · 117 comments
Assignees
Labels
sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging.

Comments

@serathius
Copy link
Contributor

serathius commented Mar 6, 2020

Enhancement Description

Define standard structure for Kubernetes log messages, add methods to klog to enforce this structure, add ability to configure Kubernetes components to produce logs in JSON format and initiate migration to structured logging.

  • Kubernetes Enhancement Proposal: Structured Logging KEP
  • Primary contact (assignee): @serathius
  • Responsible SIGs: sig-instrumentation
  • Enhancement target (which target equals to which milestone):
    • Alpha release target 1.19
    • Beta release target 1.23
    • GA release target 1.27
Alpha

Follow-up from main tasks:

Beta

Follow ups:

Please let me know if someone is interested in working on one of the tasks.

@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Mar 6, 2020
@serathius
Copy link
Contributor Author

serathius commented Mar 6, 2020

/sig instrumentation

@k8s-ci-robot k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 6, 2020
@serathius
Copy link
Contributor Author

serathius commented Mar 14, 2020

/help

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Mar 14, 2020

@serathius:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Mar 14, 2020
@yuzhiquan
Copy link
Member

yuzhiquan commented Mar 14, 2020

interested in

@tanjunchen
Copy link
Member

tanjunchen commented Mar 16, 2020

/cc

@serathius
Copy link
Contributor Author

serathius commented Mar 16, 2020

/milestone v1.19

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Mar 16, 2020

@serathius: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@justaugustus
Copy link
Member

justaugustus commented Mar 16, 2020

/milestone v1.19

@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Mar 16, 2020
@justaugustus justaugustus removed the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Mar 16, 2020
@hase1128
Copy link

hase1128 commented Mar 17, 2020

/cc

@serathius
Copy link
Contributor Author

serathius commented Mar 17, 2020

KEP was marked as "implementable", so we can start working on it.
I have created first two subtasks within specific repositories. I will add more as work progresses and new tasks are unblocked.

If your interested in contributing please volunteer directly on subtasks or ask for specific task here.

@gavinfish
Copy link
Member

gavinfish commented Mar 18, 2020

/cc

@serathius
Copy link
Contributor Author

serathius commented Apr 8, 2020

After discussing offline with @dims we concluded that upgrading klog will be non-trivial as we need to upgrade all references at once. Upgrade cannot be done partially as it would break klog configuration.

We considered two options:

  • [Preferred] Properly update of all dependencies
  • Aliasing in go modules

As we have a lot of time before 1.19 cut we would like to do it properly. I will create subtask with upgrade plan.

@fenggw-fnst
Copy link

fenggw-fnst commented Apr 9, 2020

/cc

@johnbelamaric
Copy link
Contributor

johnbelamaric commented May 4, 2020

@serathius Enhancements shadow for v1.19 here. Do you see this as moving to alpha in 1.19?

@serathius
Copy link
Contributor Author

serathius commented May 5, 2020

Yes, I think we are on track. We are finishing klog upgrade, after that next milestones will be really fast to implement.

@johnbelamaric johnbelamaric added the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label May 5, 2020
@pohly
Copy link
Contributor

pohly commented Nov 18, 2021

Docs PR for 1.23 is here: kubernetes/website#30505

@mehabhalodiya
Copy link

mehabhalodiya commented Nov 18, 2021

@Priyankasaggu11929 Can you please add this to 1.23 milestone?
Also, please add it to the sheet.
Thanks!

@reylejano
Copy link
Member

reylejano commented Nov 18, 2021

Hi @pohly,
1.23 Release Lead here.
This enhancement wasn't opted-in for the 1.23 release and through the linked issue kubernetes/kubernetes#104872, it looks like the following k/k PRs merged for the 1.23 release:

Looks like the following open k/k PRs targets v1.24:

The KEP states that this was beta in 1.22 but the documentation PR to the dev-1.23 in the k/website PR states that its beta in 1.23.

Is the doc PR correct that this went to beta in 1.23 and not 1.22?

With Enhancements Freeze and Code Freeze in effect for 1.23, if there was a graduation then this enhancement would need an exception request filed asap

@pohly
Copy link
Contributor

pohly commented Nov 18, 2021

I only joined the WG structured logging after the 1.23 cycle had already started. The intention certainly was to continue with the feature in 1.23 and I was under the impression that it had been tracked for that release. Apparently that wasn't the case.

To answer your question, yes, all the work required for beta is completed and merged, so the docs PR is correct.

Why do we still need an exception request if everything was merged already?

@reylejano
Copy link
Member

reylejano commented Nov 18, 2021

@pohly
In order to track the graduation of this enhancement to beta related to the 1.23 release, this enhancement would need an exception request for 1.23 and the KEP would need to be updated to reflect the current state of this enhancement

@serathius
Copy link
Contributor Author

serathius commented Nov 18, 2021

Technically we marked Structured Logging as beta in 1.22 https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1602-structured-logging/kep.yaml#L25-L26 there were some points still left, but release team has asked us to mark it as Beta in 1.22.

@reylejano Issues you linked as not part work for KEP. Structured Logging KEP was proposal to initiate migration, however migration itself is to large effort to be tracked as enhancement that can happen in one release (k8s has 10'000 log lines, we barely are able to do 5-10% each release) . For 1.23 we migrated 2 components, but this was not part of the work on Beta, just part of work do by WG.

I don't think there is a reason to file an exception.

@ehashman
Copy link
Member

ehashman commented Nov 18, 2021

@serathius per #1602 (comment) this missed 1.22. I was under the impression that we had retargeted beta to 1.23.

I am supportive of an exception request to ensure that this work (already merged and mostly documented) can graduate in 1.23.

I think there was some miscommunication; when SIG Instrumentation solicited KEPs for the 1.23 cycle, our internal tracking sheet says that the only SIG needed for 1.23 was #2845 and the status of this KEP was pushed to WG Logging. But if work continued to push this towards beta goals, even if the KEP didn't need to be updated, the metadata for #1602 should have been updated so the RT could include it in the release tracking.

@logicalhan
Copy link
Contributor

logicalhan commented Nov 18, 2021

@serathius per #1602 (comment) this missed 1.22. I was under the impression that we had retargeted beta to 1.23.

I am supportive of an exception request to ensure that this work (already merged and mostly documented) can graduate in 1.23.

I think there was some miscommunication; when SIG Instrumentation solicited KEPs for the 1.23 cycle, our internal tracking sheet says that the only SIG needed for 1.23 was #2845 and the status of this KEP was pushed to WG Logging. But if work continued to push this towards beta goals, even if the KEP didn't need to be updated, the metadata for #1602 should have been updated so the RT could include it in the release tracking.

+1

@reylejano
Copy link
Member

reylejano commented Nov 18, 2021

The Release Team received an exception request for this enhancement for 1.23. The Release Team APPROVE the exception request.

/milestone v1.23

@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Nov 18, 2021
@reylejano reylejano added tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team and removed tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Nov 18, 2021
@gracenng gracenng added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Jan 9, 2022
@gracenng gracenng removed this from the v1.23 milestone Jan 9, 2022
@yangjunmyfm192085
Copy link

yangjunmyfm192085 commented Jan 14, 2022

/cc

@serathius serathius added the wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging. label Feb 14, 2022
@logicalhan
Copy link
Contributor

logicalhan commented May 12, 2022

/assign @serathius

To make more detailed milestones for GA release (i.e. scheduler migration and the like).

@k8s-triage-robot
Copy link

k8s-triage-robot commented Aug 10, 2022

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 10, 2022
@pohly
Copy link
Contributor

pohly commented Aug 11, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 11, 2022
@pohly
Copy link
Contributor

pohly commented Oct 27, 2022

At this point the plan is to convert code directly to contextual logging, so see there for progress tracking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging.
Projects
None yet
Development

No branches or pull requests