Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to disable writing k8s events #13042

Open
tooptoop4 opened this issue May 13, 2024 · 3 comments
Open

Option to disable writing k8s events #13042

tooptoop4 opened this issue May 13, 2024 · 3 comments
Labels
area/controller Controller issues, panics type/feature Feature request

Comments

@tooptoop4
Copy link
Contributor

Summary

Environment variable/config to stop writing k8s events

Use Cases

When would you use this? When i want to stop etcd filling up

These parts of the code are writing k8s events which end up being stored in etcd:

case wfv1.WorkflowRunning:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeNormal, "WorkflowRunning", "Workflow Running")
case wfv1.WorkflowSucceeded:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeNormal, "WorkflowSucceeded", "Workflow completed")
case wfv1.WorkflowFailed, wfv1.WorkflowError:
woc.eventRecorder.Event(woc.wf, apiv1.EventTypeWarning, "WorkflowFailed", message)

woc.recordNodePhaseChangeEvents(woc.orig.Status.Nodes, woc.wf.Status.Nodes)

With a lot of workflows being churned through these events add a lot of storage pressure to etcd. At the same time i have not found these events useful as i can always look at logs in the controller pod

@tooptoop4 tooptoop4 added the type/feature Feature request label May 13, 2024
@terrytangyuan
Copy link
Member

terrytangyuan commented May 13, 2024

# Whether or not to emit events on node completion. These can take a up a lot of space in
# k8s (typically etcd) resulting in errors when trying to create new events:
# "Unable to create audit event: etcdserver: mvcc: database space exceeded"
# This config item allows you to disable this.
# (since v2.9)
nodeEvents: |
enabled: true

@terrytangyuan

This comment was marked as duplicate.

@tooptoop4
Copy link
Contributor Author

tooptoop4 commented May 13, 2024

@terrytangyuan i already have that as false but it still sends events!! seems i am getting events at workflow level

@agilgur5 agilgur5 added the area/controller Controller issues, panics label May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/controller Controller issues, panics type/feature Feature request
Projects
None yet
Development

No branches or pull requests

3 participants