You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, if worker/scheduler pods run for long periods of time, they may accumulate so many logs under logs.path that they may cause issues.
We should allow users to deploy a sidecar that is responsible to truncate log files after some period of time.
Here are some example values for what this feature may look like:
scheduler:
logCleanup:
# if the sidecar container is added to the scheduler Pod (default: true)enabled: true# resources for the ContainerSpecresources: {}## the number of minutes to retain log files (by last-modified time)##retentionMinutes: 21600## the number of seconds between each check for files to delete##intervalSeconds: 900workers:
logCleanup:
## SAME AS ABOVE
WARNING: we should NOT allow {scheduler,workers}.logCleanup.enabled and logs.persistence.enabled to be true at the same time (using ./_helpers/validate-values.tpl) as multiple of the sidecars may fight with each other. In the error message, tell users that they could create an airflow job instead.
The text was updated successfully, but these errors were encountered:
This issue is related to memory usage increasing over time when our scheduler liveness probe is enabled.
Something about how apache/airflow#14924 was fixed does not fix our scheduler liveness probe, which seems to have ever-increasing cache memory usage (with some improvement gained from deleting the scheduler log files).
Right now, if worker/scheduler pods run for long periods of time, they may accumulate so many logs under
logs.path
that they may cause issues.We should allow users to deploy a sidecar that is responsible to truncate log files after some period of time.
Here are some example values for what this feature may look like:
WARNING: we should NOT allow
{scheduler,workers}.logCleanup.enabled
andlogs.persistence.enabled
to be true at the same time (using./_helpers/validate-values.tpl
) as multiple of the sidecars may fight with each other. In the error message, tell users that they could create an airflow job instead.The text was updated successfully, but these errors were encountered: