Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Overriding Poll Interval for Kubernetes Jobs #2724

Merged

Conversation

richardatphilo
Copy link
Contributor

Description

This change simply updates the KubernetesJobTask class to enable setting a poll_interval attribute or property. This allows subclasses to change how often the task polls the Kubernetes server for job status.

Motivation and Context

For long-running tasks, polling Kubernetes every 5 seconds for completion wastes resources. If a task author knows that a task is expected to run for many minutes, they can specify a longer interval to poll less frequently.

How I Tested It

I set up a test task by subclassing KubernetesJobTask and enabled debug logging. When poll_interval is not re-defined on the child class, the task polls every 5 seconds for completion. When adding a class attribute for poll_interval to the subclass, the task polled according to the new value.

@richardatphilo richardatphilo force-pushed the enable_overriding_k8s_poll_interval branch from df4a6fb to f8b3218 Compare June 6, 2019 19:20
Copy link
Collaborator

@dlstadther dlstadther left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this change.

In the event someone was overriding the value of __POLL_TIME, this would break. But given that class var is denoted as "private", i think this is a fine change.

Thanks!

@dlstadther dlstadther merged commit 94c5df1 into spotify:master Jun 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants