Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow jobs to run immediately when they are unblocked #8203

Merged
merged 4 commits into from Aug 24, 2019

Conversation

@wiwa
Copy link
Contributor

commented Aug 23, 2019

Problem

A previous update to RscCompile task introduced a write_to_cache job waiting
on both Rsc and Zinc artifacts before writing to cache (commit
26e8056). However, these jobs were run at the
very end of the execution graph since they had very low priority. This means
that in a distributed run, many extra cache misses are being introduced, causing
very poor performance.

Solution

Add a run_asap field to Jobs which sets their priority to maximum of the
jobs list, allowing the scheduler to start these Jobs the moment they're
unblocked.

Result

write_to_cache jobs will set run_asap=True, causing them to be run asap.

@@ -28,13 +28,15 @@ def __init__(self, key, fn, dependencies, size=0, on_success=None, on_failure=No
:param on_success: Zero parameter callback to run if job completes successfully. Run on main
thread.
:param on_failure: Zero parameter callback to run if job completes successfully. Run on main
thread."""
thread.
:param run_asap: Boolean indicating whether or not to queue job immediately once unblocked."""

This comment has been minimized.

Copy link
@stuhood

stuhood Aug 23, 2019

Member

If this ends up needing another CI run, it would be good to add an explanation of how this relates to the size.

@stuhood stuhood merged commit 437e9b3 into pantsbuild:master Aug 24, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@stuhood stuhood added this to the 1.19.x milestone Aug 24, 2019

stuhood added a commit that referenced this pull request Aug 24, 2019
Allow jobs to run immediately when they are unblocked (#8203)
### Problem

A previous update to RscCompile task introduced a `write_to_cache` job waiting
on both Rsc and Zinc artifacts before writing to cache (commit
26e8056).  However, these jobs were run at the
very end of the execution graph since they had very low priority. This means
that in a distributed run, many extra cache misses are being introduced, causing
very poor performance.

### Solution

Add a `run_asap` field to `Job`s which sets their priority to maximum of the
jobs list, allowing the scheduler to start these Jobs the moment they're
unblocked.

### Result

`write_to_cache` jobs will set `run_asap=True`, causing them to be run asap.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.