New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rsc-compile] only metacp the jdk/scala synthetics one time per run #6614

Merged
merged 9 commits into from Oct 12, 2018

Conversation

Projects
None yet
3 participants
@baroquebobcat
Copy link
Contributor

baroquebobcat commented Oct 10, 2018

Problem

The jdk was being reindexed by metacp once for each target. Indexing the jdk took several seconds, so doing it N times, where N is the number of targets added a lot of overhead.

Solution

This patch extracts metacping the jdk into it's own job that's created before creating jobs for each target. It removes generating indices of scala synthetics and the jdk from the rsc job and makes metacp conditional in it. It's now conditional because metacp and metai only need to be invoked if there are java dependencies.

The synthetic scala generation was extracted by detecting when the current jar contains the scala standard library and telling metacp to generate the synthetics in that case.

Result

The jdk shows up in the compile execution graph as a metacp task, and both it and the scala synthetics are only generated once per run.

Because of how the jdk is generated, indexing it will happen for each run in which there are invalid targets.

Fixes #6547

@baroquebobcat baroquebobcat requested a review from stuhood Oct 10, 2018

@@ -699,6 +701,11 @@ def format_length(self):
counter.size = len(jobs)
return jobs

def pre_compile_jobs(self, counter):
# NB: Override this to provide jobs that are scheduled before any of the target jobs are

This comment has been minimized.

@stuhood

stuhood Oct 10, 2018

Member

Worthwhile probably to have this as a method comment, and to also explain what you mention in the description about not running if there are no invalid targets.

distribution,
tgt='jdk',
input_files=tuple(
# NB no input files because the jdk is expected to exist on the system in a known

This comment has been minimized.

@stuhood

stuhood Oct 10, 2018

Member

This should point to a github issue about replacing (or at least hardening) the .jdk strategy so that we can ensure that the jdk we're expecting ends up in the cache key. Maybe @illicitonion already has one.

This comment has been minimized.

@illicitonion

illicitonion Oct 11, 2018

Contributor

There isn't an existing one

@stuhood stuhood requested a review from illicitonion Oct 10, 2018

baroquebobcat added some commits Oct 10, 2018

@baroquebobcat baroquebobcat merged commit 180be54 into pantsbuild:master Oct 12, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment