Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split out a double-check-cache job for jvm/rsc compile #8221

Merged

Conversation

@stuhood
Copy link
Member

commented Aug 29, 2019

Problem

#8190 moved cache writing out of the completion of the zinc and rsc jobs and into a dependent job. But at the same time, we also had multiple attempts to "double check" the cache happening concurrently due to both the zinc and rsc jobs checking, and that race could lead to partial entries being extracted.

Solution

Since we can't actually cancel or coordinate the concurrent work, we can't safely double check the cache once either job has started. So instead, this change extracts the cache double-check into its own job that both the zinc and rsc tasks will depend on.

@stuhood stuhood changed the title Split out a double-check-cache job jvm/rsc compile Split out a double-check-cache job for jvm/rsc compile Aug 29, 2019

Split out a double-check-cache job, so that we double check the cache…
… before starting either rsc or zinc work, and then never again (otherwise hitting the cache might race with the ongoing work, which we are unable to cancel).

@stuhood stuhood force-pushed the twitter:stuhood/split-out-a-double-check-cache-job branch from 76c096c to a60f90e Aug 29, 2019

@illicitonion
Copy link
Contributor

left a comment

Seems reasonable, thanks!

@blorente
Copy link
Contributor

left a comment

Looks good, sans the failing travis shard. At least one of the test looks legitimate: JavaCompileIntegrationTest.test_java_compile_with_corrupt_remote


def __call__(self):
self.count += 1
return self.count

def increment_size(self, by=1):

This comment has been minimized.

Copy link
@blorente

blorente Aug 29, 2019

Contributor

Can this be folded into the __call__ function above?

This comment has been minimized.

Copy link
@stuhood

stuhood Aug 29, 2019

Author Member

Not without an awkward signature I think: this one increments size, while the other increments count.

@stuhood stuhood merged commit e3cf637 into pantsbuild:master Aug 29, 2019

1 of 2 checks passed

continuous-integration/travis-ci/push The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@stuhood stuhood added this to the 1.19.x milestone Aug 29, 2019

@stuhood stuhood deleted the twitter:stuhood/split-out-a-double-check-cache-job branch Aug 29, 2019

stuhood added a commit that referenced this pull request Aug 29, 2019
Split out a double-check-cache job for jvm/rsc compile. (#8221)
### Problem

#8190 moved cache writing out of the completion of the zinc and rsc jobs and into a dependent job. But at the same time, we also had multiple attempts to "double check" the cache happening concurrently due to both the zinc and rsc jobs checking, and that race could lead to partial entries being extracted.

### Solution

Since we can't actually cancel or coordinate the concurrent work, we can't safely double check the cache once either job has started. So instead, this change extracts the cache double-check into its own job that both the zinc and rsc tasks will depend on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.