Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
jest-runtime: atomic cache write, and check validity of data #4088
This change tries to address what may be a cause for #1874, where I posted some details on the approach. By doing atomic writes we ensure there's no cache file ending up being a mix of two transformed code files, and we limit the concurrency issues of having a file read and written at the same time. A prior PR #3561 tries to address the problem using a lock, but locks bring additional complexity that this change tries to avoid (ex. deadlocks).
This change also adds a checksum, because there can be other processes still writing non-atomically to cache files. Additional, not all filesystems may support atomic renames (what
This change incurs a slight performance cost that is the additional I/O call to rename the files. However I believe it is less costly than a lock solution. I don't think this should have much effect even on processing several thousands of files.
It is quite challenging to test for concurrency issues, so this change relies on the knowledge that
Jul 20, 2017
I don't know how it's possible, but apparently adding
Got it, it's because we use return paths from inside the jest cache itself for the source maps. When code outside tries to read these files, naturally it fails because of the hash header. To fix this we can write hashes to different files. The problem is that it'll cause the number of files to double, something I'm not too sure is acceptable for some of our big repos.
EDIT: actually it's not acceptable because when reading source maps we should also check the hash.
Ideally, get rid of the code that return a path, and return a memory-based hash table instead. But, this will force us to publish a new major release of
To avoid a breaking change, we can just write the source map to a temporary file, but this will clutter more and more the temporary file and it never gets cleaned.
I'm fine with the breaking change. Since the transformer in jest-runtime includes the version number, it shouldn't really matter from any one version to the next what the transform cache looks like. Do you think you could work on this tomorrow so we can tag a release and unblock shipping a bunch of fixes? :)
I figured out a way to do it without breaking change. The source maps will not be covered by the checksum verification, but to make them covered we'd have to change the model all across the stack I think, that'd take some more research and time. The transformed code itself is the most important thing that I hope this changeset gets right.
I'll land once CircleCI shows green.
@@ Coverage Diff @@ ## master #4088 +/- ## ========================================== + Coverage 60.32% 60.37% +0.04% ========================================== Files 195 195 Lines 6757 6768 +11 Branches 6 6 ========================================== + Hits 4076 4086 +10 - Misses 2678 2679 +1 Partials 3 3
pushed a commit
this pull request
Aug 21, 2017
Thank you so much for this fix!
We have a project that was manifesting this issue extremely often on CodeShip under docker. It was behaving as if input source code from