Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Use ThreadPool for cache fetching and rust tar for artifact extration #6748
I was able to repro the issue almost consistently with PyCharm debug mode with
and the issue seems to be resolved by using
Since most of the untar work was done in Python and was CPU bound, we chose to move that to rust for the performance.
@wisechengyi : One of the reasons that this is a multiprocessing pool is to avoid the GIL while unpacking cached artifacts. So you'd want to benchmark this carefully to ensure that it didn't regress.
But yea, if it doesn't regress, using a threadpool here would be strictly simpler.
A nearby middle ground would be to continue to use
We have used the