Skip to content

Conversation

@charles-zablit
Copy link
Contributor

@charles-zablit charles-zablit commented Oct 15, 2025

This patch makes update-checkout use a ThreadPoolExecutor instead of a multiprocessing.Pool. The main benefits are:

  • Using processes means that the Python modules that make up update-checkout get re-executed every time a subprocess starts. This causes problems when updating the swift repository as the update-checkout files are edited which can cause functions and modules to no longer be available. With threads, the modules are already loaded in memory, we don't rely on the filesystem.
  • The git commands are invoked through Popen meaning they are executed in a subprocess. There is no need to open a subprocess in the multiprocessing.Pool to run other subprocesses. A Python thread is a more light weight solution for scheduling tasks.
  • There were Timeout errors specifically on Python 3.9 due to CPython not freeing global locks properly (see [update-checkout] force verbose mode if python 3.9 #84825). Using a ThreadPool bypasses that issue meaning that we will be able to reenable the quiet output mode on Python 3.9.

This patch also refactors the shell.run and shell.capture invocations: before, they were relying on pushd and popd with Python context managers. This does not work with multithreading as the shell module is now shared between threads. Now, git commands are executed with git -C /repo_path instead of relying on the working directory.

rdar://162775786

@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from 25bb836 to 40c3f1d Compare October 15, 2025 16:52
@compnerd
Copy link
Member

@swift-ci please smoke test

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from 40c3f1d to a333f7b Compare October 15, 2025 17:34
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test linux

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from a333f7b to f1435cc Compare October 15, 2025 20:15
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test linux

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from f1435cc to f2a0c23 Compare October 15, 2025 20:26
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test linux

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from f2a0c23 to 7122f0a Compare October 15, 2025 20:59
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from 7122f0a to 051d570 Compare October 15, 2025 23:04
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from 051d570 to 7d41b5c Compare October 16, 2025 10:18
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test

@charles-zablit charles-zablit force-pushed the charles-zablit/update-checkout/threadpool branch from 7d41b5c to 9d3d43c Compare October 16, 2025 10:30
@charles-zablit
Copy link
Contributor Author

@swift-ci please smoke test

@charles-zablit charles-zablit added the update-checkout Area → utils: the `update-checkout` script label Oct 16, 2025
Copy link
Contributor

@adrian-prantl adrian-prantl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the explanation this sounds like a good change.

@charles-zablit charles-zablit merged commit 24d03a1 into swiftlang:main Oct 16, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

update-checkout Area → utils: the `update-checkout` script

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants