Skip to content

Parallelize repository cloning during dev environment setup#72

Draft
Copilot wants to merge 3 commits intomainfrom
copilot/parallelize-repository-cloning
Draft

Parallelize repository cloning during dev environment setup#72
Copilot wants to merge 3 commits intomainfrom
copilot/parallelize-repository-cloning

Conversation

Copy link
Copy Markdown

Copilot AI commented Apr 9, 2026

Sequential git clone calls during composer install/composer update made initial dev environment setup unnecessarily slow. The refresh step already used xargs for parallelism; cloning did not.

Changes

  • New .maintenance/clone-all-repositories.sh: Bash script that orchestrates the full clone + refresh flow. Detects CPU cores via nproc (Linux) or sysctl -n hw.logicalcpu (macOS), defaulting to 4 if neither is available. Fetches the repository list from the GitHub API using curl, parses JSON with jq, filters the skip list, applies the destination map, then clones missing repos and refreshes all repos in parallel using xargs -P$CORES.
  • New .maintenance/clone-repository.sh: Bash helper that clones a single repository, accepting <destination> and <clone_url> as arguments.
  • Removed .maintenance/clone-all-repositories.php and .maintenance/clone-repository.php.
  • Updated composer.json: pre-install-cmd and pre-update-cmd now call bash .maintenance/clone-all-repositories.sh.
# Detect CPU cores, default to 4
if command -v nproc &>/dev/null; then
    CORES=$(nproc)
elif command -v sysctl &>/dev/null; then
    CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)
else
    CORES=4
fi

# Clone and refresh in parallel, bounded by core count
printf '%s\n' "${CLONE_LIST[@]}" | xargs -n2 -P"${CORES}" bash clone-repository.sh
printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -n1 -P"${CORES}" -I% php refresh-repository.php %

Copilot AI changed the title [WIP] Implement parallel repository cloning during development setup Parallelize repository cloning during dev environment setup Apr 9, 2026
Copilot AI requested a review from swissspidy April 9, 2026 08:31
@swissspidy swissspidy requested a review from thelovekesh April 9, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize repository cloning during development environment setup.

3 participants