Skip to content

Add fetchJobs option to parallelize submodule updates #323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add fetchJobs option to parallelize submodule updates #323

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ad5dc19
Add fetchJobs option to parallelize submodule updates
trylle Aug 9, 2020
a5c1ce9
Support fetch-jobs: 0 to use as many jobs as there are processors
trylle Oct 10, 2020
cd648cf
Merge remote-tracking branch 'actions/main'
trylle Mar 22, 2022
d46e61f
Only process fetch-jobs argument if submodules are enabled
trylle Dec 31, 2022
c4fe551
Merge remote-tracking branch 'actions/main'
trylle Dec 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ When Git 2.18 or higher is not in your PATH, falls back to the REST API to downl
# Default: 1
fetch-depth: ''

# Number of fetches to perform simultaneously when updating submodules: -1
# indicates to use git default (serial updates). 0 uses as many jobs as there are
# processors.
# Default: -1
fetch-jobs: ''

# Whether to download Git-LFS files
# Default: false
lfs: ''
Expand Down
1 change: 1 addition & 0 deletions __test__/git-auth-helper.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -765,6 +765,7 @@ async function setup(testName: string): Promise<void> {
clean: true,
commit: '',
fetchDepth: 1,
fetchJobs: -1,
lfs: false,
submodules: false,
nestedSubmodules: false,
Expand Down
1 change: 1 addition & 0 deletions __test__/input-helper.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ describe('input-helper tests', () => {
expect(settings.commit).toBeTruthy()
expect(settings.commit).toBe('1234567890123456789012345678901234567890')
expect(settings.fetchDepth).toBe(1)
expect(settings.fetchJobs).toBe(-1)
expect(settings.lfs).toBe(false)
expect(settings.ref).toBe('refs/heads/some-ref')
expect(settings.repositoryName).toBe('some-repo')
Expand Down
6 changes: 6 additions & 0 deletions action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ inputs:
fetch-depth:
description: 'Number of commits to fetch. 0 indicates all history for all branches and tags.'
default: 1
fetch-jobs:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to keep the default to 0?

Any downsides of running this im parallel per default?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I'm aware. I just wanted to have the option, and thought it was prudent not to create new defaults overriding normal git behavior. However, considering that the default fetch-depth is already non-standard, there's probably no such expectation anyway.

If parallel jobs were to be the default, then that raises the question as to what this default should be. Having checked the git source code, I can see that there is actually a 0 jobs option which implies that it will run as many jobs as there are processors (referred to as some reasonable default in the documentation). To my eye, this seems a bit excessive. At any rate, the existence of that slightly hidden option means that the fetch-jobs parameter should probably use -1 instead of 0 to omit the jobs argument.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested a bit with --jobs=0, and this probably isn't a good option for GitHub (but for the opposite reason I was worried about), since the runners are specced with only 2 cores. Since git is bottlenecked waiting for IO, you would want more jobs than that.

I think the ideal number is going to depend on a lot of factors, so it's probably better if this remains a manually enabled optimization option.

description: >
Number of fetches to perform simultaneously when updating submodules:
-1 indicates to use git default (serial updates). 0 uses as many jobs as
there are processors.
default: -1
lfs:
description: 'Whether to download Git-LFS files'
default: false
Expand Down
13 changes: 11 additions & 2 deletions dist/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7096,7 +7096,7 @@ class GitCommandManager {
yield this.execGit(args);
});
}
submoduleUpdate(fetchDepth, recursive) {
submoduleUpdate(fetchDepth, recursive, fetchJobs) {
return __awaiter(this, void 0, void 0, function* () {
const args = ['-c', 'protocol.version=2'];
args.push('submodule', 'update', '--init', '--force');
Expand All @@ -7106,6 +7106,9 @@ class GitCommandManager {
if (recursive) {
args.push('--recursive');
}
if (fetchJobs > -1) {
args.push(`--jobs=${fetchJobs}`);
}
yield this.execGit(args);
});
}
Expand Down Expand Up @@ -7423,7 +7426,7 @@ function getSource(settings) {
// Checkout submodules
core.startGroup('Fetching submodules');
yield git.submoduleSync(settings.nestedSubmodules);
yield git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules);
yield git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules, settings.fetchJobs);
yield git.submoduleForeach('git config --local gc.auto 0', settings.nestedSubmodules);
core.endGroup();
// Persist credentials
Expand Down Expand Up @@ -17216,6 +17219,12 @@ function getInputs() {
result.fetchDepth = 0;
}
core.debug(`fetch depth = ${result.fetchDepth}`);
// Fetch jobs
result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '-1'));
if (isNaN(result.fetchJobs) || result.fetchJobs < -1) {
result.fetchJobs = -1;
}
core.debug(`fetch jobs = ${result.fetchJobs}`);
// LFS
result.lfs = (core.getInput('lfs') || 'false').toUpperCase() === 'TRUE';
core.debug(`lfs = ${result.lfs}`);
Expand Down
16 changes: 14 additions & 2 deletions src/git-command-manager.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,11 @@ export interface IGitCommandManager {
shaExists(sha: string): Promise<boolean>
submoduleForeach(command: string, recursive: boolean): Promise<string>
submoduleSync(recursive: boolean): Promise<void>
submoduleUpdate(fetchDepth: number, recursive: boolean): Promise<void>
submoduleUpdate(
fetchDepth: number,
recursive: boolean,
fetchJobs: number
): Promise<void>
tagExists(pattern: string): Promise<boolean>
tryClean(): Promise<boolean>
tryConfigUnset(configKey: string, globalConfig?: boolean): Promise<boolean>
Expand Down Expand Up @@ -312,7 +316,11 @@ class GitCommandManager {
await this.execGit(args)
}

async submoduleUpdate(fetchDepth: number, recursive: boolean): Promise<void> {
async submoduleUpdate(
fetchDepth: number,
recursive: boolean,
fetchJobs: number
): Promise<void> {
const args = ['-c', 'protocol.version=2']
args.push('submodule', 'update', '--init', '--force')
if (fetchDepth > 0) {
Expand All @@ -323,6 +331,10 @@ class GitCommandManager {
args.push('--recursive')
}

if (fetchJobs > -1) {
args.push(`--jobs=${fetchJobs}`)
}

await this.execGit(args)
}

Expand Down
3 changes: 2 additions & 1 deletion src/git-source-provider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,8 @@ export async function getSource(settings: IGitSourceSettings): Promise<void> {
await git.submoduleSync(settings.nestedSubmodules)
await git.submoduleUpdate(
settings.fetchDepth,
settings.nestedSubmodules
settings.nestedSubmodules,
settings.fetchJobs
)
await git.submoduleForeach(
'git config --local gc.auto 0',
Expand Down
5 changes: 5 additions & 0 deletions src/git-source-settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ export interface IGitSourceSettings {
*/
fetchDepth: number

/**
* The number of fetches to perform simultaneously when updating submodules
*/
fetchJobs: number

/**
* Indicates whether to fetch LFS objects
*/
Expand Down
7 changes: 7 additions & 0 deletions src/input-helper.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,13 @@ export async function getInputs(): Promise<IGitSourceSettings> {
}
core.debug(`fetch depth = ${result.fetchDepth}`)

// Fetch jobs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this logic should be moved below line 114

then we can do something like:

if(result.submodules){
    result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '-1'))
    if (isNaN(result.fetchJobs) || result.fetchJobs < -1) {
        etc...
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in d46e61f

result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '-1'))
if (isNaN(result.fetchJobs) || result.fetchJobs < -1) {
result.fetchJobs = -1
}
core.debug(`fetch jobs = ${result.fetchJobs}`)

// LFS
result.lfs = (core.getInput('lfs') || 'false').toUpperCase() === 'TRUE'
core.debug(`lfs = ${result.lfs}`)
Expand Down