Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fetchJobs option to parallelize submodule updates #323

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
+62 −5
Open

Add fetchJobs option to parallelize submodule updates #323

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ad5dc19
Add fetchJobs option to parallelize submodule updates
trylle Aug 9, 2020
a5c1ce9
Support fetch-jobs: 0 to use as many jobs as there are processors
trylle Oct 10, 2020
cd648cf
Merge remote-tracking branch 'actions/main'
trylle Mar 22, 2022
d46e61f
Only process fetch-jobs argument if submodules are enabled
trylle Dec 31, 2022
c4fe551
Merge remote-tracking branch 'actions/main'
trylle Dec 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -93,6 +93,11 @@ Refer [here](https://github.com/actions/checkout/blob/v1/README.md) for previous
# Default: 1
fetch-depth: ''

# Number of fetches to perform simultaneously when updating submodules. 0
# indicates default (serial updates).
# Default: 0
fetch-jobs: ''

# Whether to download Git-LFS files
# Default: false
lfs: ''
1 change: 1 addition & 0 deletions __test__/git-auth-helper.test.ts
View file
Original file line number Diff line number Diff line change
@@ -760,6 +760,7 @@ async function setup(testName: string): Promise<void> {
clean: true,
commit: '',
fetchDepth: 1,
fetchJobs: 0,
lfs: false,
submodules: false,
nestedSubmodules: false,
1 change: 1 addition & 0 deletions __test__/input-helper.test.ts
View file
Original file line number Diff line number Diff line change
@@ -75,6 +75,7 @@ describe('input-helper tests', () => {
expect(settings.commit).toBeTruthy()
expect(settings.commit).toBe('1234567890123456789012345678901234567890')
expect(settings.fetchDepth).toBe(1)
expect(settings.fetchJobs).toBe(0)
expect(settings.lfs).toBe(false)
expect(settings.ref).toBe('refs/heads/some-ref')
expect(settings.repositoryName).toBe('some-repo')
3 changes: 3 additions & 0 deletions action.yml
View file
Original file line number Diff line number Diff line change
@@ -56,6 +56,9 @@ inputs:
fetch-depth:
description: 'Number of commits to fetch. 0 indicates all history for all branches and tags.'
default: 1
fetch-jobs:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to keep the default to 0?

Any downsides of running this im parallel per default?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I'm aware. I just wanted to have the option, and thought it was prudent not to create new defaults overriding normal git behavior. However, considering that the default fetch-depth is already non-standard, there's probably no such expectation anyway.

If parallel jobs were to be the default, then that raises the question as to what this default should be. Having checked the git source code, I can see that there is actually a 0 jobs option which implies that it will run as many jobs as there are processors (referred to as some reasonable default in the documentation). To my eye, this seems a bit excessive. At any rate, the existence of that slightly hidden option means that the fetch-jobs parameter should probably use -1 instead of 0 to omit the jobs argument.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested a bit with --jobs=0, and this probably isn't a good option for GitHub (but for the opposite reason I was worried about), since the runners are specced with only 2 cores. Since git is bottlenecked waiting for IO, you would want more jobs than that.

I think the ideal number is going to depend on a lot of factors, so it's probably better if this remains a manually enabled optimization option.

description: 'Number of fetches to perform simultaneously when updating submodules. 0 indicates default (serial updates).'
default: 0
lfs:
description: 'Whether to download Git-LFS files'
default: false
13 changes: 11 additions & 2 deletions dist/index.js
View file
Original file line number Diff line number Diff line change
@@ -5939,7 +5939,7 @@ class GitCommandManager {
yield this.execGit(args);
});
}
submoduleUpdate(fetchDepth, recursive) {
submoduleUpdate(fetchDepth, recursive, fetchJobs) {
return __awaiter(this, void 0, void 0, function* () {
const args = ['-c', 'protocol.version=2'];
args.push('submodule', 'update', '--init', '--force');
@@ -5949,6 +5949,9 @@ class GitCommandManager {
if (recursive) {
args.push('--recursive');
}
if (fetchJobs > 0) {
args.push(`--jobs=${fetchJobs}`);
}
yield this.execGit(args);
});
}
@@ -6252,7 +6255,7 @@ function getSource(settings) {
// Checkout submodules
core.startGroup('Fetching submodules');
yield git.submoduleSync(settings.nestedSubmodules);
yield git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules);
yield git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules, settings.fetchJobs);
yield git.submoduleForeach('git config --local gc.auto 0', settings.nestedSubmodules);
core.endGroup();
// Persist credentials
@@ -14567,6 +14570,12 @@ function getInputs() {
result.fetchDepth = 0;
}
core.debug(`fetch depth = ${result.fetchDepth}`);
// Fetch jobs
result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '0'));
if (isNaN(result.fetchJobs) || result.fetchJobs < 0) {
result.fetchJobs = 0;
}
core.debug(`fetch jobs = ${result.fetchJobs}`);
// LFS
result.lfs = (core.getInput('lfs') || 'false').toUpperCase() === 'TRUE';
core.debug(`lfs = ${result.lfs}`);
16 changes: 14 additions & 2 deletions src/git-command-manager.ts
View file
Original file line number Diff line number Diff line change
@@ -39,7 +39,11 @@ export interface IGitCommandManager {
shaExists(sha: string): Promise<boolean>
submoduleForeach(command: string, recursive: boolean): Promise<string>
submoduleSync(recursive: boolean): Promise<void>
submoduleUpdate(fetchDepth: number, recursive: boolean): Promise<void>
submoduleUpdate(
fetchDepth: number,
recursive: boolean,
fetchJobs: number
): Promise<void>
tagExists(pattern: string): Promise<boolean>
tryClean(): Promise<boolean>
tryConfigUnset(configKey: string, globalConfig?: boolean): Promise<boolean>
@@ -308,7 +312,11 @@ class GitCommandManager {
await this.execGit(args)
}

async submoduleUpdate(fetchDepth: number, recursive: boolean): Promise<void> {
async submoduleUpdate(
fetchDepth: number,
recursive: boolean,
fetchJobs: number
): Promise<void> {
const args = ['-c', 'protocol.version=2']
args.push('submodule', 'update', '--init', '--force')
if (fetchDepth > 0) {
@@ -319,6 +327,10 @@ class GitCommandManager {
args.push('--recursive')
}

if (fetchJobs > 0) {
args.push(`--jobs=${fetchJobs}`)
}

await this.execGit(args)
}

3 changes: 2 additions & 1 deletion src/git-source-provider.ts
View file
Original file line number Diff line number Diff line change
@@ -181,7 +181,8 @@ export async function getSource(settings: IGitSourceSettings): Promise<void> {
await git.submoduleSync(settings.nestedSubmodules)
await git.submoduleUpdate(
settings.fetchDepth,
settings.nestedSubmodules
settings.nestedSubmodules,
settings.fetchJobs
)
await git.submoduleForeach(
'git config --local gc.auto 0',
5 changes: 5 additions & 0 deletions src/git-source-settings.ts
View file
Original file line number Diff line number Diff line change
@@ -34,6 +34,11 @@ export interface IGitSourceSettings {
*/
fetchDepth: number

/**
* The number of fetches to perform simultaneously when updating submodules
*/
fetchJobs: number

/**
* Indicates whether to fetch LFS objects
*/
7 changes: 7 additions & 0 deletions src/input-helper.ts
View file
Original file line number Diff line number Diff line change
@@ -88,6 +88,13 @@ export function getInputs(): IGitSourceSettings {
}
core.debug(`fetch depth = ${result.fetchDepth}`)

// Fetch jobs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this logic should be moved below line 114

then we can do something like:

if(result.submodules){
    result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '-1'))
    if (isNaN(result.fetchJobs) || result.fetchJobs < -1) {
        etc...
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in d46e61f

result.fetchJobs = Math.floor(Number(core.getInput('fetch-jobs') || '0'))
if (isNaN(result.fetchJobs) || result.fetchJobs < 0) {
result.fetchJobs = 0
}
core.debug(`fetch jobs = ${result.fetchJobs}`)

// LFS
result.lfs = (core.getInput('lfs') || 'false').toUpperCase() === 'TRUE'
core.debug(`lfs = ${result.lfs}`)