import_project performance observations with git/gerrit backend

[ali-avci]

This might be also filed into questions section.

Due to the number of weblate projects and the items in each of them it may cause from time to time service outages on our git/gerrit backend. We are investigating in both directions gerrit and weblate.

Identified a place within weblate that might be made faster ~50x (recorded on our constellation).

git clone with the following variations on a repository with many branches and many commits is causing the huge difference in time consumption.

-depth=1 (4055 ms)
-depth=1 -no-single-branch (12350 ms)
-depth=1 -b "branch-specified-in-import-project" (245 ms)

We have two instances of Weblate installed v2.19.1 and v3.6.1.

I did apply a hot fix on both instances in two places of the source code.

Just reported this in here so that you may incorporate this to the main stream with your style.

Our migration path is a little different. We migrate weblate projects with the related business process behind it from the old versioned instance to the new one. Once finished with all projects we start the same iteration with the newest version of weblate again.

Regards,
-Ali

trans/management/commands/import_project.py

class command
method: checkout_tmp(....)
.....
gitrepo = VCS_REGISTRY[self.vcs].clone(repo,workdir, branch=branch)
.....

trans/vcs.py (2.19.1)

class GitRepository
method: clone(....)
.....
b_param = ['-b', branch] if branch else ['--no-single-branch']
cls._popen(['clone', '--depth', '1'] + b_param + [source, target])
.....

vcs/git.py (3.6.1)

class GitRepository
method: clone(....)
.....
b_param = ['-b', branch] if branch else ['--no-single-branch']
cls._popen(['clone', '--depth', '1'] + b_param + [source, target])
....

[nijel]

The reason for --no-single-branch is to be able to see branches for simplified setup when translating different branch.

• There is a branch selection in component creation which relies on all branches being present.
• It was originally introduced in e952023 because of error during import_project from different branch than master #1885, though I'm not sure that this one still applies that one seems to be addressed by patch you've mentioned and I've commited as c1bbcbf.

[nijel]

(Slightly) related to #4325 - we might need to pass the actual branch to clone in all cases.

[github-actions]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.