Timeout when importing svn repository

[CarlSchwan]

Describe the bug

I'm trying to import the KDE translations inside a local docker weblate instance. The translations are available here: svn://anonsvn.kde.org/home/kde/trunk/l10n-support

When importing the translations, I'm getting a timeout after one hour.

To Reproduce the bug

My component configurations are:

• Version control system: SVN
• Source code repository: svn://anonsvn.kde.org/home/kde/trunk/l10n-support
• Repository branch: empty (putting trunk was getting me weird errors in the console about git svn not able to fetch origin/trunk)

Expected behavior

Able to import massive translations without timeout

Server configuration and status

Weblate installation: Docker

• Weblate: 4.3.1
• Django: 3.1.2
• siphashc: 2.1
• Whoosh: 2.7.4
• translate-toolkit: 3.1.1
• lxml: 4.6.1
• Pillow: 8.0.0
• bleach: 3.2.1
• python-dateutil: 2.8.1
• social-auth-core: 3.3.3
• social-auth-app-django: 4.0.0
• django-crispy-forms: 1.9.2
• oauthlib: 3.1.0
• django-compressor: 2.4
• djangorestframework: 3.12.1
• django-filter: 2.4.0
• django-appconf: 1.0.4
• user-agents: 2.2.0
• filelock: 3.0.12
• setuptools: 40.8.0
• jellyfish: 0.8.2
• openpyxl: 3.0.5
• celery: 4.4.7
• kombu: 4.6.11
• translation-finder: 2.4
• weblate-language-data: 2020.10
• html2text: 2020.1.16
• pycairo: 1.16.2
• pygobject: 3.30.4
• diff-match-patch: 20200713
• requests: 2.24.0
• django-redis: 4.12.1
• hiredis: 1.1.0
• sentry_sdk: 0.19.1
• Cython: 0.29.21
• misaka: 2.1.1
• GitPython: 3.1.9
• borgbackup: 1.1.14
• pyparsing: 2.4.7
• Python: 3.7.3
• Git: 2.20.1
• psycopg2: 2.8.6
• psycopg2-binary: 2.8.6
• phply: 1.2.5
• chardet: 3.0.4
• ruamel.yaml: 0.16.12
• tesserocr: 2.5.1
• akismet: 1.1
• boto3: 1.16.1
• zeep: 4.0.0
• aeidon: 1.7.0
• iniparse: 0.5
• mysqlclient: 2.0.1
• Mercurial: 5.5.2
• git-svn: 2.20.1
• git-review: 1.28.0
• Redis server: 6.0.8
• PostgreSQL server: 13.0
• Database backends: django.db.backends.postgresql
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.example.com
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: redis://cache:6379/1, redis://cache:6379/1, regular
• Platform: Linux 5.9.12-1-default (x86_64) * Weblate: 4.3.1
• Django: 3.1.2
• siphashc: 2.1
• Whoosh: 2.7.4
• translate-toolkit: 3.1.1
• lxml: 4.6.1
• Pillow: 8.0.0
• bleach: 3.2.1
• python-dateutil: 2.8.1
• social-auth-core: 3.3.3
• social-auth-app-django: 4.0.0
• django-crispy-forms: 1.9.2
• oauthlib: 3.1.0
• django-compressor: 2.4
• djangorestframework: 3.12.1
• django-filter: 2.4.0
• django-appconf: 1.0.4
• user-agents: 2.2.0
• filelock: 3.0.12
• setuptools: 40.8.0
• jellyfish: 0.8.2
• openpyxl: 3.0.5
• celery: 4.4.7
• kombu: 4.6.11
• translation-finder: 2.4
• weblate-language-data: 2020.10
• html2text: 2020.1.16
• pycairo: 1.16.2
• pygobject: 3.30.4
• diff-match-patch: 20200713
• requests: 2.24.0
• django-redis: 4.12.1
• hiredis: 1.1.0
• sentry_sdk: 0.19.1
• Cython: 0.29.21
• misaka: 2.1.1
• GitPython: 3.1.9
• borgbackup: 1.1.14
• pyparsing: 2.4.7
• Python: 3.7.3
• Git: 2.20.1
• psycopg2: 2.8.6
• psycopg2-binary: 2.8.6
• phply: 1.2.5
• chardet: 3.0.4
• ruamel.yaml: 0.16.12
• tesserocr: 2.5.1
• akismet: 1.1
• boto3: 1.16.1
• zeep: 4.0.0
• aeidon: 1.7.0
• iniparse: 0.5
• mysqlclient: 2.0.1
• Mercurial: 5.5.2
• git-svn: 2.20.1
• git-review: 1.28.0
• Redis server: 6.0.8
• PostgreSQL server: 13.0
• Database backends: django.db.backends.postgresql
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.example.com
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: redis://cache:6379/1, redis://cache:6379/1, regular
• Platform: Linux 5.9.12-1-default (x86_64)

Additional context

I'm getting tons of

weblate_1   | celery-main stderr | [2021-01-02 22:11:33,952: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[1f6b75bb-8175-431e-832d-3b17aa8894c7]  
weblate_1   | celery-main stderr | [2021-01-02 22:12:33,999: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[f1f9130b-d1fc-4e7e-b2f6-96f2d15c8dce]  
weblate_1   | celery-main stderr | [2021-01-02 22:13:34,047: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[16cfb531-d795-4371-9231-efa7e0a55094]  
weblate_1   | celery-main stderr | [2021-01-02 22:14:34,101: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[8c480253-2cde-4077-a8c8-b121b4f7a2c7]

but in the Performance report, I see only empty celery queues. Also uswgi is running at 100% CPU but on only one thread.

[nijel]

This uses git-svn underneath and the performance might be bad on initial clone. Can you try enabling debug mode to see which of the svn operations takes that long? There is logic which should avoid fetching all changes from the repo, but it might be broken (as Subversion is probably the least used VCS backend in Weblate).

[github-actions]

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, making a donation is the right way to say thank you!

[github-actions]

This issue has been automatically marked as stale because it has not had any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!