Implement checks for Object Pascal format() strings

[luebbe]

I'd like to add checks for Object Pascal format() strings. c-format is pretty close, but doesn't catch all cases, especially not the indexed parameters. See Gnu Gettext, which links to the Free Pascal docs.

Currently I'm trying to figure out, at which places functions for a new object-pascal-format flag have to be added.

Looking through weblate's code, I found format.py, which definitely looks like a place to start. test_format_checks.py too. Any other not so obvious places?

One question: are there any rules for the named regex capturing groups in format.py? I see that most regexes use a similar naming scheme, but it's not consistent across all regexes. Can I conclude that there is no rule for the group names?

[nijel]

You can take look at #5248, it added Lua format.

The named regex groups are AFAIK not used for anything, and I'm not convinced that they follow some consistent naming.

[luebbe]

A little status report plus a call for help:

I've made some progress on the issue. So far I managed to get dev-docker up and running (partly, see below) on lubuntu on an old laptop of mine. I could make the desired changes add tests and run them locally in docker. So far so good.

My final test would be to create a translation project in dev-docker which contains object-pascal-format strings and here is where I have problems. I can't get Weblate itself to run, only the tests.

When I just sudo rundev.sh and wait until all three containers are past their "starting" phase, accessing localhost:8080 results in a "502 bad gateway".

the output of docker ps is:

CONTAINER ID   IMAGE                COMMAND                  CREATED         STATUS                   PORTS                      NAMES
49ebe2170068   weblate-dev:latest   "/app/bin/start runs…"   9 minutes ago   Up 9 minutes (healthy)   127.0.0.1:8080->8080/tcp   dev-docker_weblate_1
744a765999cf   postgres:12-alpine   "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             5432/tcp                   dev-docker_database_1
73cb3a29aa07   redis:6-alpine       "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes             6379/tcp                   dev-docker_cache_1

the output of docker logs 49eb is:

... more check stderr messages before this line ...

check stderr | 
check stderr | System check identified 7 issues (1 silenced).
2021-04-21 12:34:27,042 INFO exited: check (exit status 1; expected)
nginx stderr | 2021/04/21 12:34:52 [crit] 237#237: *3 connect() to unix:///run/uwsgi/app/weblate/socket failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /healthz/ HTTP/1.1", upstream: "uwsgi://unix:///run/uwsgi/app/weblate/socket:", host: "localhost:8080"
nginx stdout | 127.0.0.1 - - [21/Apr/2021:12:34:52 +0000] "GET /healthz/ HTTP/1.1" 502 166 "-" "curl/7.64.0"
celery-celery stderr | [2021-04-21 12:35:22,312: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[2a14c71c-ac56-463a-afd9-e3cc30e923d7]  

... more similar nginx stderr | stdout messages after this line ...

Any hint what I can check or have to change? Admittedly my fork is a few hundred commits behind, so maybe I failed to pick up one or the other fix.

[nijel]

Any uwsgi related log entries?

[luebbe]

Yes, tons of ones like the middle line of the log output above. "... GET .../socket failed permission denied ..."

Plus these, which scrolled off screen before:

2021-04-21 12:33:56,596 INFO spawned: 'uwsgi' with pid 236
2021-04-21 12:33:57,964 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
uwsgi stderr | [uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/weblate.ini
uwsgi stderr | Wed Apr 21 12:33:56 2021 - *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Apr 21 12:33:56 2021] ***
uwsgi stderr | Wed Apr 21 12:33:56 2021 - compiled with version: 8.2.0 on 10 February 2019 02:42:46
uwsgi stderr | Wed Apr 21 12:33:56 2021 - os: Linux-5.8.0-50-generic #56-Ubuntu SMP Mon Apr 12 17:18:36 UTC 2021
uwsgi stderr | Wed Apr 21 12:33:56 2021 - nodename: 49ebe2170068
uwsgi stderr | Wed Apr 21 12:33:56 2021 - machine: x86_64
uwsgi stderr | Wed Apr 21 12:33:56 2021 - clock source: unix
uwsgi stderr | Wed Apr 21 12:33:56 2021 - pcre jit disabled
uwsgi stderr | Wed Apr 21 12:33:56 2021 - detected number of CPU cores: 2
uwsgi stderr | Wed Apr 21 12:33:56 2021 - current working directory: /
uwsgi stderr | Wed Apr 21 12:33:56 2021 - writing pidfile to /run/uwsgi/app/weblate/pid
uwsgi stderr | Wed Apr 21 12:33:56 2021 - detected binary path: /usr/bin/uwsgi-core
uwsgi stderr | Wed Apr 21 12:33:56 2021 - uWSGI running as root, you can use --uid/--gid/--chroot options
uwsgi stderr | Wed Apr 21 12:33:56 2021 - *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
uwsgi stderr | Wed Apr 21 12:33:56 2021 - your memory page size is 4096 bytes
uwsgi stderr | Wed Apr 21 12:33:56 2021 - detected max file descriptor number: 1048576
uwsgi stderr | Wed Apr 21 12:33:56 2021 - lock engine: pthread robust mutexes
uwsgi stderr | Wed Apr 21 12:33:56 2021 - thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi stderr | Wed Apr 21 12:33:56 2021 - uwsgi socket 0 bound to UNIX address /run/uwsgi/app/weblate/socket fd 3
uwsgi stderr | Wed Apr 21 12:33:56 2021 - uWSGI running as root, you can use --uid/--gid/--chroot options
uwsgi stderr | Wed Apr 21 12:33:56 2021 - *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
uwsgi stderr | Wed Apr 21 12:33:56 2021 - Python version: 3.7.3 (default, Jan 22 2021, 20:04:44)  [GCC 8.3.0]
uwsgi stderr | Wed Apr 21 12:33:56 2021 - Python main interpreter initialized at 0x555dd0cf36b0
uwsgi stderr | Wed Apr 21 12:33:56 2021 - uWSGI running as root, you can use --uid/--gid/--chroot options
uwsgi stderr | Wed Apr 21 12:33:56 2021 - *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
uwsgi stderr | Wed Apr 21 12:33:56 2021 - python threads support enabled
uwsgi stderr | Wed Apr 21 12:33:56 2021 - your server socket listen backlog is limited to 100 connections
uwsgi stderr | Wed Apr 21 12:33:56 2021 - your mercy for graceful operations on workers is 60 seconds
uwsgi stderr | Wed Apr 21 12:33:56 2021 - mapped 308064 bytes (300 KB) for 3 cores
uwsgi stderr | Wed Apr 21 12:33:56 2021 - *** Operational MODE: preforking ***
uwsgi stderr | Wed Apr 21 12:33:56 2021 - added /usr/local/lib/python3.7/dist-packages/ to pythonpath.
uwsgi stderr | Wed Apr 21 12:34:20 2021 - WSGI app 0 (mountpoint='') ready in 24 seconds on interpreter 0x555dd0cf36b0 pid: 236 (default app)
uwsgi stderr | Wed Apr 21 12:34:20 2021 - uWSGI running as root, you can use --uid/--gid/--chroot options
uwsgi stderr | Wed Apr 21 12:34:20 2021 - *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
uwsgi stderr | Wed Apr 21 12:34:20 2021 - *** uWSGI is running in multiple interpreter mode ***
uwsgi stderr | Wed Apr 21 12:34:20 2021 - spawned uWSGI master process (pid: 236)
uwsgi stderr | Wed Apr 21 12:34:20 2021 - spawned uWSGI worker 1 (pid: 400, cores: 1)
uwsgi stderr | Wed Apr 21 12:34:20 2021 - spawned uWSGI worker 2 (pid: 401, cores: 1)
uwsgi stderr | Wed Apr 21 12:34:20 2021 - spawned uWSGI worker 3 (pid: 402, cores: 1)
uwsgi stderr | Wed Apr 21 12:34:20 2021 - *** Stats server enabled on :1717 fd: 13 ***

[nijel]

WARNING: you are running uWSGI as root

Can you try whether adding this to the dev container will help?

diff --git dev-docker/weblate-dev/Dockerfile dev-docker/weblate-dev/Dockerfile
index e83036a3ec..f631cf7675 100644
--- dev-docker/weblate-dev/Dockerfile
+++ dev-docker/weblate-dev/Dockerfile
@@ -50,3 +50,4 @@ COPY requirements.txt /tmp/
 RUN pip3 install -r /tmp/requirements.txt
 # List should match weblate/weblate
 RUN chmod -R 777  /etc/nginx/sites-available/ /var/log/nginx/ /var/lib/nginx /app/data /app/cache /run /home/weblate /etc/localtime /etc/supervisor/conf.d
+USER 1000

[luebbe]

Unfortunately not. USER 1000 is run as step 10/10 but the "502 bad gateway" remains, as well as the "permission denied" uwsgi errors in the log.
The startup messages are similar. Still full of "WARNING: you are running uWSGI as root".

My user is in the docker group, so it should not need to run docker commands as sudo. But when I call rundev.sh without sudo, it does't even get past the "starting" phase.

Without sudo the log shows:

weblate_1   | [2021-04-22 09:04:40,467: WARNING/127] Handled exception: PermissionError: [Errno 13] Permission denied: '/app/data/home/gitlock'
weblate_1   | Traceback (most recent call last):
weblate_1   |   File "/usr/local/bin/weblate", line 8, in <module>
weblate_1   |     sys.exit(main())
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/weblate/runner.py", line 34, in main
weblate_1   |     utility.execute()
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/django/core/management/__init__.py", line 377, in execute
weblate_1   |     django.setup()
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/django/__init__.py", line 24, in setup
weblate_1   |     apps.populate(settings.INSTALLED_APPS)
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/django/apps/registry.py", line 122, in populate
weblate_1   |     app_config.ready()
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/weblate/vcs/apps.py", line 84, in ready
weblate_1   |     with lockfile:
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/filelock.py", line 323, in __enter__
weblate_1   |     self.acquire()
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/filelock.py", line 271, in acquire
weblate_1   |     self._acquire()
weblate_1   |   File "/usr/local/lib/python3.7/dist-packages/filelock.py", line 384, in _acquire
weblate_1   |     fd = os.open(self._lock_file, open_mode)
weblate_1   | PermissionError: [Errno 13] Permission denied: '/app/data/home/gitlock'
weblate_1   | redis is unavailable - waiting 24

in an endless loop

[luebbe]

Progress made :)

After merging with main, most problems running dev-docker went away. So I was missing some fixes before.
The only thing I had to do on top of it was to run sudo chown -R myUID:myGID dev-docker/data to get rid of a ton of warnings about "... has the wrong owner" on many files.

My tests were successful so the PR is about to come in a few minutes.

[github-actions]

Thank you for your report; the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.