1.0.0: dist targed does not check is po/POTFILES.in is up-to-date

[kloczek]

Describe the bug
On building just released gnome-disk-utility found that gnome-disk-utility-update-po fails because po/POTFILES.in is not up-to-date.
https://gitlab.gnome.org/GNOME/gnome-disk-utility/-/issues/283
With GNU automake make dist-check is noit only forming dist tar ball but is checkig is it possible to build everything of such tar ball.

To Reproduce

git clone https://gitlab.gnome.org/GNOME/gnome-disk-utility/
cd gnome-disk-utility
meson setup --buildtype=plain build
meson dist build
meson gnome-disk-utility-update-po -C build

Expected behavior
dist targed thould try to veryfi generated dist tart ball.
Eventually shpuld be provided like with automake sist-check targe which part should be as well <project>-update-po.

system parameters

• Is this a cross build or just a plain native build (for the same computer)?
  Native
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.)
  Linux x86_64
• what Python version are you using e.g. 3.8.0
  3.8.16
• what meson --version
  1.0.0
• what ninja --version if it's a Ninja build
  1.11.1

[eli-schwartz]

With GNU automake make dist-check is noit only forming dist tar ball but is checkig is it possible to build everything of such tar ball.

The GNU autotools have both make dist and make distcheck.

Meson only has meson dist, but this behaves like an autotools distcheck -- it takes the resulting dist and tries to build, test, and install it.

However it doesn't attempt to run maintainer targets, and neither should you when building the project. It is, after all, a maintainer target -- not a user target.

It's not obvious that updating po files should be regularly done, because it modifies comments and metadata every time. It's really only useful immediately before running git commit. Even if you do that, it will still repeatedly update the metadata for the date, resulting in a dirty git status. Ironically, meson then errors out stating that it cannot create a dist with uncommitted changes. Even though the only change is a timestamp.

Running it, even if it succeeds, is totally pointless for users, because the results aren't actually used unless you want to upload the resulting .po files to a service like transifex, or use it in weblate, or unless you're personally interested in translating the software yourself, in person.

What are you trying to accomplish?

[kloczek]

It's not obvious that updating po files should be regularly done, because it modifies comments and metadata every time.

That is not true. It changes .po files only when something has been changed in files listed in POTFILES.in (changed line number where is translated string or string has been changed).
In this case it is issue with file removed from source tree.
In case of automake distcheck such issues will be catched on spot and target execution will exit with non-zero exit code.

What are you trying to accomplish?

On building my rpm packages packages I'm always calling make -C po update-po or ninja <project>-udatep-po target because most of the not only gnome projects comes with not updated .po files and by running those targets is possible to remove from generated .mo files MANY strings which are no longer used (those strings are moved to comments and if they will be used again update-po target will uncomment those strings automatically.

[tkloczko@devel-g2v SPECS]$ grep "%make_build -C po update-po" *spec | wc -l; grep "%meson_build .*-update-po" *spec | wc -l
76
129

You would be surprised how many projects dist tar balls provides .po files with sometimes even 50% "garbage" because maintainer never executed *update-po target(s) (in case of some gnome there are as well help-<project>-update-po targets).

Meson dist target works with only git or mercurial tree.
IMO dist should automatically call <project>-udatep-po to asses are .po files up-to-date or not and refuse generate dist tar ball if translations needs updates.

Most of the maintainers are not aware of the fact that .po files should be updated before make release.
IMO meson should provide descent support in dist target procedure to keep .po files in the best possible condition like it is case of automake + gettext and distcheck target.
From that point of view IMO meson support has here kind of blind spot.

[eli-schwartz]

That is not true. It changes .po files only when something has been changed in files listed in POTFILES.in (changed line number where is translated string or string has been changed).

No. Again, the timestamps embedded as POT-Creation-Date are updated every time, regardless of whether anything else changed.

because most of the not only gnome projects comes with not updated .po files and by running those targets is possible to remove from generated .mo files MANY strings which are no longer used

Sounds like an upstream bug that should be reported to the project.

Most of the maintainers are not aware of the fact that .po files should be updated before make release.

... Why on earth not? How exactly are translations supposed to work, then? Does anyone ever contribute translations to those projects?

I am not used to seeing such issues, the projects I dealt with had this as part of the release checklist. Maybe these projects need help to use weblate or something though.

[kloczek]

No. Again, the timestamps embedded as POT-Creation-Date are updated every time, regardless of whether anything else changed.

If it is true that is another gettext issue.
If nothing needs to be updated in set of translated strings or comments about source file name and/or position time stamp in preamble should be not updated.

because most of the not only gnome projects comes with not updated .po files and by running those targets is possible to remove from generated .mo files MANY strings which are no longer used

Sounds like an upstream bug that should be reported to the project.

Indeed however in all those cases when I've been trying to inform about that all that where refused because more or less "dist does not show anything".
In other words there is no updates because nothing event warns that something is not up-to-date -> because .po files are not updated translators are not updating .po files.
As you see it is kind of procedural glitch/blind spot.

Most of the maintainers are not aware of the fact that .po files should be updated before make release.

... Why on earth not? How exactly are translations supposed to work, then? Does anyone ever contribute translations to those projects?

In most of the cases translations are only provided to what is in .po files.
If there is some set of radical changes of the source code which gnome project had many times in the past .po files starts not providing new strings translations and maintain strings which are no longer used.
Because most of the maintainers refuses to provide in VCS up-to-date .pot file many translators are taking one of the existing .po files to provide translation of the new language. By this such translations for new language are very often not up-to-date from first commit.

I am not used to seeing such issues, the projects I dealt with had this as part of the release checklist. Maybe these projects need help to use weblate or something though.

OK. To show you how bad it is I made small test.
In my nautilus.spec file for latest 44.alpha I have

%build
%meson \
        -D docs=true \
        -D extensions=true \
        -D introspection=true \
        -D profiling=false \
        -D selinux=true \
        -D tests=headless \
        %{nil}
%meson_build %{name}-update-po
%meson_build

With above:

[tkloczko@devel-g2v Packages]$ rpm -qpi nautilus-44.alpha-2.g2v.x86_64.rpm
Summary     : File manager for GNOME
Name        : nautilus
Version     : 44.alpha
Release     : 2.g2v
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 8349230
License     : GPL-3.0-or-later (https://spdx.org/licenses/GPL-3.0-or-later.html), LGPL-2.1-or-later (https://spdx.org/licenses/LGPL-2.1-or-later.html)
Signature   : (none)
Source RPM  : nautilus-44.alpha-2.g2v.src.rpm
Build Date  : Fri Jan 13 01:26:57 2023
Build Host  : domek.lan
URL         : https://wiki.gnome.org/Apps/Nautilus/
VCS         : https://gitlab.gnome.org/GNOME/nautilus/
Description :
Nautilus is the file manager and graphical shell for the GNOME desktop that
makes it easy to manage your files and the rest of your system. It allows to
browse directories on local and remote filesystems, preview files and launch
applications associated with them. It is also responsible for handling the icons
on the GNOME desktop.

and with commented %meson_build %{name}-update-po:

[tkloczko@devel-g2v RPMS]$ rpm -qpi nautilus-44.alpha-2.g2v.x86_64.rpm
Summary     : File manager for GNOME
Name        : nautilus
Version     : 44.alpha
Release     : 2.g2v
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 13399830
License     : GPL-3.0-or-later (https://spdx.org/licenses/GPL-3.0-or-later.html), LGPL-2.1-or-later (https://spdx.org/licenses/LGPL-2.1-or-later.html)
Signature   : (none)
Source RPM  : nautilus-44.alpha-2.g2v.src.rpm
Build Date  : Thu Jan 26 05:42:26 2023
Build Host  : domek.lan
URL         : https://wiki.gnome.org/Apps/Nautilus/
VCS         : https://gitlab.gnome.org/GNOME/nautilus/
Description :
Nautilus is the file manager and graphical shell for the GNOME desktop that
makes it easy to manage your files and the rest of your system. It allows to
browse directories on local and remote filesystems, preview files and launch
applications associated with them. It is also responsible for handling the icons
on the GNOME desktop.

With %{name}-update-po execution size is: 8349230 bytes.
Without: 13399830 bytes.
AFAIK you will find in ALL current gnome projects bigger or smaller differences doing the same as above.

[tkloczko@devel-g2v RPMS]$ echo "13399830-8349230" |bc -q
5050600

~FIVE MEGABYTEST LESS.

[kloczek]

Just in case ..
I'm fully aware that forcing update of the .po files by meson in dist target will cause wave of complains that those files should be not touched because in our projects those files are maintained other way however because that claim that those files are maintained other way is false all that could be redirected to actual results.
With that it will be no other cases that released new version will be with not updated POTFILES.in as well.

In whole workflow of projects translations updates needs to be fed somehow with proper set of strings to translate or update (in case of fuzzy entries).
Meson dist target probably is only place which can 100% correctly and in simplest possible way provide stream of such updates for translators.
If you see other way to solve current state please let me know.
On release new meson with updated dist target procedure it needs to be explained why that behaviour has been changed.
Introduction of such change probably should be added with next major meson release (1.1.0?).

[kloczek]

Any updaets/conclutions? 🤔

Just found another case with gnome project which has been released with faulty po/POTFILES.in
https://gitlab.gnome.org/GNOME/gnome-boxes/-/issues/919

[kloczek]

And yet another one https://gitlab.gnome.org/GNOME/console/-/issues/263

[xclaesse]

No. Again, the timestamps embedded as POT-Creation-Date are updated every time, regardless of whether anything else changed.

If it is true that is another gettext issue.

I gave it a try, and it is true, POT-Creation-Date change every time update-po target is rebuild. I think this could be worked around in Meson by comparing old and new po file, and it the only diff is that line it could leave the file untouched. Googling for that issue I found many projects that does that already:

• https://code.djangoproject.com/attachment/ticket/6106/6106-1.diff
• fix gettext builder that writes pot files unless the content are same without creation date sphinx-doc/sphinx#3490

If we do that, then I think it makes sense to run update-po as dist script.

[xclaesse]

First step: #11400

[eli-schwartz]

I think this is a gettext issue, ask gettext upstream to fix this.

[eli-schwartz]

I still don't understand what this is trying to solve. Running the update-po target is something the upstream maintainers need to do from a live checkout, and commit the results, and update something like transifex or weblate with.

Running it as a dist script, on the other hand, is totally pointless since it will only run inside the unpacked dist tarball and won't inform the person running it that anything has changed. They still need to manually run it as part of a release checklist.

As for checking whether POTFILES.in is up to date... Who cares? I mean it, really I do. You cannot "automatically" detect when it's missing files, only when it has too many. And frankly, I don't want to detect that either, because... updating POTFILES.in is a maintainer task to be done once as part of the release checklist, it doesn't need to be constantly updated during development and then amended again and again, if development reworks the list of source files. Checking this at all encourages people to commit updated po files, anyway.

In contrast, meson dist is a useful test to run on potentially every commit, in order to check that the code still compiles as is. Which is not obviously true, if you rely on dist scripts.

[xclaesse]

ask gettext upstream to fix this

ahahahahahh... good joke. TBH if you want to change gettext just write it from scratch and everyone will be super happy.

Running it as a dist script, on the other hand, is totally pointless since it will only run inside the unpacked dist tarball and won't inform the person running it that anything has changed. They still need to manually run it as part of a release checklist.

Hm, it's true that a dist script is run after the check for dirty tree, so that's not good. While I agree it is maintainers' job, it is also Meson's job to facilitate their work, and clearly many maintainers forgets about updating po files before a release. Maybe we could just do a "dry run" of update-po in mdist and print a warning?

[eli-schwartz]

I still don't understand how that's supposed to help... dist scripts run inside the unpacked dist tarball, not inside the git repo. It doesn't even matter whether we check if the tree is dirty before or after... Running a dist script won't make the tree dirty.

[xclaesse]

@eli-schwartz A dist script (added automatically by i18n module) could do a dry run of update-po and return an error if any file needs to be updated. Or just print a warning if we think it's not too bad.

[xclaesse]

That could also be done as a test. For example gnome.gtkdoc() optionally add a test that fails if any function is not documented, we could do the same with a test that verify po files are up to date. Do we have a way to run some tests only during dist?

[kloczek]

Any update? 🤔
Issues caused by lac of some dist target functionalities still happens https://gitlab.gnome.org/GNOME/sushi/-/issues/112

[kloczek]

Another case caused by this issue https://gitlab.gnome.org/GNOME/gnome-control-center/-/issues/2840

[kloczek]

Any update? 🤔

Yet another such case https://gitlab.gnome.org/GNOME/gnome-music/-/issues/596
I can say that recently that kind of issues are more and more common in case of all Gnome projects 😞

[eli-schwartz]

Any update? 🤔

Yes.

[kloczek]

May I as for short comment?

[eli-schwartz]

The ticket has been open for a year and I still haven't heard a convincing argument that it is correct to make such a change.

[kloczek]

|\ho issues with increasing not updated many meson based projects translations is not enough argument? 🤔