Add messages.pot update script to pre-commit.

[rebecca-shoptaw]

Closes #8818.

Feature. Automates generation of new messages.pot file for translation whenever Python or HTML files that affect i18n strings are changed.

Technical

The implementation is in two parts and mainly confined to two files:

1. pre-commit-config.yaml -- Added a new pre-commit local hook that runs the ./scripts/i18n-messages extract script to generate a new messages.pot file and is activated by changes to Python or HTML files.

Note: This hook has a few dependencies will need to be updated to match the current version when we update Babel, multipart, and/or web.py.

2. __init__.py in openlibrary/i18n -- Wrote new logic into the message extraction function, using a straightforward symmetric_difference compare between the old and new i18n messages to make sure that messages.pot only gets updated when i18n strings have been added/modified/deleted.

Note: This was done to prevent an infinite loop of failure within the pre-commit hook, as the hook will otherwise update the template time field in messages.pot on every run, and report a failed run as a result of the file modification, which prevents CI autofixing.

This extra logic also had the nice side effect of ensuring that there will be no messages.pot updates for totally unrelated files, as only actual relevant string changes will trigger the re-generation.

Testing

1. Make a change to a Python or HTML file that adds, removes or modifies i18n-formatted text
2. Install pre-commit if not installed
3. Commit the change; pre-commit should automatically generate a new messages.pot file

Stakeholders

@cdrini

[scottbarnes]

I think the SyntaxWarning: invalid escape sequence '\ ' is coming from something that i18n-messages extract is calling, rather than from pre-commit.

I don't think this is the solution, but with the change to use pass_filenames: false instead of files, and the following, pre-commit shows Passed for Generate POT:

diff --git a/scripts/i18n-messages b/scripts/i18n-messages
index 98f28e5d6..4085c927c 100755
--- a/scripts/i18n-messages
+++ b/scripts/i18n-messages
@@ -4,6 +4,10 @@ macros and write to openlibrary/i18n/messages.pot file.
 """
 import _init_path  # noqa: F401  Imported for its side effect of setting PYTHONPATH
 
+import warnings
+
+warnings.simplefilter('ignore')
+
 import sys
 from openlibrary import i18n

Of note, this SyntaxWarning only shows up with Python 3.12, and shows up regardless whether pre-commit is used. See also the second change listed here for Python 3.12: https://docs.python.org/3/whatsnew/3.12.html#other-language-changes.

[cdrini]

Awesome, thank you @rebecca-shoptaw !! I tested this in gitpod/locally, and:

• ✅ When no html/python files staged it doesn't run
• ✅ When HTML file staged with no i18n lines changed, it runs but doesn't save
• ✅ When HTML file staged with i18n lines moved but not modified, it runs but doesn't save
• ✅ When HTML file staged with i18n lines modified, it runs and saves

There's one case I realized would cause some issues; on my machine I have a bunch of untracked files, some of which are html/python! These are getting added to the messages.pot file when I do a commit, which is not what we want. I reckon we should try to exclude untracked/unstaged files when running from pre-commit. Maybe a --pre-commit option that adds these exclusions?

    # Exclude untracked git files.
    # git ls-files --others --exclude-standard
    untracked_files = {
        Path(line)
        for dir in dirs
        for line in subprocess.run(
            ['git', 'ls-files', '--others', '--exclude-standard', dir],
            stdout=subprocess.PIPE,
            text=True,
        ).stdout.split('\n')
        if line.endswith(('.py', '.html'))
    }

    # Also exclude unstaged git files.
    # git diff --name-only
    unstaged_files = {
        Path(line)
        for line in subprocess.run(
            ['git', 'diff', '--name-only'],
            stdout=subprocess.PIPE,
            text=True,
        ).stdout.split('\n')
        if line.endswith(('.py', '.html'))
    }

    excluded_files = untracked_files | unstaged_files

And then when it builds the catalog makes sure to toss these out?

Otherwise a few small code cleanups/fixes!

[cdrini]

Lgtm! One small change

[cdrini]

We talked Creation-Date potentially causing a few headaches via merge conflicts ; we decided to create a separate issue to fix the Creation-Date to the first day of the current year; this should hopefully not interfere with any translator flows, whilst reducing conflict headaches for us!