Add the ability to translate submissions

[wes-otf]

Pertaining to discussions had in #4037. This integrates the argostranslate library to allow for translation of applications. The current functionality allows staff users to translate the written answers of an application.

This feature utilizes htmx to translate the application without requiring a reload of the page, while also updating the URL so a user can have a sharable link (using the query parameters fl [from language] & tl [to language]) when collaborating with others on the application.

This feature is disabled by default & can be enabled via the SUBMISSION_TRANSLATIONS_ENABLED setting

Demos

Translating an application

Screen.Recording.2024-09-13.at.14.33.16.mp4

Clearing a translation via the form

Screen.Recording.2024-09-13.at.14.52.45.mp4

Test Steps

• …

[wes-otf]

Lots of work needed before this is ready, like:

• Better code documentation
• Cleaning/organizing initial PoC code (ie. the JS form validation/choice population in translate_application_form.html)
• Server side form validation for translate_application_form.html
• Lots & lots of unit tests
• Management command (or migration?) to install new language packages
• Perfecting formatting on language form
• Better logic for parsing & replacing translations in HTML

So far happy with the initial pass though! OTF is on board already as this is something they really hate having to keep using an external AI type service for.

[wes-otf]

if anyone wants to play around with this, to install language packages you can use the following script (will install the Arabic -> English package but from_code & to_code can be changed to whatever)

import argostranslate.package
import argostranslate.translate

from_code = "ar"
to_code = "en"

# Download and install Argos Translate package
argostranslate.package.update_package_index()
available_packages = argostranslate.package.get_available_packages()
package_to_install = next(
    filter(
        lambda x: x.from_code == from_code and x.to_code == to_code, available_packages
    )
)
argostranslate.package.install_from_path(package_to_install.download())

[frjo]

I think the UI looks really nice. Neat to make the to/from lang a query string so it can be shared.

How would downloading the language package work on production?

[wes-otf]

I was thinking via a management command that way the instance could add languages as needed. Still thinking through what that syntax would look like but maybe something like:

python3 manage.py install_language from_to ...

ie. to install the arabic -> english & french -> english package it'd look like:

python3 manage.py install_language ar_en fr_en

[wes-otf]

The previously mentioned management commands were added - had a long layover today so went a little overboard w/ verbosities and messages (ie. packages{'s were' if len(existing_packages) > 1 else ' was'}) bull overall seems to work pretty well!

Installing English -> Arabic & English -> Spanish:

python3 manage.py install_languages ar_en fr_en

Uninstalling said languages:

python3 manage.py uninstall_languages ar_en fr_en

[wes-otf]

I think this is ready for review! Added tons of unit tests - might've gone a little overboard with mocking but really wanted logic testing to be isolated. Let me know what y'all think!

[frjo]

Works well in my testing and I like the UI.

Two observations:

1. It is quite CPU intensive, I run it on an M2 and it is not fast.
2. I get the warning below. Anything we need to worry about?

/Users/frjo/Sites/hypha/.venv/lib/python3.12/site-packages/stanza/models/tokenize/trainer.py:85: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(filename, lambda storage, loc: storage)