i18n workflow: force compile fuzzy

[mb-wali]

In order to compile fuzzy marked strings, we need to force it.
todoso we have to add this line use-fuzzy = True to the [compile_catalog] section of setup.cfg file.

This should be added to the following modules:

• invenio-i18n
• invenio-app-rdm
• invenio-access
• invenio-accounts
• invenio-app
• invenio-assets
• invenio-base
• invenio-cache
• invenio-celery
• invenio-communities
• invenio-config
• invenio-db
• invenio-drafts-resources
• invenio-config
• invenio-formatter
• invenio-files-rest
• invenio-iiif
• invenio-indexer
• invenio-jsonschemas
• invenio-logging
• invenio-mail
• invenio-oaiserver
• invenio-oauth2server
• invenio-oauthclient
• invenio-pidstore
• invenio-previewer
• invenio-rdm-records
• invenio-records
• invenio-records-files
• invenio-records-permissions
• invenio-records-resources
• invenio-records-rest
• invenio-rest
• invenio-saml
• invenio-search-ui
• invenio-theme
• invenio-userprofiles
• invenio-vocabularies
• cookiecutter-invenio-rdm
• cookiecutter-invenio-module
• marshmallow-utils
• invenio-requests
• invenio-users-resources

Close related issue #inveniosoftware/invenio-app-rdm/issues/230

[chriz-uniba]

Can we check the status of this issue?

1. based on this list it seems, that we have packages where we force fuzzy and packages, where this is not the case. Is this information still correct?

2. since this lists is already some days old, I remember, that there were new packages added, what's the status with these? The documentation mentions the fuzzy: https://inveniordm.docs.cern.ch/develop/howtos/i18n/#configuration
   Related: (transifex) make a list of modules that invenioRDM - uses #99

3. are we sure that we need/want to force it?

   • Maybe there is a better way, by fixing fuzzy (when using transifex)?

• I can/will try to check transifex in the upcoming weeks.

• I would be happy to get some opinions/experience of the force-use.
  Maybe: @utnapischtim @rekt-hard @zzacharo

• When discussed point 3. I could try to get the changes in all packages.

[rekt-hard]

1. All packages which contain translatable strings should have the config. In the above list, not all packages have translations, which is probably the reason why not everything is checked. We can go through the packages and check.
2. New packages should have force fuzzy enabled by default. Still should check though.
3. It would probably require to go back and analyze where the fuzzy is being added and why. Then, we should also check which errors would be caught if we do not force it and if it affects us.

[chriz-uniba]

I think earliest end of September I can investigate that.

[chriz-uniba]

wrt. point 3:

• as described here i18n workflow: translation files (.mo) are not being created invenio-app-rdm#230

• and tested and read in the docs:

• at least, as long as we have the header we get the fuzzy

• an alternative for this header-fuzzy is to use python setup.py extract_messages --omit-header

• but then we loose the header information

• I didn't check whether there are other ways how fuzzy could get in our files

• the moment a file contains fuzzy this file will not compiled at all anymore - without any warning or error messages (others are complaining about this, too)

• this makes it hard to detect the issue

• therefore, using fuzzy in setup.cfg sounds the best way to go, for now:

  [compile_catalog]
  use-fuzzy = True   
  

wrt. point 2:

• the documentation mentions the fuzzy: https://inveniordm.docs.cern.ch/develop/howtos/i18n/#configuration

wrt. point 1:

I will try go through all packages and check for the fuzzy and report back.

[chriz-uniba]

Starting with the following:

from Pipfile.lock with invenio-app-rdm "=12.0.0b2.dev35"
all packages containing invenio in their name

checked | has fuzzy | okay without fuzzy

• ❌ ✔️ flask-collect-invenio | https://github.com/inveniosoftware/flask-collect-invenio/blob/master/setup.cfg | no translation
• ❌ ✔️ flask-kvsession-invenio | https://github.com/inveniosoftware/flask-kvsession/tree/master | no translation
• ❌ ❌ flask-security-invenio | https://github.com/inveniosoftware/flask-security-fork/blob/9541bbe7db618eae795bda772729514929c39c9d/setup.cfg#L61 | has translations
• ✔️ invenio-access | https://github.com/inveniosoftware/invenio-access/blob/a0da831b79232f3f912fef85f7a8bc2b6297d4b9/setup.cfg#L85
• ✔️ invenio-accounts | https://github.com/inveniosoftware/invenio-accounts/blob/3caced556fdc5fe5737416e8672fa8e4c03cdb7f/setup.cfg#L99
• ❌ ❌ invenio-admin | https://github.com/inveniosoftware/invenio-admin/blob/fab6dd3024ddb6dfe6f37d498b4efa7f3b8d3ac6/setup.cfg#L66 | has translation folders
• ❌ ❌ invenio-administration | https://github.com/inveniosoftware/invenio-administration/blob/0680e1856d901fdf8f42c227446a18ee490b1429/setup.cfg#L73 | has translation folders
• ❌ ✔️ invenio-app | https://github.com/inveniosoftware/invenio-app/blob/master/setup.cfg | no translation
• ✔️ invenio-app-rdm | https://github.com/inveniosoftware/invenio-app-rdm/blob/92c0e3b4b8807d773b1689e362e165ea2732292f/setup.cfg#L127
• ❌ ✔️ invenio-assets | https://github.com/inveniosoftware/invenio-assets/blob/master/setup.cfg | no translation
• ✔️ invenio-banners | https://github.com/inveniosoftware/invenio-banners/blob/30aa60b3d0863b59241e3949e24f08cb5737e751/setup.cfg#L71
• ❌ ✔️ invenio-base | https://github.com/inveniosoftware/invenio-base/blob/master/setup.cfg | no translation
• ❌ ✔️ invenio-cache | https://github.com/inveniosoftware/invenio-cache/blob/master/setup.cfg | no translation
• ❌ ✔️ invenio-celery | https://github.com/inveniosoftware/invenio-cache/blob/master/setup.cfg | no translation
• ✔️ invenio-communities | https://github.com/inveniosoftware/invenio-communities/blob/e34ae2a6b41ba0ecd8ae3bd88c595d6281369b94/setup.cfg#L105
• ❌ ✔️ invenio-config | https://github.com/inveniosoftware/invenio-config/blob/master/setup.cfg | no translation
• ❌ ✔️ invenio-db | https://github.com/inveniosoftware/invenio-db/blob/master/setup.cfg | no translation
• ✔️ invenio-drafts-resources | https://github.com/inveniosoftware/invenio-drafts-resources/blob/84ca71eebdc84483863b1cc3a2fc98622a6b9ad2/setup.cfg#L69
• ❌ ❌ invenio-files-rest | https://github.com/inveniosoftware/invenio-files-rest/blob/master/setup.cfg | has translations
• ✔️ invenio-formatter | https://github.com/inveniosoftware/invenio-formatter/blob/9cbb92559cafde9065a8ba4be209879d4d959c3e/setup.cfg#L61
• ❌ ❌ invenio-github | https://github.com/inveniosoftware/invenio-github/blob/master/setup.cfg | has translations
• ✔️ invenio-i18n |

  invenio-i18n/setup.cfg

  Line 64 in 287849c

  use-fuzzy = True

• ❌ ❓ invenio-indexer | https://github.com/inveniosoftware/invenio-indexer/blob/master/setup.cfg | has no translation, but [compile_catalog] flag
• ❌ ✔️ invenio-jsonschemas | https://github.com/inveniosoftware/invenio-jsonschemas/blob/master/setup.cfg | has no translations
• ❌ ✔️ invenio-logging | https://github.com/inveniosoftware/invenio-logging/blob/master/setup.cfg | has no translations
• ❌ ✔️ invenio-mail | https://github.com/inveniosoftware/invenio-mail/blob/master/setup.cfg | has no translations
• ✔️ invenio-notifications | https://github.com/inveniosoftware/invenio-notifications/blob/cfd1e70d792177ba605f55cb53e6571de0e5291a/setup.cfg#L86
• ✔️ invenio-oaiserver | https://github.com/inveniosoftware/invenio-oaiserver/blob/f9650d7b31613eb01cc6b7ce1c953b8076cf60ea/setup.cfg#L90
• ✔️ invenio-oauth2server | https://github.com/inveniosoftware/invenio-oauth2server/blob/b469dc5092fa4a02273f00d87f868d30cf99ebce/setup.cfg#L96
• ✔️ invenio-oauthclient | https://github.com/inveniosoftware/invenio-oauthclient/blob/7c768f2808bfe990bd62f916b77087f4895e0b53/setup.cfg#L99
• ❌ ❌ invenio-pages | https://github.com/inveniosoftware/invenio-pages/blob/master/setup.cfg | has translations
• ✔️ invenio-pidstore | https://github.com/inveniosoftware/invenio-pidstore/blob/d54359c185c92f87decc8b9ba951ed03790226f6/setup.cfg#L81
• ❌ ❌ invenio-previewer | https://github.com/inveniosoftware/invenio-previewer/blob/master/setup.cfg | has translations
• ❌ ✔️ invenio-queues | https://github.com/inveniosoftware/invenio-queues/blob/master/setup.cfg | no translations
• ✔️ invenio-rdm-records | https://github.com/inveniosoftware/invenio-rdm-records/blob/3f542b8daca944b4421dec80c0add0eab9157e58/setup.cfg#L140
• ✔️ invenio-records | https://github.com/inveniosoftware/invenio-records/blob/0ee633e3e9419b1d9a3963b36713ed160def6537/setup.cfg#L86
• ❌ ✔️ invenio-records-files | https://github.com/inveniosoftware/invenio-records-files/blob/master/setup.cfg
• ✔️ invenio-records-permissions | https://github.com/inveniosoftware/invenio-records-permissions/blob/2492d97b33c3ae2f5225c5060d0edf9527f0588d/setup.cfg#L71
• ✔️ invenio-records-resources | https://github.com/inveniosoftware/invenio-records-resources/blob/3f188b7a623a1faf53abd29ce65f2d6ccf6cb1b6/setup.cfg#L90
• ❌ ❓ invenio-records-rest | https://github.com/inveniosoftware/invenio-records-rest/blob/master/setup.cfg | has kind of translations: https://github.com/inveniosoftware/invenio-records-rest/blob/master/invenio_records_rest/translations/en/LC_MESSAGES/messages.po
• ❌ ❓ invenio-records-ui | https://github.com/inveniosoftware/invenio-records-ui/blob/master/setup.cfg | has kind of translations: https://github.com/inveniosoftware/invenio-records-ui/blob/master/invenio_records_ui/translations/messages.pot
• ✔️ invenio-requests | https://github.com/inveniosoftware/invenio-requests/blob/d885d3357da315978a3d5330c8cda655f241812b/setup.cfg#L92
• ❌ ✔️ invenio-rest | https://github.com/inveniosoftware/invenio-rest/blob/master/setup.cfg | has no translation
• ❌ ✔️ invenio-search | https://github.com/inveniosoftware/invenio-search/blob/master/setup.cfg | no translations
• ✔️ invenio-search-ui | https://github.com/inveniosoftware/invenio-search-ui/blob/6969c79fada305deb525c5a2b684fa2b5459de4f/setup.cfg#L63
• ❌ ✔️ invenio-stats | https://github.com/inveniosoftware/invenio-stats/blob/master/setup.cfg | no translations
• ✔️ invenio-theme | https://github.com/inveniosoftware/invenio-theme/blob/fc988c13b4f269f59b808ff218891b166c439eba/setup.cfg#L63
• ✔️ invenio-userprofiles | https://github.com/inveniosoftware/invenio-userprofiles/blob/7d7bd84c452636fab421c99ce0f9f3ea691c6d76/setup.cfg#L73
• ✔️ invenio-users-resources | https://github.com/inveniosoftware/invenio-users-resources/blob/dbb9126b725b26501f4921a7d73fe8e15a3d2ea8/setup.cfg#L83
• ✔️ invenio-vocabularies | https://github.com/inveniosoftware/invenio-vocabularies/blob/45856c6adf3de330a4e240dc74c45a428bbbedf6/setup.cfg#L111
• ✔️ invenio-webhooks | https://github.com/inveniosoftware/invenio-webhooks/blob/156180ab4459fde3d197f9d64d7ad43f75bc86d6/setup.cfg#L103

[chriz-uniba]

based on this:

have fuzzy:

• invenio-access
• invenio-accounts
• invenio-app-rdm
• invenio-banners
• invenio-communities
• invenio-drafts-resources
• invenio-formatter
• invenio-i18n
• invenio-notifications
• invenio-oaiserver
• invenio-oauth2server
• invenio-oauthclient
• invenio-pidstore
• invenio-rdm-records
• invenio-records
• invenio-records-permissions
• invenio-records-resources
• invenio-requests
• invenio-search-ui
• invenio-theme
• invenio-userprofiles
• invenio-users-resources
• invenio-vocabularies
• invenio-webhooks

no fuzzy but likely no problem:

• flask-collect-invenio
• flask-kvsession-invenio
• invenio-app
• invenio-assets
• invenio-base
• invenio-cache
• invenio-celery
• invenio-config
• invenio-db
• invenio-jsonschemas
• invenio-logging
• invenio-mail
• invenio-queues
• invenio-records-files
• invenio-rest
• invenio-search
• invenio-stats

no fuzzy and more likely a problem:

• invenio-files-rest
• invenio-github
• flask-security-invenio
• invenio-admin
• invenio-administration
• invenio-pages
• invenio-previewer

no fuzzy and maybe a problem:

• invenio-indexer
• invenio-records-rest
• invenio-records-ui

not checked

• cookiecutter-invenio-rdm
• cookiecutter-invenio-module
• marshmallow-utils
• react-searchkit

[chriz-uniba]

wrt 'no fuzzy but likely no problem:'

within each repo searched for _(, format, gettext, I18N, i18next, Trans

result:

flask-collect-invenio:
- found: python 'format'
- but no traces of translation intents

flask-kvsession:
- found _(...) in docs-folder
- but no traces of translation intents

invenio-app:
- found: 'gettext' and 'I18N' in documentation
- found: python 'format' in test_helpers
- but no traces of translation intents


invenio-assets:
- found: 'gettext' and 'I18N' in docs
- found: python 'format'
- but no traces of translation intents

invenio-base:
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-cache:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-celery:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-config:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-db:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-jsonschemas:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-logging:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-mail:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-queue:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents
- .tx/config:9:# TODO: Transifex integration

invenio-records-files:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-rest:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-search:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents

invenio-stats:
- found: python 'format'
- found: 'gettext' and 'I18N' in docs
- but no traces of translation intents
- .tx/config:9:# TODO: Transifex integration

[chriz-uniba]

Additionally found: https://github.com/inveniosoftware/cookiecutter-invenio-module/blob/6decff43538c9e317175bfed33878fe7b0843a53/%7B%7B%20cookiecutter.project_shortname%20%7D%7D/setup.cfg#L80

Conclusion:

• waiting for the PRs to get merged
• there is the possibility that some packages do not enable translation, although text will get to the user (maybe invenio-stats?). However, it is hard to see, which sentences will be seen by whom.
• Therefore, we will wait and check in the gui whether untranslated sentences will show up anywhere and fix it then.

[chriz-uniba]

All PRs are merged. For the moment we should be good. Let's close it. @zzacharo