Separate PO and Gettext into separate packages

[kipcole9]

I am working on an ICU Message formatter and localisation processor as the (maybe) last part of the ex_cldr. One requirement is of course to store and manage translations and PO files are clearly the defacto globally standard.

I would like to propose splitting Gettext into as PO package that looks after storage, parsing, merging and extracting. That way different localisation packages can leverage the same infrastructure - which is supported by the format tag in a PO file. As you know, today Gettext saves all messages with the format-elixir tag.

What would need to change:

1. API and usage for current Gettext users would need to remain identical for backwards compatibility

2. The extractor would need to be told hold to tag messages so they aren’t all tagged as format-elixir

3. The backend interface would be normalised. Instead of BackendModule.__gettext__(:priv) for example, it would be better to have BackendModule.__po__(:priv) in order to be more specific and not tied to the Gettext API alone

4 Compiling PO files would compile messages based upon the message format. Perhaps a behaviour would be introduced so that any message format could be compiled. Instead of use Gettext, .... it would become use PO, icu_format: MyMessageFormatter, elxiir_format: Gettext or some such. Then use Gettext, ... would also include use PO, ....

5 Lastly, this may open up the opportunity to have different storage models, like DB, or :persistent_term or even a web service that can replace PO files while still retaining the well known format.

I’m fine to work on this - its better than creating another PO file parser/merger/extracted as @tmbb as already had to do,

Thoughts?

[tmbb]

I’m fine to work on this - its better than creating another PO file parser/merger/extracted as @tmbb as already had to do

I actually think Gettext could use a new .po parser

[josevalim]

Agreed on a new parser but that is just part of the work. We can probably re-use everything else. I think Andrea considered a nimble_parsec-based PO parser but I am not sure of the status. Contributions are definitely welcome!

I am also completely happy to go in the direction proposed here. My suggestion is to start adding the new API directly to this repository. We will introduce new modules, move the API over, and once we are happy with the results, we extract it to a separate project. By then, I think we should also create an elixir-gettext org but no need to worry about it for now.

What do you think?

[tmbb]

I believe I have a functional .po parser based on nimble_parsec already. I have to look for it.

…

On Fri, 6 Sep 2019 at 17:14, José Valim ***@***.***> wrote: Agreed on a new parser but that is just part of the work. We can probably re-use everything else. I think Andrea considered a nimble_parsec-based PO parser but I am not sure of the status. Contributions are definitely welcome! I am also completely happy to go in the direction proposed here. My suggestion is to start adding the new API directly to this repository. We will introduce new modules, move the API over, and once we are happy with the results, we extract it to a separate project. By then, I think we should also create an elixir-gettext org but no need to worry about it for now. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#215>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAULSSUVNW2GLTI6YHIHQSTQIJ6XHANCNFSM4IUJDZFA> .

[kipcole9]

Thanks for the feedback, I'll start work on the functional separation first (decoupling PO and Gettext). Then aim to integrate a new PO parser. And finally package separation, possible aligned with a new elixir-gettext organization.

As José says, the parsing part is only a bit of what PO does today, albeit an important part.

[tmbb]

As promised, the gettext parser: https://github.com/tmbb/mezzofanti/blob/master/lib/mezzofanti/gettext/gettext_parser.ex

Tests here: https://github.com/tmbb/mezzofanti/blob/master/test/mezzofanti/gettext/gettext_parser_test.exs (the parser parses the file into a list of Translation structs, which can be swapped for something else that's more convenient to gettext. I believe I support all of the syntax of .po files, but it's been a couple of months since I've first written this and maybe I missed something. It's not optimized for error reporting, but it shouldn't be too hard to add.

EDIT: It turns out I had already mentioned the parser in this thread: #89 (comment)

[josevalim]

Fantastic! @tmbb would it be an option to swap in the new parser in place of the current parser but keeping the same return API? Then we can migrate it in steps and make sure nothing is lost on the way. Then the next step would be to replace the structs from Gettext.Translation to PO.Translation and similar. Btw, what should we call the project? Is po good enough?

[tmbb]

would it be an option to swap in the new parser in place of the current parser but keeping the same return API?

Sure, it's just a question of changing this function: https://github.com/tmbb/mezzofanti/blob/master/lib/mezzofanti/gettext/gettext_parser.ex#L28-L31

Btw, what should we call the project?

Expo?

[tmbb]

OTOH, if the .po parser is meant to be reused it should return its own struct and users (such as gettext) should convert from %PoTranslation{} into whatever they want. What do you think?

[tmbb]

I would like to propose splitting Gettext into as PO package that looks after storage, parsing, merging and extracting. That way different localisation packages can leverage the same infrastructure - which is supported by the format tag in a PO file. As you know, today Gettext saves all messages with the format-elixir tag.

@kipcole9 How would you deal with the fact that ICU makes directives such as

msgid_plural "%d pages read."
msgstr[0] "Eine Seite gelesen wurde."
msgstr[1] "%d Seiten gelesen wurden."

useless? Should my po parser parse these? This might make the po parser not very reusable... Currently my parser doesn't recognize this directive (nor does it concatenates multiple strings representing multiline strings, but that's easy to add and doesn't pose many problems)

[tmbb]

Parser in its own repo: https://github.com/tmbb/ex_po

[josevalim]

I like the name Expo! One word though. :D Regarding the structs, we can already change the structs in Gettext. I just think doing the changes inside gettext first is the easiest and simplest way to avoid breaking changes. -- *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D*

[whatyouhide]

FWIW, I have a working nimble-based parser too, here.

msgid_plural "%d pages read."
msgstr[0] "Eine Seite gelesen wurde."
msgstr[1] "%d Seiten gelesen wurden."

[...] useless?

@tmbb I'm not familiar with ICU. What is useless about this?

[tmbb]

In ICU messages, you encode the singular and the plural in te same string, as well as gammatical gender and other things. For example look at the example here: http://userguide.icu-project.org/formatparse/messages

[tmbb]

I've created some translation structs inside my ExPo project. I think that is gettext is going to be based on ExPo, it should use ExPo's structs, which AFAIK are backward compatible. There are still some things that need to be changed in ExPo (namely, I need to support more escape chars in strings and other such minor things), but it's in a pretty good shape already.

[josevalim]

Sounds good to me! Can you please send a PR for us to see how it would look like? Btw, what do you think about naming it Expo (app: expo)? Calling it ExPo makes it sound like "I just prepended things with Ex" but calling it Expo makes it sound like a pun on "exposition", which in my humble opinion is much better. :P