Manage help page translations

[dittaeva]

Current handling of help page translations does not scale. It needs to be handled by a translation management system.

To manage translations on Translatewiki, they need to be stored in a format that the Translate extension of Mediawiki can handle, see thread at Translatewiki.

django-easymode offers translation of database content, but currently that's not where help page content is stored.

[lonvia]

Storing the help pages in the database is not an option because then it becomes a pain to transfer between machines and there is neither backup nor versioning.

The current system was devised to make it as comfortable as possible for direct translation in the repository. If translation is moved to TranslateWiki, we could simply make the help pages Django templates with each paragraph in a blocktrans section and have the help pages appear in the .po files just like anything else. Just need to figure out how to make that all work with markdown.

[dittaeva]

I was thinking along the same lines. I'll ask Translatewiki if it is possible to keep interface messages and help page messages separate though, because the interface messages are much more important, so it should be easy to prioritise them.

[dittaeva]

If using gettext for the help pages we would almost have to be able to keep them separate from the other messages, or it would be difficult for translators to prioritise UI messages. I've asked about keeping them separate at Translatwiki.

[dittaeva]

Here's a script to combine message files before compiling. Making separate message files in the first place doesn't look to easy to do cleanly though. One can make message files per file extension (-e), and ignore files and directories (-i).

I suppose one could make a script that would run makemessages twice, first ignoring the help pages, renaming the message files, then ignoring everything but the help pages and renaming the message files. Then you'd run the above mentioned script to compile messages.

[lonvia]

Yeah, saw that the gettext route is going to be a pain. So, now I'm thinking to convert the current help files into a YAML/markdown format and keep the help infrastructure in place. It would have the advantage that no additional compilation magic is needed and the changes should be minimal. I'll write up a prototype this weekend.

[lonvia]

I've pushed a prototype for the YAML/markdown solution into branch translatable-help.

@siebrand The file to translate would be https://github.com/lonvia/waymarked-trails-site/blob/translatable-help/django/locale/qot/helppages.yaml. It isn't complete but hopefully gives you an idea. Is it usable for TranslateWiki in this way? The translated files should go into django/locale/<LANG>/helppages.yaml. It would be important that the line-breaks are preserved correctly because of the markdown format.

[Nikerabbit]

I roundtripped that example file through our Yaml parser/generator and it ended up like this: http://pastebin.com/DMPzScNL

Looks like the line breaks are preserved, but the output for translations isn't as pretty (or even in the same order) as it is for the source. That should be okay if you don't rely on the exact formatting of the Yaml file itself (instead of only the content) - is this the case?

Bigger problem is that MediaWiki doesn't support trailing whitespace on messages without kludges, so those would need special handling.

[lonvia]

I tried your generated yaml and it is processed just fine, line-breaks and all. Order does not matter at all, nor does it have to be pretty, as long as the template file in the qot directory stays as it is.

What happens with strings that have not yet been translated? Does the original string appear in the translated yaml file or will the entry be missing completely?

As for the trailing spaces, in markdown they are mandatory only for code blocks and for paragraphs in lists. If it makes MediaWiki happy and saves you some work, then we can do without those constructs in the help pages.

[siebrand]

Op 14 mei 2012 om 21:12 heeft Sarah Hoffmann reply@reply.github.com het volgende geschreven:

What happens with strings that have not yet been translated? Does the original string appear in the translated yaml file or will the entry be missing completely?

It will be absent. Can it put in the Englisg string instead, I.e. regular fallback behavior?

As for the trailing spaces, in markdown they are mandatory only for code blocks and for paragraphs in lists. If it makes MediaWiki happy and saves you some work, then we can do without those constructs in the help pages.

MediaWiki strips trailing white space, so we just cannot handle that (spaces, new lines, tabs, ...).

[lonvia]

On Mon, May 14, 2012 at 02:32:21PM -0700, Siebrand Mazeland wrote:

Op 14 mei 2012 om 21:12 heeft Sarah Hoffmann reply@reply.github.com het volgende geschreven:

What happens with strings that have not yet been translated? Does the original string appear in the translated yaml file or will the entry be missing completely?

It will be absent. Can it put in the Englisg string instead, I.e. regular fallback behavior?

No problem, can be easily implemented.

As for the trailing spaces, in markdown they are mandatory only for code blocks and for paragraphs in lists. If it makes MediaWiki happy and saves you some work, then we can do without those constructs in the help pages.

MediaWiki strips trailing white space, so we just cannot handle that (spaces, new lines, tabs, ...).

I'm a bit confused now. Can or can it not preserve new lines? Without correct line breaks markdown becomes rather pointless. Spaces really don't matter as long as it doesn't add arbitrary spaces at the beginning of the line.

I've checked the Potlach2 help page translation and there preservation of both spaces and newline seems to work just fine. I see them in the translation interface and in the final output in the Potlach2 repository. (The newlines are \n but that is sufficient.)

There is still the option to break the text into even smaller pieces, so that even new lines are no longer required. But that would mean that translators have a much harder time producing coherent text.

[siebrand]

On Tue, May 15, 2012 00:23, Sarah Hoffmann wrote:

On Mon, May 14, 2012 at 02:32:21PM -0700, Siebrand Mazeland wrote:

Op 14 mei 2012 om 21:12 heeft Sarah Hoffmann reply@reply.github.com
het volgende geschreven:

MediaWiki strips trailing white space, so we just cannot handle that
(spaces, new lines, tabs, ...).

I'm a bit confused now. Can or can it not preserve new lines? Without
correct line breaks markdown becomes rather pointless. Spaces really don't
matter as long as it doesn't add arbitrary spaces at the beginning of the
line.

This is only about trailing whitespace at the end of the string (not on
an intermediate line).

[lonvia]

On Mon, May 14, 2012 at 03:27:11PM -0700, Siebrand Mazeland wrote:

This is only about trailing whitespace at the end of the string (not on
an intermediate line).

Right, I should learn to read, sorry. Those are not relevant at all.

[dittaeva]

@siebrand @Nikerabbit lonvia has pushed the template for helppage translations now. The template is in django/locale/qot/helppages.yaml and translations should go to django/locale/<LANG>/helppages.yaml

We haven't moved existing translations yet, and think it might as well be done manually on translatewiki because of tedious formatting etc. (the originals aren't in markdown). I'll update the documentation shortly, and when you've have it up and running on Translatewiki, I'll move the nn and nb translations at least.

Interface translations and help page translations should be separate. Based on a message count of 38, where many are quite short, I think the inclusion threshold should be 40%.

[dittaeva]

I've made a post on Translatewiki support about this issue.

[dittaeva]

I had a chat with Translatewiki support, and they are concerned that duplication of work could result from adding this to Translatewiki without including the existing (old) translations, since translators will not know about the old translations. They could be made aware, but I agree, we can't really expect the Translatewiki community to do the conversion/porting for us, and it isn't really easier to port using the translatewiki interface.

Will you take care of German and Spanish @lonvia?

[jschleic]

I could do the rst->yaml conversion for german tomorrow, if you like.

[dittaeva]

Great. Then German is assigned to you Joachim unless Sarah says otherwise.
On 2012 8 15 00:03, "Joachim Schleicher" notifications@github.com wrote:

I could do the rst->yaml conversion for german tomorrow, if you like.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-7742001.

[lonvia]

I'll have a look at the Spanish translation then but it will be full of omissions where the text has changed.

[lonvia]

I've now committed a Spanish translation that should contain whatever was salvageable from the previous translation.

[dittaeva]

Help page translation is now live on Translatwiki.net and I expect @siebrand will be pushing the first languages that have passed 40 % (ms, fr, gl, da) with the next push.

[siebrand]

On Thu, Sep 20, 2012 at 7:17 AM, Guttorm Flatabø
notifications@github.comwrote:

Help page translation is now live on Translatwiki.net and I expect
@siebrand https://github.com/siebrand will be pushing the first
languages that have passed 40 % (ms, fr, gl, da) with the next push.

I just asked Niklas, and he indicated that the maintenance scripts still
have to be updated. Once done, you can expect updates every weekend.

[dittaeva]

@siebrand ok, do you have those scripts in git somewhere too? I can't find them...

[siebrand]

On Thu, Sep 20, 2012 at 1:19 PM, Guttorm Flatabø
notifications@github.comwrote:

@siebrand https://github.com/siebrand ok, do you have those scripts in
git somewhere too? I can't find them...

It's "repo*" in
https://gerrit.wikimedia.org/r/gitweb?p=translatewiki.git;a=tree;f=bin;h=50044435a625bc605a0b1c268c1daf7c3580c734;hb=HEAD