Create Diplomat translation submission interface

[sderickson]

Translations for levels are ingrained in the Level documents themselves. We need a way for Diplomats (that is, anyone who wants to) to work on these translations. So the goals for this issue are:

1. Make a way, both on the server and on the client, for Diplomats to submit translation changes to these public Levels without being able to also make changes to the rest of the parts of the Level document.
2. Minimize, prevent or handle conflicts if two or more Diplomats are submitting changes at the same time. Up until now, Levels would only ever be changed by their creators, and the infrequent times two people might be working on the same document at the same time they could easily let each other know, and so there's no system for handling the race condition where two people work on the same document at the same time and then each submitting changes, one overwriting the other. If an email goes out to a bunch of people that a new Level is ready for translating, it's much more likely this will happen.
3. Batch these changes where possible. Currently the versioning system simply copies the entire document when a change is made. This could possibly explode the number of documents in the database for a given Level, which are rather large documents, if a copy is made for each translation of each string. We don't have to minimize the number of changes, but a simple way to cut the magnitude of changes made, if it's not too difficult or time consuming, would be worth the time.

Here's one possible solution:

For each editor (Level, Thang, Article)

• When making changes, flag all documents being edited on Firebase.
• When the editor opens, check for these flags and notify the user that someone else is working on the given document, suggesting that they come back later.
• Create a tab for translations listing all translations and with form entries for them. This requires walking the whole document object looking for i18n properties, then displaying them in the view, and then gathering up the values given in the view and submitting them to the handler. Ideally this system would also know where to insert i18n properties using the document's schema.
• Set up the handler to be able to receive special put or post methods for submitting translations and only modifying those values. As part of finding the i18n properties to edit, keep the JSON paths so the handler can use these to find the subdocument for the translation and apply it, and only allow changes to paths that include i18n. It should also be able to fill out the i18n object as languages get submitted.

Other possible solutions involve:

• Create a chat room instead of the flags. More work but the existing Bus class is built to function as a chatroom for any page on the site. And this would make sense as a usage. Would probably still need flags for related documents in the Level editor, for Systems and Components.
• Hook up Treema to Firebase to make realtime collaboration on JSON objects. But that's crazy. Crazy awesome.
• Switch the version control system to diffs rather than copies. They would be easier to handle and view. That would be a major project though.
• Create a staging system. So patches would be submitted all the time but they would not be applied until... Something. Like the creator accepting them or at regular intervals.

[mquinson]

It seems to me that you are missing the part of the picture where a string that were already translated is updated in
English. When this happens, will you drop any transations? Keep them even if they may not fit the original anymore?

As a former leader of the Debian French translation team, I would advise you to use existing tools to tackle this issue. There is many solutions to edit po files out there, and they perfectly handle these issues (and some others).

Of course, the difficulty is then to bridge the po file format and the format you are currently using to actually use the translation strings. You may want to use po4a here (but my advice is biaised, I wrote this tool myself 10 years ago).

The idea would be that you produce a en.coffee file containing all the strings in english (I'm sure you already have it), and then you use po4a to extract these strings into pot file, and to produce up to date XX.coffee files with the translations from the XX.po files edited by translators. You need to write a new driver for po4a that can deal with coffee files, but I can help if you go this way.

Then, all problems of language update and everything is externalized. If you want to present a little interface to translate the po files online, that's very possible. Many similar tools exist, some free (pootle), some not (crowdin or transiflex).

That's just a suggestion here.
Mt

[GlenDC]

I noticed that diplomats sometimes make translations errors. They translate the words correctly but in the wrong context. So when this editor gets developed it's important that they instantly see their translations they are currently working on.

They should see it as a preview how it would look on the real page(s). This way they see the complete sentence and they won't make context-related translation mistakes anymore.

[phanimahesh]

I have some limited experience with translations management, and I agree with @mquinson. Using pot files and established translations workflow is very advisable.

We use launchpad translations for unity tweak tool, and I am very much satisfied with it. There is also a pool of Launchpad Translators whose team can be set as the review team, and they ensure good quality of our translations where no team member speaks the target language.

Since codecombat already has some translation workflow in place, the only missing link is a helper script to generate pot file with all translatable strings, and to convert the translated po files back to .coffee/.js files, which can be accomplished by writing a simple script.

[sderickson]

I talked with @nwinter about this and we decided that building out our existing system seems better than trying to work in pot and po files into the database objects. With the new patch system, we're already most of the way there, and we want translations to be integrated into the site where we can keep track of them and tie them in with a reward system, rather than external. The pot and po files also wouldn't be able to handle non-text translations (like audio files, or JSON objects) as well as what we've already got. We think the JSON/Treema system of having the translations integrated with the data will end up being the best way forward.

Given where we are four months after creating this issue and the patch system has been built recently, there are only a few things left to do to make editing database entities complete, and match or surpass a pot/po system:

• Have editors populate i18n objects in documents whenever changes are made, so they are present.
• Create a new view tab which takes a document and finds all i18n objects and lays them out for diplomats to translate, and to see existing translations (pending in patches or already saved in the document).
• Have editors mark translations as needing review when the base text or other data has changed.

These are basically all JSON-walking tasks and some simple interface building. This should handle the case when base text changes, and keep things integrated with the site, and with the patch system it shouldn't be hard for editors to come in and make improvements to existing translations, as @GlenDC mentioned would be useful. Regarding seeing these translations in context, that'll be a little harder, since you would need to play through the level entirely and hit all scripts in order to see all dialogue boxes in context (along with anything else translated), but having the translation system integrated with the editor would help by having it be possible to test the translations in context, whereas an external editor would not.

If this system works well, we may consider something similar for the site in general, where we have an interface object which can be edited by diplomats, and store the site strings in the database, rather than having them as files which everyone must load as part of app.js.

What do you guys think, @mquinson , @GlenDC , @phanimahesh ?

[phanimahesh]

The idea will definitely work, but I am very much biased towards the pot/po based translation system, partly because it works well and partly because it is time-tested.

But since you say this new system is mostly ready, and it also supports non-text translations, it sounds good. It would indeed be difficult to tie in the translations with a reward system if an external tool is used. These are two major use cases where the new system wins.

I did not quite understand what JSON objects need to be "translated". You mentioned them alongside audio files in non-text objects that need translations. Can you elaborate?

[GlenDC]

I agree with @sderickson and @nwinter on this one. Our system looks promising, and with enough effort this can really be something great. It will be really great when diplomats can just translate via our website on the fly. We already have many great translators that take time and effort do work their way through git and github. However, I'm sure that we'll reach many more translators, once they can just do their job via an editor on our website. It will also be a great test case for the patch system.

[mquinson]

I agree with @phanimahesh. You will certainly manage to get a great system working. I would have prefered a standard system where you improve the standard tools on need, but I understand that a fully integrated solution is also very promising.

At the end of the day, I'd love if codecombat would be perfectly translated to French so that my kids can play with it. My personal feelings about the implementation is not very important.

Thanks a lot for this work!

[sderickson]

@phanimahesh Basically in any case where the object in the JSON has two or more properties that are related that can be different between languages. The one example I can think of that happens now is an object that has the text and the audio file of a piece of dialogue. There may be other use cases in the future. For example, I can imagine a case where a JSON object represents a link, and both the text and the url need to be translated, such as to a wikipedia article for a topic. These are cases where the object as a whole should be translated, rather than each piece separately.

Thanks for your input guys!

[phanimahesh]

Sounds good. Good luck for the planned changes ahead. :)

[nwinter]

Thanks to some heroic hacking by @sderickson, It's ready! We are going to send out a big Diplomat email soon. Try it out now and let us hear your feedback and bugs: http://codecombat.com/i18n

@Imperadeiro98 @enricpc @Titounkle @kerradus @wakeup @AbnerZheng @Hamtara @gabrielcesar – would love for y'all to take it for a spin.

[Imperadeiro98]

It sounds awesome! Thank you @sderickson!!!
I'll get into it, when I can!

[gabrielcesar]

Very good! Nice work. ;)

On Wed, Oct 29, 2014 at 6:59 PM, Imperadeiro98 notifications@github.com
wrote:

It sounds awesome! Thank you @sderickson https://github.com/sderickson
!!!
I'll get into it!

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

Atenciosamente,

Gabriel Cesar
Eternity Tecnologia
www.eternity.com.br

[Titounkle]

I´ll also take a look as soon as i have time!!But at first sight looks
great!!Really nice work @sderickson

Cheers

2014-10-30 8:32 GMT+01:00 Gabriel Cesar notifications@github.com:

Very good! Nice work. ;)

On Wed, Oct 29, 2014 at 6:59 PM, Imperadeiro98 notifications@github.com
wrote:

It sounds awesome! Thank you @sderickson https://github.com/sderickson

!!!
I'll get into it!

—
Reply to this email directly or view it on GitHub
<
https://github.com/codecombat/codecombat/issues/65#issuecomment-61012652>
.

Atenciosamente,

Gabriel Cesar
Eternity Tecnologia
www.eternity.com.br

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

[Imperadeiro98]

I've now tried it out, and it is super!!! Way better than walking throw the level stuff!

[Hamtara]

It's very good and easier to use nice work @sderickson !

[sderickson]

Thanks, everyone!

[Hamtara]

I think I found a bug in the translation interface because when i select a language from the combobox and if then I click on word to translate I need reselect the language

[bitwalk]

If I do a translation with this tool, when/how does it go live? I'm trying to help with swedish translations.

[nwinter]

It goes live immediately, unless you translated something that doesn't show the translation yet (like the ThangType names only show up i18n in the new inventory and store views, which are basically done but not yet deployed). The stuff in the levels should show up right away, unless it thought it needed to submit a patch (like for a change to an existing translation) instead of just saving the change (for adding new translations).

Are there some translations you think should be showing up that aren't yet?

[bitwalk]

Yeah, it seems to work like you describe. I managed to translate first dungeon Dungeons of Kithgard. So now I can go ahead and translate the rest.

One more thing. How can I change translations of things I already translated that doesn't feel right in context?

Thanks!

[bitwalk]

Found now that previous translations end up in the bottom of the list.