Create automated workflow for translation of Markdown content

[mfc]

Transifex does not support Markdown:
http://docs.transifex.com/formats/

Do we need some type of process that converts Markdown to text, gets them translated, and then converts those texts back to Markdown? @axon-qubes @marmarek

An alternative localization platform to Transifex, Weblate, also doesn't support Markdown: https://docs.weblate.org/en/latest/formats.html

[mfc]

from Nina at Transifex support@transifex.com:

Hello Mike,

Thank you very much for your prompt reply and for providing me with
clarifications.

Regarding your question, I would suggest you use plain text files
(.txt) by simply changing the extension of your current .md files.
Then we will be able to upload them on Transifex.

After you finish the translation you can download them again as .txt
and change the extension to .md without losing any information.

Another possible workaround is to convert the .md files to one of the
currently supported file formats [1]. You can have a look at our
documentation and decide which of them fits you best.

I hope that the above answers your question. Please let me know if you
would like me to clarify anything else.

Regards,
Nina

Links:

[1] http://docs.transifex.com/formats/#supported-file-formats

[mfc]

from Nina at Transifex support@transifex.com:

Hello Mike,

Thank you very much for your prompt reply and for providing me with
clarifications.

Regarding your question, I would suggest you use plain text files
(.txt) by simply changing the extension of your current .md files.
Then we will be able to upload them on Transifex.

After you finish the translation you can download them again as .txt
and change the extension to .md without losing any information.

Another possible workaround is to convert the .md files to one of the
currently supported file formats [1]. You can have a look at our
documentation and decide which of them fits you best.

I hope that the above answers your question. Please let me know if you
would like me to clarify anything else.

Regards,
Nina

Links:

[1] http://docs.transifex.com/formats/#supported-file-formats

[andrewdavidwong]

I don't know how exactly Transifex works. (Their website seems to be a bit light on the details.) However, if we just need to provide .txt versions of all of our .md files, that sounds pretty easy:

$ cd qubesos.github.io/
$ shopt -s globstar
$ rename ".md" ".txt" **

But the next step ("get them translated," as you said). I have no idea what that entails.

[andrewdavidwong]

I don't know how exactly Transifex works. (Their website seems to be a bit light on the details.) However, if we just need to provide .txt versions of all of our .md files, that sounds pretty easy:

$ cd qubesos.github.io/
$ shopt -s globstar
$ rename ".md" ".txt" **

But the next step ("get them translated," as you said). I have no idea what that entails.

[mfc]

Okay, from my understanding it would be:

1. they upload these renamed text strings (.txt) to Transifex servers
2. this provides a nice UX for translators to translate these strings
3. we would download the translated text strings (.txt)
4. rename them to .md and upload them back to github.

Can you follow up on that email thread with Nina and Dragana and we can see how the process will be?

Also not sure if you have any ideas on what we should prioritize getting translated (documentation vs website vs qubes-specific code (like Qubes VM Manager). But for now we should test it out and see whether this is an efficient way to get things translated.

[mfc]

Okay, from my understanding it would be:

1. they upload these renamed text strings (.txt) to Transifex servers
2. this provides a nice UX for translators to translate these strings
3. we would download the translated text strings (.txt)
4. rename them to .md and upload them back to github.

Can you follow up on that email thread with Nina and Dragana and we can see how the process will be?

Also not sure if you have any ideas on what we should prioritize getting translated (documentation vs website vs qubes-specific code (like Qubes VM Manager). But for now we should test it out and see whether this is an efficient way to get things translated.

[andrewdavidwong]

Can you follow up on that email thread with Nina and Dragana and we can see how the process will be?

Yep, just included you in the reply.

Also not sure if you have any ideas on what we should prioritize getting translated (documentation vs website vs qubes-specific code (like Qubes VM Manager). But for now we should test it out and see whether this is an efficient way to get things translated.

For now, I've just included (almost) everything in the qubesos.github.io repo (and submodules) which is written in English.

Maybe @marmarek and @bnvk have ideas about which other components and modules should be prioritized for translation. (I assume the ideal thing would be for each user's entire Qubes experience to be in their native language.)

[andrewdavidwong]

Can you follow up on that email thread with Nina and Dragana and we can see how the process will be?

Yep, just included you in the reply.

Also not sure if you have any ideas on what we should prioritize getting translated (documentation vs website vs qubes-specific code (like Qubes VM Manager). But for now we should test it out and see whether this is an efficient way to get things translated.

For now, I've just included (almost) everything in the qubesos.github.io repo (and submodules) which is written in English.

Maybe @marmarek and @bnvk have ideas about which other components and modules should be prioritized for translation. (I assume the ideal thing would be for each user's entire Qubes experience to be in their native language.)

[mfc]

Yep, just included you in the reply.

Perfect, thanks.

I assume the ideal thing would be for each user's entire Qubes experience to be in their native language.

Exactly, that is the aim. Also why issues like this will need to be tackled: #1059

[mfc]

Yep, just included you in the reply.

Perfect, thanks.

I assume the ideal thing would be for each user's entire Qubes experience to be in their native language.

Exactly, that is the aim. Also why issues like this will need to be tackled: #1059

[adrelanos]

FYI: There is also https://translatewiki.net/ but I don't know much more
on that.

[adrelanos]

FYI: There is also https://translatewiki.net/ but I don't know much more
on that.

[mfc]

The Localization Lab team is still interested in helping translate the most important aspects of Qubes.

https://www.transifex.com/otf/qubes/

Regarding what to prioritize, Qubes VM Manager seems like a pretty small amount of context (text) compared to Qubes documentation. However within the website / documentation there may be some content that is higher prioritize. Also it is a lot easier to push updates to the website / docs than to push updates to the Qubes VM Manager.

I think we should start with something to get the translation community started. Also the Localization Lab just emailed out saying that they are going to do a Spanish translation sprint, which I think would be great for Qubes to participate in.

[mfc]

The Localization Lab team is still interested in helping translate the most important aspects of Qubes.

https://www.transifex.com/otf/qubes/

Regarding what to prioritize, Qubes VM Manager seems like a pretty small amount of context (text) compared to Qubes documentation. However within the website / documentation there may be some content that is higher prioritize. Also it is a lot easier to push updates to the website / docs than to push updates to the Qubes VM Manager.

I think we should start with something to get the translation community started. Also the Localization Lab just emailed out saying that they are going to do a Spanish translation sprint, which I think would be great for Qubes to participate in.

[andrewdavidwong]

I'm confused. We already sent all the Qubes docs to Nina back in November.

[andrewdavidwong]

I'm confused. We already sent all the Qubes docs to Nina back in November.

[mfc]

ah okay so what we're waiting on is for content to be translated, I'll start a thread with them and you.

[mfc]

ah okay so what we're waiting on is for content to be translated, I'll start a thread with them and you.

[bnvk]

Regarding what to prioritize, Qubes VM Manager seems like a pretty small amount of context (text) compared to Qubes documentation

True. Although, this is going to be a bit in-flux with the new manager that I am working on. I'll look into how to export strings at some point soon!

[bnvk]

Regarding what to prioritize, Qubes VM Manager seems like a pretty small amount of context (text) compared to Qubes documentation

True. Although, this is going to be a bit in-flux with the new manager that I am working on. I'll look into how to export strings at some point soon!

[mfc]

True. Although, this is going to be a bit in-flux with the new manager that I am working on. I'll look into how to export strings at some point soon!

we already shared a bunch of text strings with them, so no worries, the issue is getting people to translate them currently.

[mfc]

True. Although, this is going to be a bit in-flux with the new manager that I am working on. I'll look into how to export strings at some point soon!

we already shared a bunch of text strings with them, so no worries, the issue is getting people to translate them currently.

[andrewdavidwong]

@mfc: It's been almost a year since we first sent our docs to Transifex. Are you still in touch with them?

[andrewdavidwong]

@mfc: It's been almost a year since we first sent our docs to Transifex. Are you still in touch with them?

[mfc]

yeah i just talked with them, i think there was miscommunication where they thought they were waiting on us for something. i will try to push this forward again. maybe we should upload updated documentation to transifex given it's a year-old now.

[mfc]

yeah i just talked with them, i think there was miscommunication where they thought they were waiting on us for something. i will try to push this forward again. maybe we should upload updated documentation to transifex given it's a year-old now.

[mfc]

One note is that we can't have hard line breaks in our qubes-doc documentation (or webpages), as these make the translation strings break with each line instead of being each paragraph or sentence. I'm going to fix this manually for the pages we are most interested in getting translated for the upcoming spanish localization sprint (Intro, Getting Started, etc) but it would be nice for us to move towards that practice more generally.

I'm going to create a ticket about it so we can discuss +/- there.

[mfc]

One note is that we can't have hard line breaks in our qubes-doc documentation (or webpages), as these make the translation strings break with each line instead of being each paragraph or sentence. I'm going to fix this manually for the pages we are most interested in getting translated for the upcoming spanish localization sprint (Intro, Getting Started, etc) but it would be nice for us to move towards that practice more generally.

I'm going to create a ticket about it so we can discuss +/- there.

[mfc]

also, I can't find the ticket where we discussed URLs and web interface for language switching/choosing, it would be good to move on that so we can start testing out spanish content on the website in march.

[mfc]

also, I can't find the ticket where we discussed URLs and web interface for language switching/choosing, it would be good to move on that so we can start testing out spanish content on the website in march.

[andrewdavidwong]

Markdown conventions updated.

also, I can't find the ticket where we discussed URLs and web interface for language switching/choosing, it would be good to move on that so we can start testing out spanish content on the website in march.

The ticket is here: #1333

But I think you're remembering our recent email thread on this subject.

[andrewdavidwong]

Markdown conventions updated.

also, I can't find the ticket where we discussed URLs and web interface for language switching/choosing, it would be good to move on that so we can start testing out spanish content on the website in march.

The ticket is here: #1333

But I think you're remembering our recent email thread on this subject.

[andrewdavidwong]

As for fixing the actual line breaks, my dear friend @gsganden developed this handy tool: https://github.com/gsganden/sentence-separator

Perhaps it would be useful here.

[andrewdavidwong]

As for fixing the actual line breaks, my dear friend @gsganden developed this handy tool: https://github.com/gsganden/sentence-separator

Perhaps it would be useful here.

[mfc]

that tool looks perfect! can we test out moving everything to the new policy? then we can re-upload everything to transifex in one go.

[mfc]

that tool looks perfect! can we test out moving everything to the new policy? then we can re-upload everything to transifex in one go.

[andrewdavidwong]

@mfc: Depending on the number of false positives we get, it may take a lot of time and labor to convert and debug all the files. I, personally, don't have much time to work on this right now.

[andrewdavidwong]

@mfc: Depending on the number of false positives we get, it may take a lot of time and labor to convert and debug all the files. I, personally, don't have much time to work on this right now.

[mfc]

okay i think the priority is actually getting updated content onto transifex, we haven't updated the base content since 2015.

looking at our priorities for content for the localization sprint on March 5th, they are:

• vm manager
• the most looked-at/important docs/pages (listed in order of priority)
  • intro
  • user faq
  • glossary
  • getting started
  • mailing lists? (or as higher priority?)
  • doc e.g. the list of titles/names of docs? (https://www.qubes-os.org/doc/)

i have already (mostly) manually fixed the line wrapping for /intro in transifex, but moving to single sentences for translation rather than huge blocks of text would probably be preferred. so i will manually do that this week. there doesn't seem to have been any content change from a year ago for /intro. other priority pages are:

• /user-faq looks good in transifex, needs content updated
• /glossary looks good in transifex, needs content updated
• /getting-started looks good in transifex, needs content updated
• vm-managerlooks good in transifex, not 100% sure if content needs to be updated

[mfc]

okay i think the priority is actually getting updated content onto transifex, we haven't updated the base content since 2015.

looking at our priorities for content for the localization sprint on March 5th, they are:

• vm manager
• the most looked-at/important docs/pages (listed in order of priority)
  • intro
  • user faq
  • glossary
  • getting started
  • mailing lists? (or as higher priority?)
  • doc e.g. the list of titles/names of docs? (https://www.qubes-os.org/doc/)

i have already (mostly) manually fixed the line wrapping for /intro in transifex, but moving to single sentences for translation rather than huge blocks of text would probably be preferred. so i will manually do that this week. there doesn't seem to have been any content change from a year ago for /intro. other priority pages are:

• /user-faq looks good in transifex, needs content updated
• /glossary looks good in transifex, needs content updated
• /getting-started looks good in transifex, needs content updated
• vm-managerlooks good in transifex, not 100% sure if content needs to be updated

[andrewdavidwong]

okay i think the priority is actually getting updated content onto transifex, we haven't updated the base content since 2015.

As you know, I sent them an updated ZIP file last month. After some initial confusion, Erin's response was:

The resources can be updated manually from the zip, which I am happy to
help with, however the best way to keep everything up-to-date would be
through integration with Github or mapping all the resources to the
up-to-date source URL.

We don't have GitHub integration in place yet, so they should use the ZIP file for now.

i have already (mostly) manually fixed the line wrapping for /intro in transifex, but moving to single sentences for translation rather than huge blocks of text would probably be preferred. so i will manually do that this week.

Sounds good.

other priority pages are: [...]

Can you just download those files from GitHub and upload them to Transifex?

[andrewdavidwong]

okay i think the priority is actually getting updated content onto transifex, we haven't updated the base content since 2015.

As you know, I sent them an updated ZIP file last month. After some initial confusion, Erin's response was:

The resources can be updated manually from the zip, which I am happy to
help with, however the best way to keep everything up-to-date would be
through integration with Github or mapping all the resources to the
up-to-date source URL.

We don't have GitHub integration in place yet, so they should use the ZIP file for now.

i have already (mostly) manually fixed the line wrapping for /intro in transifex, but moving to single sentences for translation rather than huge blocks of text would probably be preferred. so i will manually do that this week.

Sounds good.

other priority pages are: [...]

Can you just download those files from GitHub and upload them to Transifex?

[mfc]

okay, thanks for the reminder of the zip. yeah i'll upload updated versions of those i mentioned (1-4) and ensure good line breaks. if you or others have preferences on what should be priority pages lemme know.

[mfc]

okay, thanks for the reminder of the zip. yeah i'll upload updated versions of those i mentioned (1-4) and ensure good line breaks. if you or others have preferences on what should be priority pages lemme know.

[mfc]

done. we have vm manager, intro, user faq, glossary, and getting started. I think that's enough for localization sprint on sunday.

[mfc]

done. we have vm manager, intro, user faq, glossary, and getting started. I think that's enough for localization sprint on sunday.

[mfc]

FYI this was a resource that came out of the event: https://docs.google.com/spreadsheets/d/1mT7rAvW0V6Ia7C_qfHIIwyRuIW8I9RWK0Vxm4b4V2lc/edit

[mfc]

FYI this was a resource that came out of the event: https://docs.google.com/spreadsheets/d/1mT7rAvW0V6Ia7C_qfHIIwyRuIW8I9RWK0Vxm4b4V2lc/edit