Remove forced line-breaking in qubes-doc

[mfc]

This breaks our ability to move content seamlessly to our translation platform Transifex and back: #1452 (comment).

If this can be implemented before March 5 this would help with a lot of manual line-break-fixing on my end in the lead-up to the spanish localization sprint.

[jpouellet]

I currently try to follow a balance between "new sentence => new line" and staying below 80 chars per line, as those rules make reviewing diffs easier.

I propose we consider not requiring giant single lines and instead attempt automatic combination of multi-line markdown paragraphs as part of the automated translation workflow of #1452, unless it proves to be too error prone.

[jpouellet]

I've created https://www.transifex.com/qubes-testing to do some simple testing without bothering anyone, and have identified a few more issues.

I definitely agree that the current "wrap at 80 chars" causes clear problems for translation (single sentences being split across identified strings), but it's not clear to me that combining everything to single paragraphs is what we want either. If we have a large paragraph which has been translated, then make a commit to fix a little typo somewhere in the english version, the whole thing gets marked as needing translation again. If however we split strings by sentence, then it is less work for the translator.
As for languages not necessarily having a 1:1 mapping in number of sentences to convey equivalent meaning, I think this can be handled by simply having some sentences translate to empty strings, and others translate to multiple sentences (which works fine as long as their translations are entered on a single line).

Also, our style of

some [link name] in sentence

[link name]: url

causes problems when the [ contents ] are translated without exactly matching the link again.

[jpouellet]

Looks like there have been previous attempts to do round-trip conversions from markdown => json k/v pairs => markdown specifically for use with transifex: https://iilab.github.io/contentascode/technology/translation/

[unman]

If you do this you will remove one of the key benefits of using markdown, that it is easily readable. The current style guide calls for a hard wrap to maximise accessibility and readability. I think it should be retained, and would be strongly opposed to the proposed change.

[marmarek]

Normally I'd say I agree with @unman. But while it's easy to turn on line wrapping in editor/viewer, hard line wrapping in source files pose major problem for translations. And actually for meaningful diffs too (re-wrapping makes diffs mostly useless).

So, if we can have easily some workflow for uploading non-wrapped text to transifex (or some middle ground like @jpouellet described), while still keeping original files wrapped, then ok. But if not - implementing this ticket makes life much easier for translators at not so high cost for others.

@andrewdavidwong what's your opinion here?

[andrewdavidwong]

I'm in favor of changing our policy from hard wrapping at 80 characters to inserting a newline at the end of each sentence. This mostly preserves source readability, results in the most useful diffs, and is easiest for translators.

[andrewdavidwong]

As for problems with our reference-style links (#2639 (comment)), I think the solution there is to ask translators to translate both occurences of the reference (i.e., copy and paste).

[andrewdavidwong]

@mfc: I can't tell whether you intended #1452 or this issue to track the actual line break changes (as opposed to the policy change). I intepreted this issue as being only for the policy and #1452 as being for the actual line breaks in the source, so I've closed this one. If that's not what you had in mind, feel free to re-open.

[andrewdavidwong]

One major problem with this is that it's proven very difficult to convert documents that are hard-wrapped at 80 chars to the new one-sentence-per-line standard. The result is an inconsistent mess of documents, some hard-wrapped and some not. Unless a solution for this is forthcoming, I'm moving strongly in favor of going back to hard-wrapping at 80 chars, since this can easily be done for all existing non-hard-wrapped documents with a few keystrokes in Vim. I've also seen no evidence that our efforts to implement one-sentence-per-line have produced any beneficial translation results, but perhaps there are other problems holding translation back.

[marmarek]

but perhaps there are other problems holding translation back.

Yes, there are. I think the most important one is lack of (even semi-) automated synchronization, in any direction (github -> transifex: #1452, transifex -> github: #2926). This resulted in translated content being out of sync, and ultimately a decision to hold the translation effort until such mechanism will be implemented, to avoid wasting time of translators.

[andrewdavidwong]

Another major problem: Doc contributors usually do not follow the one-sentence-per-line policy. Instead, they often do not wrap at all. Since our policy is not to reject or delay good contributions on the basis of things like formatting, we now have a large mess of documents that are not consistently wrapped in any one way.

However, completely unwrapped lines are fine for translators, since they just need newlines after each sentence or paragraph. The completely unwrapped text provides the latter.

This means that we can make all of our documentation consistent and translatable by simply unwrapping everything. This is easy to do in Vim:

:set wm=0 tw=999999
:normal gggqG 

The only problem with this method is that it also unwraps the YAML heading at the top and the reference-style links at the bottom, but there should be a way to spare those with regex. So, we could come up with a script that unwraps all of our docs appropriately. But once we have such a script, why not simply make it part of the localization process? Why impose the unwrap requirement on human authors and on our original source files when we can automatically unwrap it immediately prior to giving it to Transifex? Our model should be:

Original source (hard wrapped at 80) --> Auto-unwrap --> Transifex translation --> Language-specific repo

Here are some links on unwrapping in Vim:

• https://stackoverflow.com/questions/11348668/vim-convert-hard-wrap-to-soft-wrap
• https://stackoverflow.com/questions/11039544/unwrap-text-in-vim
• http://vim.1045645.n5.nabble.com/Does-automated-quot-line-unwrap-quot-exist-td1157525.html

[andrewdavidwong]

Code blocks could be tricky, since we don't want to unwrap those. They're probably easier to detect when explicitly fenced (with ``` or ~~~), but they could also be space-indented inside of lists.

[andrewdavidwong]

I'm still seeing the Qubes documentation in a mixed state: some hard wrapped lines, some unwrapped, sometimes even in the same document! I'm surprised this hasn't come up in any localization discussions outside of this issue. Is it not really a problem for localization?

@tokideveloper, you've been doing great work with converting our reference-style links. Perhaps this might be a similar type of mass conversion?

CC: @maiska

[maiska]

Hello @andrewdavidwong,

do you think it is worth it, since the policy is: accept anything in formatting terms? Are there anywhere some recommendations for writing docs, or best practices? Or you want to have a consistent baseline and see from there?
About localization or translation on Transifex it seems not to be an issue, see both files with wrapped/unwrapped lines as in introduction/support.md & introduction/reporting-bugs.md.

[andrewdavidwong]

do you think it is worth it, since the policy is: accept anything in formatting terms?

I care about making the docs clean, consistent, organized, and maintainable. I absolutely believe that is a worthwhile goal.

The policy is to accept contributions that would be a net benefit to the documentation. For example, it would be unwise for us to reject a contribution that adds valuable and well-written content just because it doesn't follow our line-break rules. The idea is not to allow the perfect to be the enemy of the good. If accepting a PR adds more in value than it detracts, we want to accept it.

However, this policy should not be interpreted as giving up on consistent formatting. We still care about formatting for many reasons, for example:

1. Having readable plain text source has been one of our primary goals from the beginning and was one of the main motivations for our current documentation system.
2. Consistent formatting makes the documentation easier for contributors and maintainers (more on this below).
3. Consistent formatting can help anywhere automation is involved, such as efforts in translation and offline documentation.

On a personal note, I find that working with clean, well-organized source files motivates me to keep them that way. By contrast, being forced to deal with messy, disorganized source files saps my motivation to work on them and makes me way to escape them as soon as possible.

Are there anywhere some recommendations for writing docs, or best practices?

Yes, of course: https://www.qubes-os.org/doc/doc-guidelines/

I'm surprised you didn't already know about this page!

In any case, as it says on this page under Markdown conventions, the current policy (due to this issue) states:

• Insert a newline at, and only at, the end of each sentence, except when the text will be reproduced outside of the Qubes website repo (see previous item for examples).
  • Rationale: This practice results in one sentence per line, which is most appropriate for source that consists primarily of natural language text. It results in the most useful diffs and facilitates translation into other languages while mostly preserving source readability.

(Our previous policy was to hard wrap lines at 80 characters.)

The new policy might sound good in theory. However, it's been in place for a few years now, and I can tell you that, in practice, most people ignore the policy entirely -- more so than the old hard-wrap policy, perhaps because the new one is harder to follow or not as evident from merely looking at the existing text. Moreover, even when they are aware of the policy and are trying to follow it, many people still forget to insert line breaks between sentences. Apparently there is something rather unnatural about inserting a newline in between each sentence. I suspect that people often forget them when they consider multiple sentences to be parts of the "same thought."

Importantly, we have never found a way to mass convert text that does not follow this policy so that it does follow the policy. Given that most people ignore the policy, this is a gaping hole in the system. I mostly just manually fixed the line breaks when I had time, but it's an incredibly tedious task, and as the years wore on, I eventually gave up.

By contrast, hard wrapping text at 80 (or any number) of characters per line is a few keystrokes in Vim, which made it much more sustainable and maintainable as a policy. As the documentation maintainer, having no such easy way to convert text to one-sentence-per-line or one-paragraph-per-line makes those policies much less appealing to me. (I have no doubt that code could be written that allows a user to easily handle those cases, but in four years, no one -- including me -- has had the time, ability, and inclination to write or discover it, and I have no reason to believe that will change any time soon.) Therefore, if we can switch back to the hard-wrap policy without any detriment to localization (or anything else), then that would be a boon.

About localization or translation on Transifex it seems not to be an issue, see both files with wrapped/unwrapped lines as in introduction/support.md & introduction/reporting-bugs.md.

If this is (now) true, then it's huge news for this issue. As you can see from this issue's history, it has been open for over four years, and for all that time, until now, we believed that line wrapping was a make-or-break issue for Transifex translation. Perhaps that was true in the past, and Transifex gained the ability to handle arbitrarily-wrapped lines at some point in the past four years. Is there any way to confirm that this is the case? @mfc, would it be easy for you to check whether the line-break problem you first observed four years ago is still there? There must be some explanation for why this was a big problem during the localization effort four years ago but never even popped up on the radar for the more recent localization push. I think it would be prudent for us to be sure about this before attempting to proceed.

[tokideveloper]

I'm still seeing the Qubes documentation in a mixed state: some hard wrapped lines, some unwrapped, sometimes even in the same document! I'm surprised this hasn't come up in any localization discussions outside of this issue. Is it not really a problem for localization?

Concerning Markdown files in the Transifex editor in the present, paragraphs are connected to a single string with all the linebreaks inbetween. Linebreaks between sentences are fine but linebreaks within a sentence look ugly to me and may irritate the translator ("What to do with that linebreak?"). So, it's probably a matter of taste. I don't like those linebreaks within sentences but I can understand that you don't want to correct them all the time. Maybe we should have neither a 80-columns-line-break-rule nor a linebreak-after-each-sentence-rule. At least not the 80-columns-line-break-rule since this would lead to linebreaks within sentences.

@tokideveloper, you've been doing great work with converting our reference-style links. Perhaps this might be a similar type of mass conversion?

Yeah, I could do it. But first we should find a solution to that issue.

[tokideveloper]

I suggest the rule that linebreaks within sentences should be avoided. Maybe this is sufficient in practice.

[maiska]

do you think it is worth it, since the policy is: accept anything in formatting terms?

I care about making the docs clean, consistent, organized, and maintainable. I absolutely believe that is a worthwhile goal.

The policy is to accept contributions that would be a net benefit to the documentation. For example, it would be unwise for us to reject a contribution that adds valuable and well-written content just because it doesn't follow our line-break rules. The idea is not to allow the perfect to be the enemy of the good. If accepting a PR adds more in value than it detracts, we want to accept it.

However, this policy should not be interpreted as giving up on consistent formatting. We still care about formatting for many reasons, for example:

1. Having readable plain text source has been one of our primary goals from the beginning and was one of the main motivations for our current documentation system.

2. Consistent formatting makes the documentation easier for contributors and maintainers (more on this below).

3. Consistent formatting can help anywhere automation is involved, such as efforts in translation and offline documentation.

On a personal note, I find that working with clean, well-organized source files motivates me to keep them that way. By contrast, being forced to deal with messy, disorganized source files saps my motivation to work on them and makes me way to escape them as soon as possible.

totally agree with all of the above

Are there anywhere some recommendations for writing docs, or best practices?

Yes, of course: https://www.qubes-os.org/doc/doc-guidelines/

When we are on the topic what would you think to incorporate the markdown linter as a best practice?
(I used the gem mdl to clean up a little bit of the docs. Now basically just inline htmls are popping up as md formatting problems and some fresh irritating ones :) ) Or should I open a new github issue specifically on this topic?

I'm surprised you didn't already know about this page!

yes, of course i know it, how can i forget ... was lazy thinking and in a chat mode writing style...

In any case, as it says on this page under Markdown conventions, the current policy (due to this issue) states:

• Insert a newline at, and only at, the end of each sentence, except when the text will be reproduced outside of the Qubes website repo (see previous item for examples).

  • Rationale: This practice results in one sentence per line, which is most appropriate for source that consists primarily of natural language text. It results in the most useful diffs and facilitates translation into other languages while mostly preserving source readability.

(Our previous policy was to hard wrap lines at 80 characters.)

The new policy might sound good in theory. However, it's been in place for a few years now, and I can tell you that, in practice, most people ignore the policy entirely -- more so than the old hard-wrap policy, perhaps because the new one is harder to follow or not as evident from merely looking at the existing text. Moreover, even when they are aware of the policy and are trying to follow it, many people still forget to insert line breaks between sentences. Apparently there is something rather unnatural about inserting a newline in between each sentence. I suspect that people often forget them when they consider multiple sentences to be parts of the "same thought."

Importantly, we have never found a way to mass convert text that does not follow this policy so that it does follow the policy. Given that most people ignore the policy, this is a gaping hole in the system. I mostly just manually fixed the line breaks when I had time, but it's an incredibly tedious task, and as the years wore on, I eventually gave up.

By contrast, hard wrapping text at 80 (or any number) of characters per line is a few keystrokes in Vim, which made it much more sustainable and maintainable as a policy. As the documentation maintainer, having no such easy way to convert text to one-sentence-per-line or one-paragraph-per-line makes those policies much less appealing to me. (I have no doubt that code could be written that allows a user to easily handle those cases, but in four years, no one -- including me -- has had the time, ability, and inclination to write or discover it, and I have no reason to believe that will change any time soon.) Therefore, if we can switch back to the hard-wrap policy without any detriment to localization (or anything else), then that would be a boon.

About localization or translation on Transifex it seems not to be an issue, see both files with wrapped/unwrapped lines as in introduction/support.md & introduction/reporting-bugs.md.

If this is (now) true, then it's huge news for this issue. As you can see from this issue's history, it has been open for over four years, and for all that time, until now, we believed that line wrapping was a make-or-break issue for Transifex translation. Perhaps that was true in the past, and Transifex gained the ability to handle arbitrarily-wrapped lines at some point in the past four years. Is there any way to confirm that this is the case? @mfc, would it be easy for you to check whether the line-break problem you first observed four years ago is still there? There must be some explanation for why this was a big problem during the localization effort four years ago but never even popped up on the radar for the more recent localization push. I think it would be prudent for us to be sure about this before attempting to proceed.

Yes, I just uploaded the two mentioned resources in their current state to transifex with the keyword wrapped, to make sure that the already available resources do not have any weird formatting. And no, there are no issues due to hard/no-hard wrapping.

In my experience with dealing with the documentation I think the following:

• it is an overhead to make sure that every sentence is on one line - it is cumbersome for writers and maintainers alike, and for translators/reviewers in that line of thought
• since there is no visible problem with the transifex platform (they should correctly parse md now i suppose) (please confirm @mfc, not that i am looking incorrectly... ), i would suggest keeping 80 characters wrap a standard, at least i my experience it is easy to read and work with, and I would fathom that a substantial part of the writers/maintainers have also tech background, so that this would be often seen as the default conf/setting

[jpouellet]

Andrew David Wong wrote:

Importantly, we have never found a way to mass convert text that does not follow this policy so that it does follow the policy.

Huh, I could have sworn that I did, but it appears not. Oh well. IIRC, https://github.com/QubesOS/qubesos.github.io/blob/master/_utils/spellcheck.sh should have most of the hard parts figured out, including correctly parsing content (via pandoc) to ensure any transformations can be context-aware, in order to not modify lines inside code blocks (which is important not just for code, but also to avoid mutating contents of pgp-signed blocks (and thereby invalidating their signatures) that appear in various places in the docs), and not raising false-positives for intentional exceptions by only caring about parts which has have just changed, unless you intentionally do a bulk sweep through everything. I vaguely remember extending the above to cover line-wrapping policy as well, but I guess not. Or if I had, I guess I never upstreamed it :/ Shouldn't be too hard if somebody is in a position to do so now and feels so inclined. if it turns out to be useful.

[andrewdavidwong]

Concerning Markdown files in the Transifex editor in the present, paragraphs are connected to a single string with all the linebreaks inbetween.
Linebreaks between sentences are fine but linebreaks within a sentence look ugly to me and may irritate the translator ("What to do with that linebreak?").

Hm, I'm a bit confused. This sounds like more or less the original problem with line breaks in Transifex. Are you saying, contrary to @maiska, that it actually is a problem for Transifex translation? Did we just get lucky that @maiska's test cases did not cause translators to get confused by seeing mid-sentence line breaks?

So, it's probably a matter of taste. I don't like those linebreaks within sentences but I can understand that you don't want to correct them all the time. Maybe we should have neither a 80-columns-line-break-rule nor a linebreak-after-each-sentence-rule. At least not the 80-columns-line-break-rule since this would lead to linebreaks within sentences.

Hard wrapping at 70-80 characters has been a standard thing for a long time, mainly because it makes reading text on terminals easier. I've heard many terminal users express the opinion that hard-wrapped text looks much, much better than unwrapped text. Terminal reading is not as common nowadays. Nonetheless, one of our goals is to make the documentation source files easy to read, including on terminals.

In any case, I think that there are much more important factors to consider here than personal aesthetic preferences, particularly maintainability and consistency, as I discussed above. The availability of tools to modify the text in order to make it consistent (i.e., maintainability) is paramount. And, now that I think about it, this cannot just be a single mass-conversion, since the docs will gradually become inconsistent with new additions. It has to be easy to correct each PR, which no one has shown is feasible except with hard-wrapped lines.

I suggest the rule that linebreaks within sentences should be avoided. Maybe this is sufficient in practice.

I do not think it will be, since then we would have, at the very least, both one-sentence-per-line and no special line breaks (which we currently have, plus more). More importantly, if someone did submit a hard-wrapped or one-sentence-per-line PR, we would have no sufficiently-easy way to fix it by transforming it into completely-unwrapped text. (By contrast, it is easy to convert it to hard-wrapped text.)

When we are on the topic what would you think to incorporate the markdown linter as a best practice?
(I used the gem mdl to clean up a little bit of the docs. Now basically just inline htmls are popping up as md formatting problems and some fresh irritating ones :) ) Or should I open a new github issue specifically on this topic?

Good idea, but separate issue, please.

Yes, I just uploaded the two mentioned resources in their current state to transifex with the keyword wrapped, to make sure that the already available resources do not have any weird formatting. And no, there are no issues due to hard/no-hard wrapping.

Amazing news!

In my experience with dealing with the documentation I think the following:

• it is an overhead to make sure that every sentence is on one line - it is cumbersome for writers and maintainers alike, and for translators/reviewers in that line of thought

Definitely.

• since there is no visible problem with the transifex platform (they should correctly parse md now i suppose) (please confirm @mfc, not that i am looking incorrectly... ), i would suggest keeping 80 characters wrap a standard, at least i my experience it is easy to read and work with, and I would fathom that a substantial part of the writers/maintainers have also tech background, so that this would be often seen as the default conf/setting

I am inclined to agree, mainly because it's the only method that will actually work right now (i.e., the only method for which we have a reliable tool).

Huh, I could have sworn that I did, but it appears not. Oh well. IIRC, https://github.com/QubesOS/qubesos.github.io/blob/master/_utils/spellcheck.sh should have most of the hard parts figured out, including correctly parsing content (via pandoc) to ensure any transformations can be context-aware, in order to not modify lines inside code blocks (which is important not just for code, but also to avoid mutating contents of pgp-signed blocks (and thereby invalidating their signatures) that appear in various places in the docs), and not raising false-positives for intentional exceptions by only caring about parts which has have just changed, unless you intentionally do a bulk sweep through everything. I vaguely remember extending the above to cover line-wrapping policy as well, but I guess not. Or if I had, I guess I never upstreamed it :/ Shouldn't be too hard if somebody is in a position to do so now and feels so inclined. if it turns out to be useful.

Thank you, but I still see several problems:

• No usable tool exists right now, and it would require time, inclination, and ability for someone to finish it. (To be more precise, no tool that I can use exists right now, and no one who has such a tool and is capable of using it is also willing to take over the job of maintaining the formatting consistency of our docs.)
• I highly doubt that there is anyone who will have the time, inclination, and ability to finish it. (This is not to fault or blame anyone. Everyone probably has more important things to do, and that's fine and expected. So we should just use the working tool we already have.)
• Even if someone did finish it, it would still not be as easy to use as a built-in Vim function.
• Even if it were made easy to use, it would not be as reliable. It might be easy and reliable on common use cases, but we would inevitably hit unexpected cases that would require either:
  1. Contacting the code author, asking them to fix the bug, waiting for them to fix the bug (possibly forever), using the updated tool, possibly seeing that it still doesn't work, etc. In the meantime, we would drift back to our current problem with the docs being inconsistently formatted; or
  2. Me resorting to manually fixing the problematic cases. Understandably, no one else would notice or care, and after some time, I would cease caring too.

[tokideveloper]

Concerning Markdown files in the Transifex editor in the present, paragraphs are connected to a single string with all the linebreaks inbetween.
Linebreaks between sentences are fine but linebreaks within a sentence look ugly to me and may irritate the translator ("What to do with that linebreak?").

Hm, I'm a bit confused. This sounds like more or less the original problem with line breaks in Transifex. Are you saying, contrary to @maiska, that it actually is a problem for Transifex translation? Did we just get lucky that @maiska's test cases did not cause translators to get confused by seeing mid-sentence line breaks?

Just to make sure that we are on the same page, with line breaks within sentences I mean the blue new line characters like these:

(from the Transifex resource wrapped_doc_introduction_support)

You can easily delete them or insert new ones by hitting the Enter key. IMHO they could disturb translators. But maybe I'm the only one who thinks so.

So, it's probably a matter of taste. I don't like those linebreaks within sentences but I can understand that you don't want to correct them all the time. Maybe we should have neither a 80-columns-line-break-rule nor a linebreak-after-each-sentence-rule. At least not the 80-columns-line-break-rule since this would lead to linebreaks within sentences.

Hard wrapping at 70-80 characters has been a standard thing for a long time, mainly because it makes reading text on terminals easier. I've heard many terminal users express the opinion that hard-wrapped text looks much, much better than unwrapped text. Terminal reading is not as common nowadays. Nonetheless, one of our goals is to make the documentation source files easy to read, including on terminals.

In any case, I think that there are much more important factors to consider here than personal aesthetic preferences, particularly maintainability and consistency, as I discussed above. The availability of tools to modify the text in order to make it consistent (i.e., maintainability) is paramount. And, now that I think about it, this cannot just be a single mass-conversion, since the docs will gradually become inconsistent with new additions. It has to be easy to correct each PR, which no one has shown is feasible except with hard-wrapped lines.

Okay, if it's more practical then hard-wrapping at 80 characters should be fine and we forget the one-sentence-per-line policy. I can live with that.

[andrewdavidwong]

Closing as "won't do." If anyone has a new reason for why this should be done, please leave a comment, and we'll be happy to take another look. Thank you.

[andrewdavidwong]

More importantly, if someone did submit a hard-wrapped or one-sentence-per-line PR, we would have no sufficiently-easy way to fix it by transforming it into completely-unwrapped text. (By contrast, it is easy to convert it to hard-wrapped text.)

I'm actually going to disagree with my past self: Neither conversion (unwrapped to hard wrapped or hard wrapped to unwrapped) is easy. If someone submits an unwrapped doc with long headings and/or code blocks, it's not possible to hard wrap the entire thing (sans YAML header) at once, since that would break the headings and/or code blocks (along with any other intentionally-unbroken long lines). Conversely, if someone submits a hard wrapped doc with code blocks, it's also not possible to unwrap the entire thing (sans YAML header) at once, since that would break the code blocks (and any other intentional line breaks). In both cases, I have to go through and selectively highlight blocks of text that are safe to [un]wrap, which is a manual process that has to be performed on each document each time it is significantly changed. I'm now thinking that completely unwrapped text might have the advantage for several reasons. For example, most people who submit small edits to existing docs don't bother to re-wrap the text they're editing, which probably wouldn't be a problem if the text were unwrapped to begin with. So, I think I might experiment with an unwrapped text policy.