A possible solution to the problem of redundant blank lines.

[itraveller1]

First of all, thanks for your work!

The project is great, but the problem of redundant blank lines is turning off many potential users.

I suggest introducing the empty string equivalent as a separate hard break token, like \\<nl>. This solution is orthogonal to the existing rules and not only solves the problem of vertical space, but also, I think, makes the source text more readable by highlighting semantic headings.

For example:

## Section header \\
     Description line.

List header: \\
1. Item string.
2. Item string. \\
   - item
   - item
3. Item string.

[jgm]

This is a bit like @matklad's idea in #128 (comment)
which uses || as a "block break" device.

As applied to this particular issue, I am not sure I see the utility. The people who don't want to insert a blank line before a list are probably also not going to want to put \\ at the end of the line. And from the point of view of readable source, I think that the blank lines are better: \\ is non-linguistic noise, whereas the blank lines visually set off the list as a distinct block, just as it would be set off in a published book.

[itraveller1]

Yes, it does look very similar. But so much the better - one more problem will be solved.

Blank lines fail to convey the semantic commonality of the blocks. This is what subconsciously annoys some people at the sight of their monotonous alternation, and not at all the concern for saving space.

In addition, this "garbage" even formally is by no means alone in its logic:

1. <nl> - soft break
2. \<nl> - line break
3. \\<nl> - hard break

[jgm]

Blank lines fail to convey the semantic commonality of the blocks.

What do you mean by that?

[itraveller1]

What do you mean by that?

What I mean is that an empty string is not capable of representing a relationship between, for example, the title of a list and the list itself. This is of course purely contextual. In any case, it does not break the format, but makes the text less readable.

[itraveller1]

You are absolutely right about non-linguistic noise, and this is the only but very serious obstacle.

It is possible to offer :<nl> as a hard break token along with a blank line. If there are serious objections to this, then I see no other way to resolve this issue within the framework of the linguistic field.

Respectfully yours.

[bpj]

I don't really see where this hostility to blank lines comes from. As @jgm says it would be there in print. The reason for that is that blank space between blocks enhances readability and anything which enhances readability is good. Also it does not increase file size that much since it is only one or two extra bytes each time. Lastly it makes automatic reformatting easier (although this is less important if you have a good reformatting function — I always reformat using Pandoc with the format at hand as both reader and writer!)

I guess this comes from the one-line-per-paragraph crowd, whose obsession I, like jgm, don't understand.¹ I do understand the one-sentence-per-line crowd better since that at least enhances diffing. It is anyway, as jgm repeatedly has said, important not to force the one-line-per-paragraph style on users. If blank lines is the price to pay for a saner parser I'll gladly pay it. It is vastly superior to adding extra non-blank noise to indicate the exception.

I think it also may be that more people are using small screens nowadays, which I empathize with but don't sympathize with, because it usually comes with smaller font size/line height which ought to compensate. It isn't all that different from working in a terminal window, which many of us used to do even before handheld devices.

tl;dr if a few extra blank lines in a plain text document drives some people away then maybe plain text editing isn't for them (and I challenge them to come up with a sane parser which meets their requirement without a lot of complication — I have tried it!)

Footnotes

1. I do get that it might be a generational thing though! ↩

[itraveller1]

It is not about imposition. Аnd the one-line-per-paragraph style has nothing to do with it at all. This decision does not affect the existing order of things. If you want to use only blank lines as block separators, use.

And if we are to be completely categorical, then the items in the lists must be separated by an empty line too. In the end, these are also different blocks. But djot does not require this. So you can go to the meeting of readability, if it does not violate the main principles and does not require complex heuristics.

[jgm]

The logic has to do with the principle of uniformity. If a sequence of lines form a block, they should still have the same meaning when indented and put into a list item. So if

aaa
- b

isn't a paragraph and a list, then

- aaa
  - b

shouldn't contain a sublist. That logic doesn't apply to subsequent list items at the same level.

Given this principle of uniformity, there are two viable strategies:

1. don't let a list interrupt a paragraph without an intervening blank line, and always require blank line before sublists
2. allow a list to interrupt a paragraph without an intervening blank line, and don't require blank line before sublists

Djot goes with 1. There are fans of 2, but 2 is problematic if one hard wraps text, because this can cause inadvertent list starts. (Similar issues for block quotes.) So fans of 2 are often die-hard anti-hard-wrappers. Fans of 1 think that hard-wrapping should be allowed (but not required).

[itraveller1]

Yes, I understand and fully support the first principle.

Perhaps I didn't describe the essence of the proposal clearly. The new token will simply be another form of а blank line in every possible sense. That is, \\<nl> and <nl><nl> are the same thing.

[jgm]

I do understand that.

[itraveller1]

Last try :)

I understand that this topic has set the teeth on edge, but an attempt to comprehend it led to interesting conclusions. Since in a nutshell I fail to show the true essence of the problem, I decided to submit for your judgment a kind of essay.

---

Solving the problem of blank lines in LML Djot

Project "Djot"

The LML syntax is important in areas where the focus of the work is shifted to the source text: keeping notes and blogs, writing articles, prototyping publications, etc.

The main disadvantages of modern LMLs are usually an extension of their merits. For example, the ergonomics of Markdown comes at the cost of performance, while the completeness of Asciidoc comes at the cost of verbosity. Even Orgmode is losing appeal outside of its app.

Attempts to create a truly universal LML only expanded their nomenclature, because came from the practice of plugging holes, not fundamental principles. But today we have such a chance - it's Djot.

The Djot project is based on a set of principles that should ensure that the language is ergonomic, efficient, and extensible in a consistent way.

Djot's main problem

A well-composed and balanced syntax satisfies almost all the basic requirements for LML, but the abundance of empty lines in the source text causes subconscious irritation.

The author of the project considers this problem to be a matter of aesthetic preferences and urges to put up with this syntax, since all other known approaches violate the fundamental principles.

One single detail prevents Djot from becoming a truly popular and universal solution.

The real cause of the blank lines problem

Does the blank line problem really just reflect established preferences, or is there something more fundamental at play here?

Any text is essentially a set of related meanings, accentuated with the help of the format. Blank lines reflect the degree of semantic proximity between blocks of text in it, which significantly improves its readability.

In Djot, blank lines are used only to separate blocks, which does not reflect their true nature.

Observing a picture of evenly alternating blocks of text, our consciousness itself is forced to identify the degree of their semantic similarity, based on the context. This subconscious work and causes a feeling of irritation.

What's stopping Djot from solving the blank lines problem

The most thorough method of text analysis is line by line. It gives maximum opportunities, but it also requires the same amount of resources.

Djot saves a significant amount of resources by moving from line-by-line to block-by-block analysis. Now you do not need to check the format for each individual line. However, the price of this decision was a certain loss of flexibility in the analysis of blocks.

The only natural block delimiter was the blank line. Any attempts to switch to other markers or a partial return to line-by-line analysis will contradict the main principles of the project: we lose either ergonomics or efficiency.

Metablocks: Solving the Blank Line Problem

If neither line-by-line nor block-by-block analysis satisfies us, then we need some other basic structure. What if we try to return to empty lines their natural essence - the separation of semantic blocks?

Djot, like Markdown, does not support semantic division. This decision is based on the fact that we are trying to implement only those format elements that are in demand in normal writing and already have established display forms.

But the child was thrown out with the water: such things as list headings and descriptions for headings represent very popular and well-established forms, although the colon, as a marker of enumerations or explanations, has been used in ordinary writing for a long time and everywhere.

Due to line-by-line analysis, Markdown partially eliminates this problem, but for Djot this has become an unsolvable task.

Let's call a group of semantically related blocks a metablock. Empty lines will be used to delimit metablocks (taking into account nesting in containers), and a colon will serve as a natural marker for separating metablocks into blocks.

Conditional break marker

Let's name the new marker "cond break". It should be noted that we do not need any colon, but only the one at the end of the line.

Break	of format	of semantics	Marker
soft	no	no	line feed
line	no	yes	escaped line feed
hard	yes	yes	blank line
cond	yes	no	coloned line feed

The decision on the need for a break will be made based on a comparison of the formats of the separated strings. If the string formats are completely the same, then nothing needs to be broken. Otherwise, for example when moving from a header to a simple line, from a simple line to a list, or from a key line to something else, we will get a gap between blocks.

Thus, the decision to break will be made depending on the continuation format.

Since the colon can be used directly in text, we need two kinds of cond break markers: with and without preserving the character:

1. Explicit cond break

   Here we will deal with the usual colon, standing directly at the end of the word. It will be saved as a visible character on the output.

2. Implicit cond break

   In this case, the colon must be separated from visible characters by at least one space. It will not be saved on output.

Conclusion

The proposed solution to the empty lines problem in Djot is not just syntactic sugar: metablocks implement that part of the semantic markup that is clearly identified and is ubiquitous in almost any text.

Metablocks do not claim to be mandatory inclusion in the AST. This is a support structure. The main purpose of the metablocks is to preserve the natural structure of the source text in Djot without violating the core principles of the project.

An example of Djot source text with metablocks:

# Header :
   Description

: Key string :
   Definition

List header:
- item
- sublist header:
   - item
   - item
- item

And the same, but without metablocks:

# Header

   Description

: Key string

   Definition

List header:

- item
- sublist header:

   - item
   - item
- item

[gavinhoward]

Not the author, just a user.

I like your model of metablocks vs blocks.

However, I'm not so sure I agree with the colon+newline idea; my brain reads that as part of the text. Maybe I could get used to it, but personally, I am having an easier time getting used to extra blank lines in djot, though that may be simply because I prefer having more whitespace vs having less.

[itraveller1]

I honestly tried to find at least some collisions where the colon at the end of the line could be interpreted differently than we used to expect from it, but I could not.

Even the implicit conditional break marker, while avoiding displaying the colon, does so where the colon is essentially implied. And you can put more spaces in front of it so that it is not perceived as a regular sign.

Of course, not everyone uses things like header descriptions. The main thing is that Djot will have the means to express the required syntax, commonly used in plain text.

[jgm]

Djot saves a significant amount of resources by moving from line-by-line to block-by-block analysis. Now you do not need to check the format for each individual line. However, the price of this decision was a certain loss of flexibility in the analysis of blocks.

This is a misunderstanding of the motivation. I tried to explain this above. The reference parser in fact does do line by line analysis. This isn't about performance, it's about uniformity of the syntax.

Side note: in docbook, list elements can have "titles." I think you're looking for something like that. But we should distinguish between the "title" of a list and the paragraph preceding it. Perhaps there should be a special syntax like the one you propose for "list titles." But I think most people don't distinguish between list title + textual content before.

[itraveller1]

If I understood you correctly, the main thing is that any text between blank lines is interpreted in the same way with any transfer. But metablocks behave similarly, don't they?

Even for metablocks, this will be more natural, because the transfer of related blocks is more logical without breaking it into parts.

As for the differences between а list header and the simple paragraph preceding the list, this is not required. The metablocks don't need to be mapped to the AST, so it will just be a sequence of regular blocks, as it is now. I am by no means trying to inflate Djot to full support for semantics.

In a purely applied plane, my proposal can be regarded simply as an attempt to introduce a conditional hard break by means that are purely linguistic in form and meaning (I mean the colon).

[karlb]

I tried to find another way to get rid of the blank lines with an example implementation in this PR against djot.js: allow sublists to interrupt single-line list items without a blank line, but still require the blank line once the item has continuation lines.

So

- foo
  - bar

creates a nested list, but

- foo
  cont
  - bar

still requires the blank line. The idea is that intent is unambiguous when the item is a single line, which works well after testing against a bunch of real world documents.

While the approach seems quite practical for users, it does violate the principle of uniformity and has been therefore rejected. After all, being principled is what makes Djot so nice. I'm just documenting this here so that we can either get enough inspiration to find a better way, or at some point say "We've considered all options are are sure the current syntax is the best we can do".

[dereuromark]

Thats a valid point.
Maybe my parser is a bit too lenient here, but in "markdown" mode ("Significant newlines" on), it works for both cases here.
https://sandbox.dereuromark.de/sandbox/djot?d=LSBmb28KICAtIGJhcgoKLSBmb28KICBjb250CiAgLSBiYXIK&sig_newlines=1

The dash has to be followed by a space, otherwise it would not work of course.

I like the opt-in way for this behavior, as by default this is not valid (of course), but in some scenarios with users coming from markdown, this can be a helpful middleground. E.g. blogging software like I use in WordPress.