Possible Solution to Newline Problem and Other Enhancements

[gavinhoward]

Not the author, just a user and future implementer.

In this post, I will attempt to solve the newline problem while enhancing djot according to its design rationale.

It should be obvious now that the biggest problem people have with djot is the requirement for extra newlines when hard-wrapping.

As someone who prefers more whitespace than not (see this code), I personally don't have much problem with this. But it is a sticking point, apparently.

And this comment does have a good point: there are use cases where hard-wrapping is just bad. I'm interested in those use cases too, especially for comment systems. (I am making an all-in-one VCS that includes a bug tracker, and that will have a comment system. I want to use djot for that.)

This comment also has a good point that creating a new language may not be the best idea.

In addition, there is some discussion about adding document-wide metadata. I agree with this because of djot's rationale number 9:

It should be possible to attach arbitrary attributes to any element.

Currently, it's not possible to attach arbitrary attributes to the entire document, which is an element. I also use Pandoc to generate manpages, and Pandoc uses document-wide metadata to do this. If djot is to be useful for that purpose, and it should be, there must be some way of having such document-wide metadata.

So I solved two birds with one stone: djot should have document-wide metadata, and that metadata should be able to describe how to parse the rest of the document as well.

Here's how it will work:

1. There will be some well-defined parse options.
2. Those options should have defaults or be present in the metadata.
3. If an option does not exist, or the metadata does not exist, defaults should be used.
4. Once the metadata is parsed, the parser now knows how to parse the rest of the document.
5. To enable hard-wrapping, a parse option called hard_wrap should be set to true in the metadata.

The reason that hard-wrapping defaults to off is for two three reasons:

• It seems (I could be wrong) that most people are against it.
• When using djot in a comment system or such-like, metadata usually won't exist, and djot should default to no hard-wrap.
• Hard-wrapping is most useful in documents, where the metadata will most likely exist anyway.

This design has a few advantages beyond just solving the biggest gripe with djot:

• It will be possible to set a document title, author, and other universal metadata.
• It will enable other useful document options, such as adding a table of contents.
• It will make djot more universally usable, such as in static site generators, which almost always use metadata headers.
• If we really wanted to go for broke, there might be an option to enable hard-wrapping, but also to require indentation for lists so that they could use both hard-wrapping and not require extra blank lines. (I would use this because this is my style in Markdown already.)

There is one big problem: the format of the metadata. People have suggested YAML and TOML, but there doesn't seem to be any consensus.

Shameless plug: I suggest a format I came up with, called GAML (Gavin's Accessible Markup Language). It looks like this:

project: #com.yzena.yc

version: {
    min: #0.0.0
}

mode: {
    language: #declarative
    stampers: #modified
    dependencies: #static
}

presets: {
    debug: {
        debug: true
        optimization: "0"
    }
    release: {
        debug: false
        optimization: "2"
    }
}

default_development: #debug
default_release: #release

deftarget: #all

Think of it like JSON with the following changes:

• There are comments.
• Keys can be straight identifiers, instead of needing to be strings.
• Newlines can serve as commas.
• Identifiers can be items as well (called symbols).
• It is strictly typed.

It can be parsed in about 1710 lines of portable C, (not including an insertion-ordered map), and Lua should require much less. It could also be used as an AST format.

I am willing to make the changes to the current implementation to parse metadata and to parse GAML, including adding the hard-wrap-disabled mode, even if just as an experiment. Please let me know if there is interest. I'm also working on my own implementation in C.

Edited to Add: While there is no specification for GAML as of yet, I'm willing to write one right away should this community want one before adopting it.

[jgm]

Metadata is definitely on the table for consideration; I haven't gotten around to that yet.

I'm not too enthusiastic about the metadata including parsing directives; I'd rather have the metadata be, well, metadata.

I just want to make one point about the phrasing of the argument:

And ... does have a good point: there are use cases where hard-wrapping is just bad. I'm interested in those use cases too, especially for comment systems.

Sure, I agree that there are use cases where hard-wrapping is undesirable. But djot doesn't require hard-wrapping, it only allows it. It's perfectly possible to write djot in its current form without hard-wrapping. You just need to include those blank lines.

[gavinhoward]

I just want to make one point about the phrasing of the argument...Sure, I agree that there are use cases where hard-wrapping is undesirable.

Yes, my phrasing was too harsh.

More specifically, when a person is writing a comment, it should be less strict. This is because anything that the format requires of the commenter slows them down. This cost is okay in documents because documents are usually crafted, where being slow is a virtue, but comments are quick, one-and-done. It's two different styles of writing: scratch vs crafted.

djot is just about perfect for crafted writing (it just needs metadata, IMO), but for scratch writing, it's not ideal. Whether that matters to you as the author is your choice, though.

Regardless, I'm still happy with djot and will continue the work to adopt it.

[vassudanagunta]

djot is just about perfect for crafted writing (it just needs metadata, IMO), but for scratch writing, it's not ideal.

I think what distinguishes djot from Markdown is that the former aims to have simpler, coherent rules and the latter aims to let humans do whatever comes naturally (what I think you mean by "scratch writing") as much as possible and have whatever parsing rules are necessary to handle that. Compare the length of the djot specification to that of CommonMark's. See my comment here and John's reply.

More specifically, when a person is writing a comment, it should be less strict.

Whether or not your statements are true, I think we want a world with different philosophical, utilitarian and aesthetic choices. The value of djot is that it presents a new option with clear principles and goals. It's good that we have HTML, Markdown, Asciidoc, Org mode and now djot, just as it's good we have C, Haskell, Go and Rust. The right answer to many of the feature requests expressed on the CommonMark forum is "That's what HTML is for; let's not turn Markdown into HTML".