Try parsing semantic elements instead of line-based

[schoettl]

In this branch, I work on the higher level syntax according to https://orgmode.org/worg/dev/org-syntax.html

Specifically, I want to check out, if we can move away from line-based parsing towards more semantical blocks, called "elements". The orgmode parser used for export is also called org-element.el.

The spec says, that most elements of the syntax are not context-free and the categories for these elements are

“Greater elements”, “elements”, and “objects”

Greater elements are e.g. #+BEGIN_EXAMPLE blocks. Some of these blocks contain raw text (EXAMPLE, SRC, COMMENT, ...), others can contain formatted text (CENTER, QUOTE, ...). Hence, it's better to parse context-aware and parse the multi-line raw content in EXAMPLE but formatted text in CENTER block.

Also, paragraphs, multi-line footnote definitions, lists, tables, property drawers are maybe better parsed as units instead of line-based.

• And implement comments: https://orgmode.org/manual/Comment-Lines.html

[munen]

Best of luck with this approach, the described rationale seems absolutely reasonable to me! 👍

[schoettl]

Note to self: I thought tests passed before rebase at bf7104c ...

[munen]

@schoettl The tests were crashing on this branch, because of (what I think is) a typo. I took the liberty to fix the typo in this commit, if it was not accidental, I apologize for stepping on your toes^^

[schoettl]

Nice, this was a typo. Thanks!

[munen]

You're welcome. Good work on the PR so far!

NB: I've asked my business partner on feedback on how to work with semantic elements and he's writing something up as I type this^^

[schoettl]

Hi @munen @branch14 ,

I have some changes I'd like to merge:

• Rename token head-line to headline (better now than later)
• Add some elements to EBNF (with tests), including horizontal-rule, comment-line, TODO keyword
• Refactor and document some tokens in EBNF, including drawers, property drawers
• Fix <word> regex (there was a $ inside [...]
• Use insta/failure? instead of map? in tests
• Add tests

My TODOs after merge:

• I also added and then deleted cacf5db a bunch of tests for semantic parsing of whole blocks. I'll reintroduce them in a new PR.
• I'll also create a separate PR for parsing content-line as more than just .* (see Parser will not parse styles within sections. #26 ).

[branch14]

@schoettl This looks great!

I was initial wary of this PR mainly because of its title. A previous attempt of mine to move from line based parsing to semantic blocks failed miserably.

I just read through your changes and I'm totally happy with where this is going. 😄

Definitely LGTM! 👍

[schoettl]

I was initial wary of this PR mainly because of its title. A previous attempt of mine to move from line based parsing to semantic blocks failed miserably.

Thanks! Right, the title of this PR is wrong. I didn't switched to parsing of semantic blocks yet, but I prepared parts of it :) I'll try again in new PR.

[munen]

@schoettl I'm late to the party, but I don't want to miss the chance to say that this PR looks great. Good work and solid merge! 🙏