Section Headings Not Always Matched

[tajmone]

Currently the syntax only recognizes as valid section headers which don't contain attributes in their contents.

There might also be problems with handling title styles and some other inline constructs (need to provide tests coverage for the details).

Being able to properly detect and scope sub-elements within a section title shouldn't be a problem, we only need to create a dedicated context for headings that includes the required in-line elements contexts.

The problem we might facing (based on similar work with other markup syntaxes) is that Sublime Text can't handle proper indexing of TOC symbols which contain sub-elements. E.g.

== Heading Containing {some-attribute} Within

Chances are that the attribute won't be indexed. The only way to allow its indexing would be to not scope the attribute as entity.name.section.asciidoc, but this will leave a hole in the indexed heading — which, is worth remembering, we wish to index because we want to be able to navigate quickly through the various sections headings via "Goto Symbol".

It would also be nice to execute some transformation on the indexed headings, e.g. convert AsciiDoc straight quotes and backticks to typographic quotes, etc. If there was a way to expand attributes to their values (e.g. via the API) it would be great, but I doubt it.

The most important thing right now would be to ensure that no valid section headings are unmatched due to inline markup in their contents.

Also need to check how the syntax handles heading spanning multiple lines.

[tajmone]

More Precisely...

The problem only affects section titles which start with an attribute reference, e.g.:

== {Lorem} Ipsum Dolor

whereas if the attribute appear within the title contents it's simply treated as normal text, and the title is correctly indexed:

== Lorem {Ipsum} Dolor

Attributes Are Now Supported!

Now, thanks to @polyglot-jones's contribution (see 8c99451), the syntax does support attributes references.
From Asciidoctor.sublime-syntax (L412):

  attribute_reference:
    - match: ({)([A-Za-z0-9_][A-Za-z0-9_-]*)(})
      captures:
        1: constant.character.attributes.reference.begin.asciidoc
        2: support.variable.attribute.asciidoc
        3: constant.character.attributes.reference.end.asciidoc

Right now, the headings definition (section_titles) doesn't include attribute_reference in its own context — and probably doing so would break proper indexing of section titles, as mentioned in the original post.

I need to test how inclusion of @polyglot-jones's section_titles works, i.e. if it doesn't break correct title indexing.

Nested Symbols Dilemma

I have stumbled upon the same problem with PML, another lightweight markup syntax similar to AsciiDoc and Markdown (in many respects).

PML also supports "constants" definitions and references, which are very similar to AsciiDoc attributes. The problem here is trying to index both symbols: i.e. having ST track occurrences of constants/attributes references, and also indexing section titles (for quick navigation via Ctrl+R).

While working on the Sublime Text syntax for PML I couldn't find a way to preserve "constants" (very similar to ADoc attributes) and proper title indexing: if I wanted to preserved the attribute as a symbol (i.e. being indexed and listed by ST) I had to accept breaking-up the title indexing, and the only way to preserve the correct section title in the symbols index was to give-up tracking the occurrence of the attribute among the symbols list.

Right now, I don't recall all the details, but I do remember that I did stumble upon some limitations on how ST fails to handle nested indexed-Symbols.

For more details, see tajmone/Sublime-PML#13, which should contain some useful reminders and tips on the problem (although the PML syntax might have changed in the meantime, and no longer be as mentioned in the Issue).

In any case ... it would be a mistake to think that by simply including attribute_reference in section_titles should work flawlessly — we need to check if the presence of the attribute interferes with proper indexing of the section title, and also bear in mind that tracking attributes definitions might be a desirable feature in future.

The idea is that by properly scoping attributes definitions and references, end users can then select an attribute and exploit ST functionality like "Goto Symbol" and "Goto Definition" to quickly navigate through attributes occurrences and their definitions — a feature which is obviously useful, especially in big projects.