How could I handle title depth in org-mode ?

[marius851000]

I want to use tree-sitter to create a tree for the org file. I have a potentially infinite depth of section. For example :

* level 1
sub level 1 content
lala
** level 2
sub level 2 content
...
**** level 4
sub level 4 content
*** level 3
* level 1-bis
** level 2-bis

This is similar to what is possible in with "#" in markdown. However, for folding reason, I would like to make section rule, that embed (for exemple) both * level 1, it's content and all child section. I could represent this like :

section_deeper: $ => seq(
  $.heading_deeper,
  $.heading_end,
  optional($.content),
  repeat(
    $.section_deeper
  )
)

section_any: $ => seq(
  choice(
    $.heading_deeper,
    $.heading_same,
    $.heading_lighter
  ),
  $.heading_end,
  optional($.content),
  repeat(
    $.section_deeper
  )
)

I finally found that I could implement that with C code, and that's what I did (this one doesn't compile to wasm due to printf):

#include <tree_sitter/parser.h>
#include <string.h>
#include <stdio.h> //TODO: remove

enum TokenType {
  HEADING_MORE_LEVEL = 0,
  HEADING_LESS_LEVEL = 1,
  HEADING_SAME_LEVEL = 2,
};

struct OrgData {
  unsigned int current_heading_level;
};

struct OrgData * tree_sitter_org_external_scanner_create() {
  printf("create\n");
  struct OrgData org_data_heap = {
    .current_heading_level = 0,
  };
  struct OrgData * org_data = (struct OrgData *) malloc(sizeof(struct OrgData));
  *org_data = org_data_heap;
  return org_data;
}

void * tree_sitter_org_external_scanner_destroy(struct OrgData *payload) {
  printf("destroy %d\n", payload->current_heading_level);
  free(payload);
}

unsigned tree_sitter_org_external_scanner_serialize(
  struct OrgData *payload,
  char *buffer
) {
  printf("serialize %d\n", payload->current_heading_level);
  memcpy(&payload, &buffer, sizeof(struct OrgData));
  return sizeof(struct OrgData);
}

void tree_sitter_org_external_scanner_deserialize(
  struct OrgData *payload,
  const char *buffer,
  unsigned length
) {
  printf("deserialize %d\n", payload->current_heading_level);
  memcpy(&buffer, &payload, sizeof(struct OrgData));
}

bool tree_sitter_org_external_scanner_scan(
  struct OrgData *payload,
  TSLexer *lexer,
  const bool *valid_symbols
) {
  printf("scan %d\n", payload->current_heading_level);
  if (valid_symbols[HEADING_LESS_LEVEL] || valid_symbols[HEADING_MORE_LEVEL] || valid_symbols[HEADING_SAME_LEVEL]) {
    unsigned int depth = 0;
    while (lexer->lookahead == '*') {
      depth += 1;
      lexer->advance(lexer, false);
    };
    if (depth != 0) {
      enum TokenType appropriate_symbol;
      if (depth < payload->current_heading_level) {
        appropriate_symbol = HEADING_LESS_LEVEL;
      } else if (depth == payload->current_heading_level) {
        appropriate_symbol = HEADING_SAME_LEVEL;
      } else {
        appropriate_symbol = HEADING_MORE_LEVEL;
      };
      payload->current_heading_level = depth;
      if (valid_symbols[appropriate_symbol]) {
        lexer->result_symbol = appropriate_symbol;
        return true;
      } else {
        return false;
      }
    };
  };
  return false;
}

and

module.exports = grammar({
  name: 'org',

  externals: $ => [
    $.heading_more_level,
    $.heading_less_level,
    $.heading_same_level,
  ],

  rules: {
    source_file: $ => repeat($._definition),

    _definition: $ => choice(
      seq(
        $.heading_more_level,
        optional("\n")
      ) //TODO: not heading, but section
      //TODO: other kind of section (header)
    ),
  }
});

However, with this test file :

*

It work as expected for the native version ("heading_more_level"), but the wasm version display and error and detect "heading_same_level". (but can hardly do more debugging due to not having printf).

The stdout is:

create
deserialize 0
deserialize 0
scan 0
serialize 1
deserialize 1
scan 1
deserialize 1
scan 1
deserialize 1
(source_file [0, 0] - [1, 0]
  (heading_more_level [0, 0] - [0, 1]))
destroy 1

What seem strange to me is that it deserialize before any serialisation, and I suspect that the result is caused by buffer defaulting to 0 in linux, but default to 1 in wasm for some reason.

I need help to understand why this doesn't, and ideally how I should make this work (are there a better way to handle this kind of syntax ? I can't base myself on tree-sitter-markdown syntax, as it doesn't implement this feature.

(I will add that I don't have a good knowledge of C)

Thank.

[marius851000]

So I found that tree-sitter-python have a similar system with tabulation. Will read how it's done.