[feature request] trim common leading whitespace

[benlubas]

problem

Some filetypes (notably org and norg) have code blocks that are potentially indented.

In a norg file, you might have something like this for example:

* Heading
  @code python
  print('hi there')
  @end

The otter buffer then looks like:

  print('hi there')

and then pyright complains about unexpected indentation.

proposed solution

Automatically trim white space (or add an option to enable this) when an injected chunk of code all has common leading white space. We'd then have to track the whitespace offset for each chunk and account for it when adding diagnostics or suggesting completions.

---

What are your thoughts on this? I'm happy to take a stab at implementing it.

[jmbuhr]

I like this. To avoid having to count spaces for all lines of each code chunk I would start by assuming that the first line of a chunk defines the indentation oft the whole chunk. Or do you already know of an example where this assumption does not hold?

Tracking the offsets for all the lsp requests, diagnostics and completion might become cumbersome, though.

If it would add too much logic, I would rather not implement it, at least for now. After all, the current implementation works for formats in which code chunks have to be syntactically valid (e.g. quarto, html, javascript etc.) because they are actually going to be executed when rendered / compiled. Something like neorg on the other hand is not designed to execute its code chunks, so it would feel very artificial and sort of pointless to add language server features to the code.

[benlubas]

Something like neorg on the other hand is not designed to execute its code chunks

I think it was designed with code running in mind. There is a neorg module that executes norg code cells, with the goal of this becoming a core module one day iirc.

Even if not running the code it's still nice to get LSP autocomplete while writing it.

---

I'm not sure about how complex this will be honestly. I'm hoping that it's simple as adding + offset in a couple places, and then tracking that value.

[jmbuhr]

It's a bit more involved than you'd think at first, because the otter.ask_ functions don't worry about what code chunk they are in, they just forward cursor position and swap out the main with the otter document where necessary in requests and responses. Especially for something like autocompletion I'd worry about the performance with those extra checks.

I'm not aware of people using norg as computational notebooks so far. There is no renderer, or is there? If you have a lot of code in norg files, something has probably gone wrong in the workflow and this code should be somewhere else. So I'd consider it a rather niche usecase -- allthough a fun one -- for which I would not want to compromise performance.

[benlubas]

Definitely harder than I would have hoped.

it seems like we already sync the buffers on each keystroke, so hopefully it's not too bad to do indents too. We'll see.

[benlubas]

screen_cap.mp4

I still have to add the offsets for things that aren't completion and diagnostics. But I have a function that I can just call from almost anywhere to grab that information.

I'm also going to make this configurable in the case that it does cause performance problems on lower end hardware you can just turn it off.

[jmbuhr]

Checking the value of the configuration is still overhead and it adds more code to maintain. I'm not yet sold on it being necessary as long as specifically python in norg is the only usecase. Sure, building this is fun, but do you actually need this? Just thinking long term here. What part of a workflow is covered by writing code in norg?

[benlubas]

python in norg is the only usecase

• There are definitely other languages and language servers that will have problems like this. Haskell is white space sensitive, and I'd imagine anyone that has a linter setup will see a ton of unnecessary errors too.
• Formatting just does not work for indented code blocks, it'll always try to de-dent lines
• even if it was only python in org files, I think it would still be worth while due to how popular python and org are

This change will also fix a bug with the way things currently operate in indented cells. The bug breaks plugin functionality on the first line of any indented code cell.

The first line doesn't get captured with its leading indents so the otter buffer looks like this:

print('first line')
	print('the first line is indented in the real buffer')

which throws off all the cursor locations on the first line.

still overhead and it adds more code to maintain

One if statement of overhead is nothing compared to the code that cmp already runs on every single keystroke, even without that check, the performance is not noticeably worse on my machine.

Maintenance is definitely a concern. I'll do my best to make this unobtrusive. I don't think that it's too bad currently, but we will see what it looks like when I'm done.

What part of a workflow is covered by writing code in norg[/org]?

Kinda the same as Quarto, you can write literate programming documents. (n)org is more centered on note taking, but it's still very valuable to have LSP features when writing code in notes. There is a school of thought that writing literate programs in org and then using "tangle" to produce a pure code file is the way to write code. Not that I agree with that, but people definitely write a lot of code in org files.

There are also renderers for org files.

[jmbuhr]

ok, I'm sold on the idea :)