lsp-mode getting a bit confused

[jdtsmith]

I've had success setting up python-mode and markdown-mode together in syntactically valid python files, with file layout like:

#* Title [markdown]
"""
Markdown here
"""

#* Other Title
for i in range(10): print(i)

It works beautifully for keymaps, font-locking, navigation, etc. (though there are a few markdown commands the "leak" out of the markdown chunks).

I wanted to ask specifically about lsp-mode though. I had hoped that since the file was and remains fully syntactically valid python, lsp-mode would have no issues with it. And in fact that's mostly the case. What I find however is that after some editing, lsp-mode get slightly lost, for example placing fly check errors in the wrong spot in the buffer, a few lines off. This situation persists and requires a reset of the server. Clearly it has "lost sync"; its idea of the source code layout is no longer fully correct. The way lsp works is the full file is sent to the server initially, but after that, only a series of textDocument/didChange events, like:

{
  "textDocument": {
    "uri": "file:///XXX/code/python/scraps/test.py",
    "version": 31
  },
  "contentChanges": [
    {
      "range": {
        "start": {
          "line": 49,
          "character": 2
        },
        "end": {
          "line": 49,
          "character": 2
        }
      },
      "rangeLength": 0,
      "text": "c"
    }
  ]
}

get sent. Clearly these lsp-mode generated updates to the server are getting confused by polymode, and stop reflecting the actual structure of the file. Can you hazard a guess as to how that might come to pass?

Thanks for polymode, it's really very powerful.

[jdtsmith]

OK I have looked closely and I believe I understand WHY this is happening: each mode sets up its own before-change-functions and after-change-functions, and only the relevant buffer's version is being run (despite "both" buffers obviously being changed). I see you encountered this same issue for ppss cache flushing:

  ;; Modification hooks are run only in current buffer and not in other (base or
  ;; indirect) buffers. Thus some actions like flush of ppss cache must be taken
  ;; care explicitly. We run some safety hooks checks here as well.

lsp-mode adds lsp-before-change to before-change functions, as well as flycheck-handle-change and lsp-on-change to after-change functions. markdown-mode adds markdown-gfm-checkbox-after-change-function to the latter.

What I don't yet understand is how this could be addressed. Ideally we could say "go ahead and forward these modification updates over to the base buffer's before/after change functions". That's effectively what you are doing with polymode-flush-syntax-ppss-cache: in all connected buffers, run this. For me it would be enough if the indirect (markdown) buffer would forward its modifications events to the base buffer.

So perhaps adding our own "forwarding" before/after-change functions (for me, in the indirect buffer) that does a (notional):

(with-current-buffer (buffer-base-buffer (current-buffer))
  (dolist (func before-change-functions)
    (run-hook-with-args func beg end )))

and similar for after-change. Thoughts?

[jdtsmith]

The following seems to be working well:

  (defun my/lsp-before (&rest r)
    (with-current-buffer (buffer-base-buffer)
      (apply #'lsp-before-change r)))
  (defun my/lsp-after (&rest r)
    (with-current-buffer (buffer-base-buffer)
      (apply #'lsp-on-change r)))
  (defun my/lsp-conduit (type)
    (when (eq type 'body)
      (add-hook 'before-change-functions #'my/lsp-before t t)
      (add-hook 'after-change-functions #'my/lsp-after t t)))

with my/lsp-conduit in my inner (markdown) mode's :init-functions. Let me know if you spot any issues or a better approach. I've also changed the markdown stanzas to raw strings, so lsp ignores them.

[vspinu]

@jdtsmith Thanks for figuring this out and sorry for inactivity.

I would like to integrate this into polymode proper. Would it be possible for you to share your lsp+python+polymode config so that I could replicate the problem?

Is this markdown inside python a standard thing for python? Would it make sense to create a dedicated polymode for it?

[vspinu]

with my/lsp-conduit in my inner (markdown) mode's :init-functions.

So, if I understand correctly is that inside markdown chunks there are no lsp-on-change hooks. Thus, in order to have a generic solution for this one would need to trigger lsp-on-change from inside inner modes whenever lsp mode in base buffer is detected.

[jdtsmith]

Thanks for getting back to me. It seems there are a few people working on "lightweight" literate setups with fully syntactically-correct files (wrapped markdown/etc. in block comments or multi-line strings, so in principle any language). In some ways this would compete with org-mode and its source blocks, which have more features, but are harder to get working well with tools that expect to "own" the entire file (linters, language servers, etc.). The lsp-mode folks are trying to get this to work, but it's hard. Tagging @astoff as well in case he is interested (author of code-cells).

So, if I understand correctly is that inside markdown chunks there are no lsp-on-change hooks. Thus, in order to have a generic solution for this one would need to trigger lsp-on-change from inside inner modes whenever lsp mode in base buffer is detected.

That's correct. Maybe it's as simple as documenting "how to forward modification hooks and custom local variables" between inner and outer modes, with some examples. But you might also think about a more generic framework for this. I suspect most people who report that polymode is "brittle" or mysterious probably haven't fully understood they are dealing with two basically independent buffers syncing only content between them, even for counterintuitive things like modification hooks.

Here's my working markdown+python(+lsp-mode) polymode, which uses raw multi-line strings to hold markdown. See also this discussion, which has a picture. BTW, one other thing I noticed is that the adjust-face properties do not get added immediately, but only after editing the header line. I can open an issue if this isn't something simple/obvious.

;;;;;; polymode: mix modes via indirect buffers  
(use-package polymode
  :ensure t
  :init
  (defun my/lsp-before (&rest r)
    (with-current-buffer (buffer-base-buffer)
      (apply #'lsp-before-change r)))
  (defun my/lsp-after (&rest r)
    (with-current-buffer (buffer-base-buffer)
      (apply #'lsp-on-change r)))
  (defun my/lsp-conduit (type)
    (when (eq type 'body)
      (add-hook 'before-change-functions #'my/lsp-before t t)
      (add-hook 'after-change-functions #'my/lsp-after t t)))
  :config
  (define-hostmode my/poly-python-hostmode)

  (define-innermode my/poly-python-markdown-innermode
    :mode 'markdown-mode
    :name "Markdown Cell"
    :head-matcher (rx bol ?# ? (>= 1 ?*) (* nonl) ?\n
		      "r'''" (* nonl) ?\n)
    :tail-matcher (rx bol "'''" (* nonl) ?\n)
    :head-mode 'host
    :tail-mode 'host
    :head-adjust-face `(:background ,(alect-get-color 'dark 'bg+1) :extend t)
    :adjust-face nil
    :init-functions '(my/lsp-conduit))

  (define-polymode my/poly-python-mode
    :hostmode 'my/poly-python-hostmode
    :innermodes '(my/poly-python-markdown-innermode))

  :hook ((python-mode) . my/poly-python-mode))

[jdtsmith]

Here's a more recent look at what I'm working on (polymode + python + outshine + markdown + lsp-mode + flycheck):

[vspinu]

This is very interesting. I would like to get involved a bit more closely. Is there a public repo where you keep all of the stuff?

There have been a few requests for python specific polymodes over the years (#153, #180) and I think a poly-python repo as part of this organization is really overdue. So maybe it's a good time to create one right now with all of the commonly used poly-python modes. I can give you owner rights for it if you are interested in participation. I am not a heavy python user (not at all recently), nor even polymode user to be frank, so it would be difficult for me to pull all that by myself.

In any case, I see you don't use code-cells at all. I guess code-cells would be primarily useful for ipynb conversion back and forth, right?

Maybe it's as simple as documenting "how to forward modification hooks and custom local variables" between inner and outer modes, with some examples.

This is a good idea. I think that disabling lsp hooks in inner modes might be a good idea, at least till they figure out how to handle chunks in lsp servers.

The lsp-mode folks are trying to get this to work, but it's hard.

Interesting. If they went so far with org-mode I bet it shouldn't be hard to do it generically with arbitrary polymode.

[jdtsmith]

Very glad to hear of your interest. No repo so far, this is just playing around; it was surprisingly easy to get this far. In fact I was only planning a gist or similar on how to "roll your own" simple literate setup which works with lsp-mode and friends (and future friends like tree-sitter), for people who don't like the idea of tangling files and all that. Since I suspect external processes which scan, report & operate in real time on your (full) code file will be even more common going forward, it just seems like a simple and obvious approach (with credit to @astoff and code-cells for inspiration!). I don't see that this type of thing should pertain only to python, since any language which supports block-level comments would work (e.g. /* ... */ in C-family languages).

I see you don't use code-cells at all.

I started with code-cells, hoping to get lsp-mode and multi-markdown working with it. But outshine, polymode, and code-cells all parse and act on the header lines # *, which led to some conflicts.

I guess code-cells would be primarily useful for ipynb conversion back and forth, right?

This is indeed a very useful feature, but it comes with some constraints on the header format (e.g. "percent" format from Spyder, etc.). I would like to be able to write to and read from ipynb notebooks retaining Emacs normal (nested!) outline format (# *** sub-sub-heading). I've posted an issue over on mwouts/jupytext#902. It's possible the light format can be massaged to "do the right thing". I guess emacs outline header title could map to a top-level markdown header, for example. Also as mentioned there, raw multi-line strings (r'''...''') are preferable, since linters etc. complain about incorrect escape sequences otherwise (e.g. in LaTeX equations).

I think a poly-python repo as part of this organization is really overdue. So maybe it's a good time to create one right now with all of the commonly used poly-python modes.

What other custom polymodes are there? Do they go beyond just "enabling multiple specific modes in a buffer"? I envision cell-multiline command sending (ala, or via code-cells), rich TOC/outline navigation/editing (via outshine), lsp-mode/flycheck, and likely some custom bindings (e.g. to create a new code/markdown cell, to evaluate a cell and move to the next one, etc.). Perhaps a poly-python-markdown mode could be rolled that handles a portion of this? But what I envision does rely on several other package. E.g. some people use eglot instead of lsp-mode, so you'd hate to hard-code those things (and yet they need some real non-obvious config to work in polymodes). I also don't want to duplicate what @astoff is already doing. Maybe a reasonable approach is to start with code-cells, integrate polymode-based support for markdown (or other doc format) blocks, and make room for outline management via outshine.

One issue is my inner (markdown) mode's tail-matcher is not robust, since of course you can have multi-line strings in valid python code blocks. I should probably let the tail matcher continue on to match the next header, or the eob, but in testing I wasn't able to get that working (tail and head overlap!). Any thoughts you had would be appreciated.

I think that disabling lsp hooks in inner modes might be a good idea, at least till they figure out how to handle chunks in lsp servers.

Maybe this is what you meant, but in fact you must explicitly call these modification hooks as I've done, or the server gets very confused about the state of the file (the full file is sent only once on first open).

[vspinu]

from ipynb notebooks retaining Emacs normal (nested!) outline format (# *** sub-sub-heading).

This is of course nice but then other editors won't be able to understand that. Maybe a better option would be to stick with %% and customize the outline regexp instead. Ideally light format would allow multiple % do represent nestednes. Thoush, what's the real use case here? I think 99% of the cases is to use single level nesting, thus %% is in principle good enough already.

So I would start a support for the default matchers first and then we see. Though, I am not entirely clear how the default matchers work in the jupytext. Do they support both "light" delimiters and percentage delimiters simultaneously?

I don't see that this type of thing should pertain only to python, since any language which supports block-level comments would work

Yes. This part makes me really exited. There are many languages supported by jupytext but not all have multiline constructs. In the docs they say something that the block matchers can be customized, with one explicit option {{{ standing out as being supported by multiple editors. How a {{{ matched cell looks exactly? Can one specify a language?

raw multi-line strings (r'''...''') are preferable,

My taste would be to have it similarly to how markdown does it but within string delimiters 'md ...', 'R, title, arg=val} ... '. In other words use strings inside the core language, which are multi-line already (in most languages). Or use multiline coments like /*md ...*/. But I guess it's an entirely different format, reverse-markdown or something. Have you heard of anything along these lines out there?

rich TOC/outline navigation/editing (via outshine),

I think code folding and TOC/outline intergration in polymode needs to be implemented generically. It has been requested here before. Let's start a new issue for this if you don't mind. I am curious what are you thoughts on it.

some people use eglot instead of lsp-mode, so you'd hate to hard-code those things (and yet they need some real non-obvious config to work in polymodes

To my mind lsp-mode is a clear winner by now and it looks like for such a complex tooling people should focus on one thing. Bringing a full support for lsp-mode to polymode is a good starting point IMO.

It looks to me that LSP support for markdown proper is not that difficult. So maybe the right approach is to focus on bringing LSP to polymode generically and then have jpynb converted back and forth to markdown. I plan to spend some time with lsp-mode and lsp-org let's see what comes out of it.

One issue is my inner (markdown) mode's tail-matcher is not robust, since of course you can have multi-line strings in valid python code blocks.

This is why a code block of the form r'''md ...''' might be a better idea. But polymode has facilities to not match inner strings already. I just forgot what's the slot name for that config and how it's done. (Have to run now. Will let you know later.)

[astoff]

I think code folding and TOC/outline intergration in polymode needs to be implemented generically.

FWIW, code cells already takes care of outlines. Specifically, code-cells-minor-mode adjusts the outline regexps so that the top level outline is the cell structure, and inside each cell the original outline is kept (suitably demoted). Therefore polymode only needs to take care of the "inner" outline.

I guess code-cells would be primarily useful for ipynb conversion back and forth, right?

It does that, but as noted above it also provides a bunch of additional features to work with "lightweight notebooks". I think that the ability to understand markdown in the text cells is the only big feature that's missing.

Or use multiline coments like /*md ...*/

I'm not sure I understand this. Is this about whether markdown cells go inside a multiline string or a comment?

Note that some languages don't have multiline strings, and some don't have multiline comments.

To my mind lsp-mode is a clear winner by now

I thought exactly the opposite, so we are probably both wrong :-)

[vspinu]

It does that, but as noted above it also provides a bunch of additional features to work with "lightweight notebooks".

I wonder how much of that functionality is a generic functionality that would apply equally well to an arbitrary multi mode buffer. If so then maybe the most utility can be derived by integrating such features into polymode and making a lightweight jypytext mode with converter on top of it. Remains to be seen.

Therefore polymode only needs to take care of the "inner" outline.

It could be that extending code-cells might be the best option in the long run. No-one wants to do double (or triple) work. But how can we do that more concretely? I guess it would not make sense to add polymode dependency to code-cells, right? Maybe there could be a check, if polymode is available, kick in the integration. Alternatively, a separate code-cells-poly-mode can be conceived separately. In any case, the code-cells repo seems like the best place for this. WDYT?

I'm not sure I understand this. Is this about whether markdown cells go inside a multiline string or a comment?
Yes. I was suggesting a lightweight minormode which would work in all prog modes automatically and mimic markdown code blocks syntax but with comments or strings. Not jupyter related.

I thought exactly the opposite, so we are probably both wrong :-)

Hmm. Interesting. In any case if one is integrated, I don't see why the other could not be done similarly.

[jdtsmith]

from ipynb notebooks retaining Emacs normal (nested!) outline format (# *** sub-sub-heading).

This is of course nice but then other editors won't be able to understand that. ..., what's the real use case here? I think 99% of the cases is to use single level nesting, thus %% is in principle good enough already.

I find nested hierarchy an incredibly useful feature of Jupyter notebooks (which use ### markdown headings for the nesting level). Yeah # %%%% Deep Level Heading would also be fine, but I don't think there is back and forth support for even that (?). Ideally you'd be able to convert between (in a script for python-poly-code-cells-outline-lsp-mode):

# %%% Heading [markdown]
r'''
body
'''

and (in an ipynb file):

### Heading
body

in both directions.

So I would start a support for the default matchers first and then we see. Though, I am not entirely clear how the default matchers work in the jupytext. Do they support both "light" delimiters and percentage delimiters simultaneously?

I've not used it so will defer to @astoff, but it appears you have to select your format.

I don't see that this type of thing should pertain only to python, since any language which supports block-level comments would work

Yes. This part makes me really exited. There are many languages supported by jupytext but not all have multiline constructs. In the docs they say something that the block matchers can be customized, with one explicit option {{{ standing out as being supported by multiple editors. How a {{{ matched cell looks exactly? Can one specify a language?

I mean even languages with an interpreter which are not supported in jupyter would be useful to have this.

raw multi-line strings (r'''...''') are preferable,

My taste would be to have it similarly to how markdown does it but within string delimiters 'md ...', 'R, title, arg=val} ... '. In other words use strings inside the core language, which are multi-line already (in most languages). Or use multiline coments like /*md ...*/. But I guess it's an entirely different format, reverse-markdown or something. Have you heard of anything along these lines out there?

I'm not an expert on all the various blended/literate formats, but a requirement for me is "syntactically valid at all times", which limits the markdown-wrapping to either multi-line strings, or block comments (assuming you don't want to go to extra trouble filling/refilling all the line-level comments + adapting markdown-mode to deal with them!).

rich TOC/outline navigation/editing (via outshine),

I think code folding and TOC/outline intergration in polymode needs to be implemented generically. It has been requested here before. Let's start a new issue for this if you don't mind. I am curious what are you thoughts on it.

I mean outshine is just some fancy bindings, navigation, fontification, and conveniences on top of the in-built outline-mode. Both work great, and are the clear choice IMO.

It looks to me that LSP support for markdown proper is not that difficult. So maybe the right approach is to focus on bringing LSP to polymode generically and then have jpynb converted back and forth to markdown. I plan to spend some time with lsp-mode and lsp-org let's see what comes out of it.

That could be in interesting approach: multiple LSP servers servicing one document. But AFAIK the entire protocol assumes full and complete access to an entire code file.

One issue is my inner (markdown) mode's tail-matcher is not robust, since of course you can have multi-line strings in valid python code blocks.

This is why a code block of the form r'''md ...''' might be a better idea. But polymode has facilities to not match inner strings already. I just forgot what's the slot name for that config and how it's done. (Have to run now. Will let you know later.)

OK thanks. I think matching:

# *** Something
r'''

makes for a perfectly cogent head, just the tail is problematic. Could go with:

'''
{any number of blank lines}
# [*]+ .*

but then the final line serves double duty for tail of one and head of the next cell. But yeah "no-tail" (just heads) would be good for this too.

[jdtsmith]

I think code folding and TOC/outline intergration in polymode needs to be implemented generically.

FWIW, code cells already takes care of outlines. Specifically, code-cells-minor-mode adjusts the outline regexps so that the top level outline is the cell structure, and inside each cell the original outline is kept (suitably demoted).

Yeah python-mode does some shenanigans with outline-regexp, which I disable. I don't want to see for loops etc. in my outline. That was the point of the demotion, right?

[vspinu]

With the most recent polymode you can now do

  (add-to-list 'polymode-run-these-after-change-functions-in-other-buffers 'lsp-on-change)
  (add-to-list 'polymode-run-these-before-change-functions-in-other-buffers 'lsp-before-change)

instead of your workarounds.

In fact I have a preliminary generic support for LSP in lsp branch. A bit more work is needed but it's almost there.

[jdtsmith]

Did you figure out the tailless section matching?

[vspinu]

I was confused. There are slots for not allowing nested modes (:allow-nested and :can-nest). But you use case is different. And now that I thought about it I don't quite understand it. How can you have ''' inside r''' ... ''' block in the first place? that would invalidate the block in the host python, no?

[jdtsmith]

OK you're right, to be syntactically correct, you can't nest multi-line strings. So it means you have to be careful not to use ''' in code blocks in your markdown. For other languages this will be a bigger issue, since the tail-matcher may be pretty generic, e.g.

// * Markdown block
/*
- This is markdown
  - And it's good
  - Code looks like:
```c
my_func()
/* A multi-
    line
    comment
*/
```

*/

Just a limitation of this approach. You could always make the code starter/ender more special, like:

// * Markdown block
/* <==START==>
- This is markdown
  - And it's good
  - Code looks like:
```c
my_func()
/* A multi-
    line
    comment
*/
```
<==END==> */

or similar. But that probably doesn't help you, since it won't be syntactically valid for most C-compilers (nested block comments).

Can you briefly mention how your LSP support will work? Setup the on-changes? What else?

[vspinu]

It is already working and it's very simple. It constructs document content and document changes by filling blank lines for other modes and actual text for current mode. Thus line numbers are the same across servers and no buffers or extra files are constructed. It's here. Works for python.

But then I realized today that some servers (java and clangd) require real files and that's a bit more complicated, but well not much more complicated. I don't really know how people are using java and C++ in multimode environments so it's a bit of a stopper ATM.

I think I can merge already, because those servers won't work anyhow, but at least smart servers would work.

[jdtsmith]

Very interesting approach. That’s also how python.el preserves line numbers in tracebacks. Is it similar to lsp-org’s approach? Does it work reliably with changes that span multiple regions? I guess then you fall back on resending the entire file.

So the idea is you don’t need syntactically valid files to keep LSP happy? Not sure about linters/fly-check etc. though. And of course you may have other reasons to do this, like wanting to run the file as a script unmodified.

[vspinu]

It's the simplest approach. No need for indirection, extra buffers or files. For python it works except that linter complains about extra spaces. An edge case for python could be if two inline chunks are on the same line then the "spaced" file might not be syntactically valid, but that's a corner case I am not sure it's worth explicitly dealing with.

I merged lsp branch. let's start experimenting!

[jdtsmith]

It's definitely a very interesting approach! It does seem much simpler than "translating the point to the LSP positions back and forth so the language server thinks that Emacs has opened the original file".

I still think the absolute SIMPLEST approach (from the tooling perspective) is to keep files always syntactically valid, but I understand that does require extra attention and care (e.g. don't ruin it by including nested block comments/multi-line strings inside your markdown region!). Even this could be finessed if you do something like:

• Use line-level comments on EACH Line (# , // ) of your markdown block.
• Hide those line level comments.
• Automatically prepend line level comment marks to each line.

But honestly that sounds complicated to do reliably.

There are also a couple other cases where always-syntactically-valid is desirable:

• most of your server feedback comes over the LSP channel, but you ALSO run some linter/checker/formatter on the actual file itself. ELPY for example does a lot of that.
• You also want to run the file as a script from the command line (python foo.py).

So it would be nice if there were some help maintaining syntactic validity, if that's your preference.

[vspinu]

I still think the absolute SIMPLEST approach (from the tooling perspective)

Right, but this approach has it's limitations, most notably - no multiple inner modes within a file. So I would prefer to make it work generically and disable polymode lsp support for special cases like yours in the host buffer.

[jdtsmith]

I still think the absolute SIMPLEST approach (from the tooling perspective)

Right, but this approach has it's limitations, most notably - no multiple inner modes within a file.

Why would that be? I could for example:

function do_it():
    print("Do it")

# * A markdown cell [markdown]
r'''
- This 
  - list
'''

# ** A code cell
do_it()

# * An ORG-mode cell [org-mode]
r'''
* A Headline

  Some text.
'''

# * A perl cell [perl]
r'''
$var = "testing"
$d{$var}++
'''

[vspinu]

Sure, but then don't you want LSP for perl?

[jdtsmith]

Mind blown! So you want to talk to multiple different LSP servers all from one file! That will be quite the trick. BTW, I was also looking into tree-sitter highlighting; ever tried that? It replaces normal regex-based font-lock highlighting. Not sure if polymode has any way of dealing with many matchers all fighting to highlight the same text.

[vspinu]

So you want to talk to multiple different LSP servers all from one file

That's the current implementation. They all should work because they work independently from different buffers.

I was also looking into tree-sitter highlighting; ever tried that?

Not yet. But I will have a look how to integrate it. Polymode has an abstraction layer over font-lock, so it should be pretty easy I think.

[malcook]

I recently started trying to use lsp in conjunction with poly-org mode (with inner R code blocks) and am finding that if I cut&paste an org section to eslewhere in the document then I find emacs' insertion-point has been relocated to some odd location, oftento end of (outer) buffer, sometimes to first line of next inner buffer.

Before I try and characterize this reproducibly, I looked to see if I should even expect polymode to play nice with lsp and I found this open issue, so I thought I'd ask here what I should expect to work at this point....

I'm using:

GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
version 1.15.12, Xaw3d scroll bars) of 2022-01-19

Org mode version 9.5.2 (release_9.5.2-9-g7ba24c )

poly-R 20210930.1921 installed Various polymodes for R language
poly-org 20220201.1514 installed Polymode for org-mode

[scohen]

I'd like to bump this:
I recently switched from mmm-mode to polymode 20220820.1630, but this caused lsp-mode to start sending invalid document changes. I was able to replicate this fairly easily in a buffer with elixir-mode and smartparens-mode by typing:

defmodule<spc>

which would then create

defmodule do 
end

(smartparens adds the do/end, which I think is messing up polymode)
But the lsp client would send a command that effectively delete the second line from the server, despite it being there in emacs. There was no way to recover this line, and the server would forever be in an invalid state.

[jdtsmith]

Did you try the approach in this comment?

BTW, for people trying to roll their own always-syntactically-valid-literate setup, you might be interested in outli, a lightweight outliner based on outline-mode which includes the styling in the image above.

[scohen]

I did, and it didn’t help matters.
My use case isn’t literate programming, but to support inline templates via sigils, but the issues are identical

[fast-90]

With the most recent polymode you can now do

  (add-to-list 'polymode-run-these-after-change-functions-in-other-buffers 'lsp-on-change)
  (add-to-list 'polymode-run-these-before-change-functions-in-other-buffers 'lsp-before-change)

instead of your workarounds.

In fact I have a preliminary generic support for LSP in lsp branch. A bit more work is needed but it's almost there.

Thank you so much, your post helped me solve a different issue I had with Eglot! (joaotavora/eglot#1229)