Use completion-at-point and abbrev-mode for LaTeX substitution

[non-Jedi]

Disclaimer

Additional commits have been pushed to this branch since this original post was written. This PR now implements immediate substitution of the Unicode symbol for the TeX string upon completion instead of requiring entering a space or symbol or manually triggering expand-abbrev which is shown and described here. This comment is my most up to date summary of the pros and cons of this change.

Original Post

I think this is a more sane version of #82. When | represents the cursor, all of \l|am, \lam|, and \lam|someothertext will complete as expected. The latex backslash-string won't be automatically substituted for the unicode symbol unless the user inserts a whitespace or calls expand-abbrev C-x a e. Ping @nverno as being similar to your personal setup.

Summarizing this PR:

1. Under julia-mode completion-at-point will now complete LaTeX symbols to their full text for both the case of cursor in the middle of the symbol and cursor at the end (\la|m -> \lambda and \lam|othertext to \lambdaothertext).
2. TAB in julia-mode now first tries to indent the line, and if already indented uses completion-at-point to complete LaTeX strings.
3. expand-abbrev (C-x a e) or entering a non-word-syntax character after a LaTeX string will now replace the LaTeX string with its unicode character.
4. TAB does not replace LaTeX strings with unicode characters.

As with #82, I've tried to break this up into conceptually cohesive commits that can be reviewed independently.

Demonstration gif:

[non-Jedi]

@tpapp I was going to push an additional commit that also overrides TAB to dynamically do expand-abbrev as well as completion, but it results in some ambiguous situations: what should be the result of pressing TAB with \sqrt entered, √ or \sqrtbottom (another example, \in could be ∈, \int, \increment, \infty, etc.)? The REPL currently completes it to √, but I'm not convinced this is the best solution for emacs.

I would prefer to keep the concept of completion and abbrevs separate conceptually and possibly bind some other easily accessible key combination to expand-abbrev. Thoughts?

[tpapp]

Sorry for not getting back to you on this yet, I will need some time to review it but may not get to it until the middle of this week.

I will think about this question too.

[tpapp]

Thanks for doing this, and in such a clean way. I like the code.

I would like to merge this in some form, but I am wondering what the best interface is (or if people prefer various versions, how to support them all) so I am inviting people to comment on how they use completion.

Personally, I would prefer the following: a single TAB

1. first tries to indent. if it did, then it is done. if the indentation was unchanged, then:
2. whenever the point is in or immediately after a complete LaTeX abbreviation, it is replaced by the relevant abbrev, eg \sqrt is replaced by √,
3. if no complete abbreviation is found, the abbrev window pops up with possibilities (if there are any). so TAB on \sqr would offer \sqrt, \sqrint, and \sqrtbottom. If the users selects one, the replacement would just happen without further prompting.

The motivation for the above is that trying to enter a \alpha into a Julia buffer is probably not a frequent use case, the user just wants the α, so we should go for it immediately without further keypresses. I find having to Space BackSpace cumbersome.

I don't know enough about abbrev to say whether that is the right implementation for this, but I am fine with using it.

[nverno]

I think this is a huge improvement -- nice work!

first tries to indent. if it did, then it is done. if the indentation was unchanged, then:

This is the configuration I prefer as well, but it should all conform to the emacs conventions set with tab-always-indent, in that case, using complete.

whenever the point is in or immediately after a complete LaTeX abbreviation, it is replaced by the relevant abbrev, eg \sqrt is replaced by √,

I don't think converting \sqrt to √ without further input is a good solution, cause you might still be planning to write something else -- the julia REPL doesn't do this either (at least not in my version).

I do think it would be an improvement to convert the abbrevs when the next input is a symbol, howerver, eg. \alpha* => α*. In the sqrt case, it would be nice to expand if the next input was a number, perhaps, but sadly numbers have word syntax - so some other workaround would be required probably.

Last time I looked into it, abbrevs were expanded after word syntax at the C level (meaning if the last user input was a '\sqrt5', only an abbrev matching that would expand - and expansion isn't attempted if the last input is non-word syntax, eg '*'), but customization could be added in a post-self-insert-hook (I left one that works for symbols in the review).

The only thing I would change is giving the latex abbrev table a non-default name, so that user defined abbrevs still work (eg. they aren't also required to have the special expansion regexp requirement). An alternative could be to add the latex abbrev table as a parent of the julia-mode-abbrev-table.

[nverno]

If you want to expand-abbrevs after any punctuation, eg. \alpha* => {laxex alpha}*, you could add
a local post-self-insert-hook along the lines of

(defun julia-abbrev-expand-after-symbols-hook ()
  (and (memq (car (syntax-after (1- (point)))) '(0 3))
       (setq this-command 'julia-abbrev-expand-after-symbols)
       (expand-abbrev)))

which I use in, eg. elisp, where I want my abbrevs to expand after I enter a ')' for example.
It could be another user choice to consider. IIRC I needed to add this as a post-self-insert command because abbrev expansion after word syntax was hard-coded in C, but the implementation may have changed.

[non-Jedi]

The problem with this is all of the LaTeX codes that include punctuation (or even parentheses). Some examples: \^), \_=, \:soccer:, \^+. I guess in practice, as long as none of the LaTeX codes that include symbols have other LaTeX codes that are superstrings of them cut off at the symbol it could be fine.

[tpapp]

I don't think converting \sqrt to √ without further input is a good solution, cause you might still be planning to write something else

Then this may just be my personal preference --- if I want to choose between \sqrt and \sqrtbottom, I would complete on \sqr and pick from the popup. If feasible, keeping this modular and customizable would be nice.

[nverno]

Some examples: ^), _=, ⚽, ^+

Good point -- the above is too simple for this case

I suppose you could make it more complicated and check that the prefix so far isn't a prefix of other latex symbols in your table -- but then I think that would be pushing the amount of computation you would want to do in a post-self-insert-hook

[nverno]

I also really like the completion working in the middle of symbols, as it does in elisp.

[nverno]

I would prefer to keep the concept of completion and abbrevs separate conceptually and possibly > bind some other easily accessible key combination to expand-abbrev. Thoughts?

Agreed, I think intertwining them is likely to break stuff. Abbrevs have been around forever, having their own distinct behaviour. Also, as with everything, the more modular the better - you would want to allow people to opt out of abbrevs, but use completion, and vice versa IMHO.

[non-Jedi]

I don't know enough about abbrev to say whether that is the right implementation for this, but I am fine with using it.

It would be a bit hacky, but I can build that interface on top of capf and abbrev, which would still allow people to customize a variable to use capf and abbrev in the emacs default manner. As mentioned above, doing so would not be my preference, but I can certainly implement that if it's the consensus.

[non-Jedi]

The only thing I would change is giving the latex abbrev table a non-default name, so that user defined abbrevs still work (eg. they aren't also required to have the special expansion regexp requirement). An alternative could be to add the latex abbrev table as a parent of the julia-mode-abbrev-table.

Could you expand on what you mean by here? Specifically "default name" isn't clear to me, nor is it clear how this interferes with regexps for user-defined abbrevs. And what latex abbrev table are you recommending we inherit from? Keep in mind this PR is my first time dipping my toes in the waters of abbrev-mode, so if something isn't in the abbrev info nodes (either under elisp or emacs), I probably don't know it.

[nverno]

Call it julia-mode-latex-abbrev-table and set it as the parent of julia-mode-abbrev-table -- then you get normal behaviour

[non-Jedi]

I've rebased on master and pushed a commit fixing the issue with the regex clobbering user abbrevs noted by @nverno. Now just awaiting feedback/decision on integrating LaTeX substitution/expansion.

[tpapp]

Can you briefly summarize that the alternatives are, and the trade-offs (if any?)

[non-Jedi]

Sure. Trying to be as brief as possible:

1. Leave as is in this PR.

• Unless user customizes julia-force-tab-complete, TAB tries to indent then complete to LaTeX code, and either inserting a whitespace char or calling M-x expand-abbrev (bound to C-x a e) causes the unicode character to replace the LaTeX code.
• Advantage: Fits naturally into the builtin Emacs concepts of completion (substring to full string) and abbrevs (string to other string) which will be less surprising for people used to Emacs conventions for other languages.
• Advantage: less code to maintain and generally the most robust solution.
• Disadvantage: Requires more key-strokes.
• Disadvantage: Inserting LaTeX symbols into a string, docstring, or comment becomes difficult.
• Disadvantage: abbrev-mode and completion-at-point need identical tables in slightly different data structures, so twice as much memory is used.

2. Add an :exit-function which calls expand-abbrev to the completion-at-point functions to replace LaTeX codes with unicode symbols as was done in Add completion-at-point function for completing latexsubs #82. We can check a customizable variable to decide whether to do the replacement or not. To make this work robustly with the many completion frameworks out there, the :exit-function will have to run on both 'sole and 'finished.

• Advantage: feels very natural to use--similar to Julia LaTeX substitution in the REPL (with company-mode).
• Disadvantage: it's unclear when a properly implemented completion framework might call :exit-function with 'sole and how performing the substitution before the completion is 'finished might screw things up.
• Disadvantage: abbrev-mode and completion-at-point need identical tables in slightly different data structures, so twice as much memory is used.

3. Do option 2 but only when :exit-function called with 'finished.

• Disadvantage: ivy is buggy (ivy-completion-in-region-action should pass 'finished to exit function if no further completion possible abo-abo/swiper#2345), so users using ivy won't have their LaTeX codes substituted.
• Advantage compared to Add completion-at-point function for completing latexsubs #82: Those users can still manually call expand-abbrev.

4. Do option 1 but bind M-s to expand-abbrev.

• Less keystrokes than 1 but still more than 2.

5. Create our own function to bind to TAB which calls out to completion-at-point and manually substitutes unicode characters afterward.

• Disadvantage: company-mode doesn't use completion-at-point to trigger it's completion, so this approach is incompatible with it.
• Disadvantage: requires reimplementing a lot of Emacs machinery that already exists.

6. Turn all of this into a company backend instead.

• Advantage: company gives significantly better completion experience than Emacs out of the box.
• Advantage: probably allows us to implement both completion and substitution at once without making tradeoffs.
• Disadvantage: Probably jarring for somebody used to builtin completion, ivy, or auto-complete.
• Disadvantage: julia-mode is no longer dependency-free.

There may be other options and advantages/disadvantages I've missed, but those are the ones that have been bouncing around in my head.

[tpapp]

Sorry for being dense, but I am somewhat confused. I have re-read the history, and found that the original motivation of this string of PRs, starting with #80, was that

This would integrate nicely with e.g. company and would generally make this package a better emacs citizen.

I am generally in favor of doing these things, but from various experiments it appears to be difficult to do this in a way that preserves the single-keystroke behavior we have currently. Given that we have a working solution, should we switch to one of the backends at this cost, just to be a better Emacs citizen?

Note that I like the current setup, so it is possible that I am missing something that someone else's workflow/setup requires. I am happy to accommodate that, but I am a bit lost why this is needed.

That said, this PR has a lot of code improvements which should be incorporated.

[non-Jedi]

I'm afraid my phrasing in #80 reflects my poor understanding (at the time) of how completion works in Emacs. Sorry about the confusion caused. The real benefit here is that by tying into completion-at-point, we get real completion of LaTeX strings (\lam to \lambda) and substitution rather than just substitution of already-complete LaTeX strings (\lambda to λ).

[non-Jedi]

@tpapp what's the path forward here given the deafening silence of user-input on desired behavior?

[tpapp]

Apologies for the lack of feedback. This is something I want to think about a bit and experiment with, but I am overwhelmed with other stuff at the moment. I understand that putting in the work and not getting it merged can be frustrating, but I ask for your further patience.

[non-Jedi]

No problem. And no pressure from me with respect to getting this or any pull-request i make merged (and definitely no frustration). I am already happily using all changes I've proposed on my own machines. Just wanted to make sure you weren't waiting on some other input or change from me. :)

[non-Jedi]

@tpapp you had time to look at this yet? I understand if not, but I thought I'd ask since rebasing my personal devel branch onto this when I make an unrelated PR is becoming slightly burdensome.

I thought I should also explain a bit better my objection to our current approach: it breaks composibility (and doesn't offer real completion). Emacs has builtin facilities for both "completion" (\la to \lambda is done by completion-at-point) and "replacement" (\lambda to λ is done by abbrev-mode). Both have all sorts of hooks that users and other emacs packages can use to customize how they work; if we implement our own version of either (as julia-latexsub currently does for replacement), none of that works. It's the same reason it's generally a good idea to use the "Iterable" interface from Base rather than rolling your own: you get all sorts of great interop and functionality for free.

[tpapp]

Thanks. Can the warning

Compiling file /home/tamas/src/julia-emacs/julia-mode.el at Mon Dec 19 22:27:30 2022
Entering directory ‘/home/tamas/src/julia-emacs/’
Warning (bytecomp):  docstring wider than 80 characters
Warning (bytecomp): the function ‘julia-company-indent-for-tab-command’ is not known to be defined.

be somehow suppressed?

FWIW, I am always happy to drop support for old Emacs versions.

[non-Jedi]

I'm not seeing byte-compilation errors locally after the last commit.

[non-Jedi]

@tpapp Friendly ping. Are you ready to merge this? Should I push an additional commit changing minimum emacs version and test targets and a changelog update?

[tpapp]

@non-Jedi: sorry I was sick for 2 weeks and I am still catching up.

I promise that I will merge this PR once I had a chance to evaluate it. The problem is that my emacs config contains a lot of idiosyncratic cruft that interferes with completion. I need to bisect and clean it up.

I realize thay you did a lot of work on this and it is frustrating to wait for one person. So if anyone else wants to merge this, feel free to do so

In the meantime, please understand that additional work you do on this will not be wasted and this will definitely get merged.

[non-Jedi]

@tpapp I hope you're feeling better! It always stinks being sick over the holidays.

No pressure on getting this merged; just let me know if I can help any more with making this play nicely with your config (I had hoped the recent company-specific commits would have fixed most issues, but no dice, huh). I appreciate you doing the thankless work of maintaining this and reviewing PRs.

[tpapp]

@non-Jedi: thanks for your patience. I have cleaned up my Emacs config and it works fine (I think it was some interaction with some ido thing, but I did not track it down). I have now switched to corfu.

I have been testing it for a week and it is superb. I am now happy to merge this, I think it has been reviewed quite thoroughly so I will not ask for another round of reviews. I will wait until Monday for comments though.

[tpapp]

NOTE after merging,

1. drop support for 25, require at least Emacs 26
2. remove 25.* from tests
3. bump version to 0.5, this is a significant feature change

[tpapp]

I have an issue with this PR, I wonder if anyone can replicate: in the buffer I have

α\hat_mean

with the cursor on the _. I press TAB, menu pops up, I select \hat. Then I end up with

α̂

ie the _mean disappears.

[non-Jedi]

Can confirm locally. I'll take a look. Seems to be an issue with my code as it happens with both emacs-native completion and corfu.

[tpapp]

@non-Jedi: I have been studying your code, the Emacs completion system (of which I am completely ignorant), and thinking about the design of my ideal completion system.

It is clear that completion should always start on the \, and if we traverse backwards and encounter something that cannot be part of a valid symbol, we should cop out early. So I am fine with how julia--latexsub-start-symbol works.

I am wondering if more effort could be spent on finding the end though. Specifically, from the \ search until the longest string is found which could be part of a valid prefix. Thus,

\lam|da00
\lambda|=

would complete on \lambda (| is the point), while

\lam|=
\l|am_1

would complete on \lam. I realize that it may be expensive to do this, but perhaps an efficient data representation can be found for substrings, eg using a tree with valid next characters.

Then, given this, we could offer a pre-filtered table from the julia-mode-latexsubs, just with the valid completions based on the part we found. Does this make sense? Or do we have to return the whole list?

[tecosaur]

Have you seen https://github.com/ymarco/auto-activating-snippets ?

[tpapp]

@tecosaur: thanks, I was not aware of it. I would prefer to keep the completion depend only on the contents of the buffer and the location of the point.

[tecosaur]

Sure, but it's doing exactly what you talked about regarding an efficient (tree) representation of valid next chars.

[non-Jedi]

@tpapp the problem is that trying smartly decide where the entity you're completing from the middle of results in ambiguity.

Let's take \l|am as an example where | represents point. Our completion function needs to return a list of the form (START END COLLECTION . PROPS). start is obviously 1 where the backslash is, but end could be either 3 or 4 depending on what the user wants. Maybe the user is trying to write ⪫m (\lat m), and we should make end 3, or maybe the user is trying to write λ (\lambda). One of those seems much more likely to me as a human, but I don't know how to programmatically distinguish between the cases.

One hacky solution I could try would be to create a series of ~10 completion-at-point-functions with increasing static amounts of look-ahead ((setq end (+ (point) lookahead)), but doing so is more than a bit of a hack, and I'd start to be slightly worried about slowing down completion in that case.

[non-Jedi]

EDIT: I pushed a commit that fixed this issue without any compromises. Leaving the original text as-is for the explanation of the bug. See f359fd0 for the solution.

I have an issue with this PR, I wonder if anyone can replicate: in the buffer I have

α\hat_mean

with the cursor on the _. I press TAB, menu pops up, I select \hat. Then I end up with

α̂

ie the _mean disappears.

I figured out what's going on here, but I'm not sure what the best way to fix it is. Here's
step-by-step how we end up in this situation:

1. julia-mode-latexsub-completion-at-point-around determines the completion candidates
   will be for the buffer positions corresponding to \hat_mean.
2. The completion-at-point machinery deterines that \hat is a potential completion for
   \hat_mean according to the user's completion-styles--specifically the emacs22
   completion style which ignores text in the buffer after point.
3. When the user selects \hat as the completion, completion-at-point does what it's
   supposed to do and replaces the completion of candidate of \hat_mean with the selected
   completion of \hat.
4. The :exit-function then replaces \hat with it's appropriate unicode character.

Here's roughly what we'd like to happen instead which is what does happen if emacs22 is
removed from completion-styles:

1. julia-mode-latexsub-completion-at-point-around determines the completion candidates
   will be for the buffer positions corresponding to \hat_mean.
2. The completion-at-point machinery finds no potential completions for \hat_mean.
3. completion-at-point then calls julia-mode-latexsub-completion-at-point-before which
   determines that completion will be for the buffer positions corresponding to \hat.
4. From here, everything proceeds as the user would expect.

Right now, I only see a few options for fixing this:

1. Disable all completion styles other than basic in the julia-mode buffer.
   • This will subvert user expectations if they have any other
     completion-at-point-functions that are active in julia-mode buffers.
   • I don't think emacs provides any facilities for modifying completion-styles only for
     specific completion-at-point-functions.
2. Remove julia-mode-completion-at-point-around.
3. Pre-filter the selection of completions returned for
   julia-mode-latexsub-completion-at-point-around.
   • This is the option I'm currently planning on implementing.
   • The emacs manual advises against doing so, but I think our application makes sense as
     an exception to the guideline:

Additionally, the COLLECTION should generally not be pre-filtered
based on the current text between START and END, because that is
the responsibility of the caller of ‘completion-at-point-functions’
to do that according to the completion styles it decides to use.

[tpapp]

At this point I have nothing against pre-filtering, but I realized that I cannot spare the effort of reading the whole Completion chapter in the manual and then looking at what other language modes do. As typical for Emacs, there are a few hundred different ways of doing what we want, we should opt for the one that meshes the best with the completion frameworks people introduced recently. But I have no detailed knowledge of how these work.

Also, I am happy to get rid of "around" completion if we need to, or severely restrict it (eg only works on full escapes, or whatever we need).

[tpapp]

The latest commit does not fix the disappearing suffix issue for me.

However, I am now warming to the idea of removing "around" completion. I previously thought that it is a nifty feature, but now I do not think it is so important. The main purpose of this feature is input of UTF-8 characters, ie you input them and you are done. Typing \alpha, not replacing, and then going back to get α is perhaps not something that we should handle via this mechanism, in Emacs it would be more natural to use query-replace etc for that.

But I would like to make this mode comfortable for all users if feasible, so please comment. @adam-higuera (cf #70)?

[tpapp]

@non-Jedi: I brute-forced end lookup with a hash table, please take a look at non-Jedi#1, a PR against your branch. Personally I find it performant.

[tpapp]

I have been testing it the above modification two days and find it nice, does exactly what I want. Is anyone else testing this PR in practice? Please try out non-Jedi#1 and let me know how it goes, especially wrt to performance and corner cases.

Now that tree-sitter is on the horizon I would like to merge this in a couple of weeks.