Migrate JavaScript development to Eglot from Emacs LSP : Node, Deno, and Monorepos

[cowboyd]

Hello,

My Holiday project is to take a stab at migrating from emacs-lsp to eglot for JavaScript development. Unfortunately, I've had a pretty rough go of it so far. The problem I'm having is that the repos I'm operating in have multiple JavaScript projects, some of which use Node as the primary development tool, others which use Deno as the primary development tool.

I need (I think) for Eglot to recognize each individual javascript package as a separate project apart from the main vc-project. Furthermore, if the package has a deno.json file, then use the deno lsp command, otherwise, use the typescript-language-server command.

Thus far, I've tried to follow what I can piece together from the documentation and this discussion on using an alternate project root to add a hook to project-find-functions. Here's my set up so far

(defun @cowboyd/find-nearest-dominating-file (files dir)
  "Find first one of `files' searching upwards from `dir' "  
  (if (string= dir "/") nil
    (let* ((contents (directory-files dir))
	  (file (seq-find (lambda (file) (seq-contains-p contents file)) files)))
      (or
       (when file (concat (file-name-as-directory dir) file))
       (@cowboyd/find-nearest-dominating-file files (directory-file-name (file-name-directory dir)))))))

;; get nearest Deno project file that does not have a Node project file in front of it.
(defun @cowboyd/deno-project (&optional startdir)
  "Is the current file inside a deno (and not a node) project. If a 'package.json' is found first, then this is not a deno directory"
  (let* ((dir (directory-file-name (or startdir (file-name-directory (buffer-file-name)))))
	(project-file (@cowboyd/find-nearest-dominating-file
		       (list "deno.json" "package.json")
		       dir)))
    (when (string= "deno" (file-name-base project-file)) project-file)))

(defun @cowboyd/deno-eglot-project (dir)
  "Return a deno directory scoped project root for eglot if this file is in a deno directory"
  (and (boundp 'eglot-lsp-context) eglot-lsp-context ;;; short circuit if this is not eglot searching for project
    (let ((deno-config (@cowboyd/deno-project dir)))
      (when deno-config (cons 'transient (file-name-directory deno-config))))))
  
(add-hook 'project-find-functions '@cowboyd/deno-eglot-project)

However, as far as I can tell whenever I switch to the main project and find a file located in the sub-package, (project-current) is already computed for the buffer and assigned somewhere, and so my custom find function is never run. As a result, Eglot retrieves the root vc directory instead of the specific directory.

It does appear to get the right directory if I open the sub-package file directly.

Finally, given that I have a way to detect between Deno projects and Node projects, what is the best way to get Eglot to select the right language server dynamically?

emacs-lsp solves this problem by letting you explicitly state via a config file that a directory is a workspace root. I haven't seen this as an option in Eglot (I could be wrong). But anyway, I've been looking to migrate over to eglot for some time, and so any insight would be very much appreciated.

[joaotavora]

I'll try to read this later, as one of my holidays projects is resting, but seems very similar to #1336

[dgutov]

project.el supports subprojects using project-vc-extra-root-markers.

I'm surprised you don't remember that.

[joaotavora]

@cowboyd (presumably) wants these smaller projects for Eglot connection purposes. If they use your technique, C-x p f and a lot of other commands becomes useless for jumping between files of different modules in the overarching project.

Even if @cowboyd doesn't want to preserve that specifically in this instance, I do. So Eglot-specific subprojects are better than no subproject support at all in project.el. It that existed in some useful form, this probably wouldn't be needed.

I'm surprised you don't remember that.

I'm surprised you didn't notice my explicit ping here: #1336 (comment), where I told that other user there are easy ways to control projects Emacs-wide:

You tell your Emacs session as a whoel that the root of the project is not the root of the repo. -- for that particular project. That affects everything project related, not just Eglot.
[...]
Maybe there's an easy way to do the first. Ask the project.el manager @dgutov .

[joaotavora]

I didn't tell the user about project-vc-extra-root-markers specifically because I couldn't remember the name, I don't exactly know how they work, and I suspected they would be rather useless (at least for me) with the above limitation. I seem to have confirmed my suspicion. If I put (setq project-vc-extra-root-markers '("package.json")) and have a large hierarchy and project.json appears in some intermediate dir. I can't use C-x p f to jump to files outside that intermediate dir from directories inside that intermediate dir.

EDIT: even stranger is that if I jump from the outside project to the inner one with C-x p f, I'm trapped, can't jump back.

[dgutov]

I didn't tell the user about project-vc-extra-root-markers specifically because I couldn't remember the name

M-x customize-group is your friend. No need to call me in for simple questions every time.

I can't use C-x p f to jump to files outside that intermediate dir from directories inside that intermediate dir.

You either can't use that, or you can't use it to jump across the narrowed "subproject" only, which is also useful. The choice between the two is arbitrary, and it's not obvious which one is more useful in general. At least the current way allows "switching" to top-level project by going up a few dirs.

even stranger is that if I jump from the outside project to the inner one with C-x p f, I'm trapped, can't jump back

The users who I asked previously said that this is okay and in some ways preferable. Projectile does this too, for example.

Would you prefer that the "outside" project excluded subproject files? That would be "consistent", but would it be more useful?

Anyway, I'm sure you know the appropriate venue for reporting bugs and requesting features.

[joaotavora]

M-x customize-group is your friend. No need to call me in for simple questions every time.

Why bother? Just because I didn't know the name didn't prevent me from correctly it would be useless, as has been shown. But I'll leave you alone. Just don't be "suprised"

The users who I asked previously said that this is okay and in some ways preferable. Projectile does this too, for example.

Preferable to be stuck?

Would you prefer that the "outside" project excluded subproject files?

No. I would prefer subprojects to be useful in navigating for large monorepos.

Anyway, I'm sure you know the appropriate venue for reporting bugs and requesting features.

Don't hold your breath. I've already done that, and even supplied a patch.

[cowboyd]

When it came to auto selecting typescript or deno, this worked like a charm! Now I just need to figure out how to parse it in my mind. In particular, I haven't yet grokked this expression:

`((js-mode (typescript-ts-base-mode :language-id "typescript")) .
               ,(my/deno-or-tss))

But I'm keen to figure it out.

While the lsp connection worked for individual node and deno projects, I was not able to get the project-find-function to work as it keeps giving an error in all contexts:

Symbol's value as variable is void: eglot-lsp-context

[joaotavora]

I haven't yet grokked this expression:

That's just creating a function instead of a list of strings, which is a legal object according to the docstring of eglot-server-programs. The function of one argument is returned by a call to (my/deno-or-tss). There are other ways to do it, like putting the lambda there directly. In fact, eglot-alternatives is another example of such a function-returning function.

Symbol's value as variable is void: eglot-lsp-context

You have to have either the latest bleeding edge eglot or put (setq eglot-lsp-context nil) somewhere in there.

[joaotavora]

That's just creating a function instead of a list of strings, which is a legal object according to the docstring of eglot-server-programs. The function of one argument is returned by a call to (my/deno-or-tss). There are other ways to do it, like putting the lambda there directly. In fact, eglot-alternatives is another example of such a function-returning function.

I was overcomplicating this. If you want, erase the above from your head, and use this instead, which is easier to grok:

(defun my/find-marker ()
  (or my/marker
      (setq my/marker
            (catch 'done
              (locate-dominating-file
               default-directory
               (lambda (d)
                 (dolist (f '("package.json" "deno.json"))
                   (let ((marker (expand-file-name f d)))
                     (when (file-exists-p marker) (throw 'done marker))))))))))

(defun my/deno-or-tss-v2 (&optional interactive)
  (let* ((m (my/find-marker)))
    (cdr
     (assoc
      (file-name-nondirectory m)
      '(("package.json" "typescript-language-server" "--stdio")
        ("deno.json" "deno" "lsp" :initializationOptions '(:enable t :lint t)))))))

(defun my/project-find-function (dir)
  (when-let ((f (and eglot-lsp-context (my/find-marker))))
    `(transient . ,(file-name-directory f))))

(add-hook 'project-find-functions 'my/project-find-function)
(add-to-list 'eglot-server-programs
             '((js-mode (typescript-ts-base-mode :language-id "typescript"))
               . my/deno-or-tss-v2))

I will be working to simplify this further in the bleeding-edge Eglot, and maybe put some recipe in the manual

[joaotavora]

In the very latest Eglot, you can now do this. Take note, however that the
project type represented by (eglot--project ROOT KEY1 VAL1 KEY2 VAL2) is still experimental
and so is this whole technique. But I won't remove support for it unless we get something much better.

I'm still working out how this fits with the existing project-specific and user-specific configuraiton.

(defun my/deno-or-tss-v3 (interactive project)
  (let ((marker (and (eq (car project) 'eglot--project)
                     (plist-get (cddr project) :marker))))
    (cond ((string= marker "package.json")
           '("typescript-language-server" "--stdio"))
          ((string= marker "deno.json")
           '("deno.json" "deno" "lsp" :initializationOptions '(:enable t :lint t)))
          (interactive
           (message "Can't guess server invocation for '%s'" (project-root project))
           nil))))

(defun my/project-find-function (dir)
  (let* (marker
         (root (locate-dominating-file
                default-directory
                (lambda (d)
                  (dolist (f '("package.json" "deno.json"))
                    (when (file-exists-p (expand-file-name f d))
                      (setq marker f)))))))
    (when root
      ;; Use experimental eglot--project project type which has some
      ;; space for user-defined props, `:marker' in this case
      `(eglot--project root :marker marker))))

(add-hook 'project-find-functions 'my/project-find-function)
(add-to-list 'eglot-server-programs
             '((js-mode (typescript-ts-base-mode :language-id "typescript"))
               . my/deno-or-tss-v3))

[cowboyd]

@cowboyd (presumably) wants these smaller projects for Eglot connection purposes. If they use your technique, C-x p f and a lot of other commands becomes useless for jumping between files of different modules in the overarching project.

☝🏻 This is actually what caused me to seek help in the first place. Using C-x p f was always assigning the wrong project to typescript files in the deno-based sub-folder.

As far as retaining the ability to have your own concept of an eglot project, I think it's wise. One of the things that emacs-lsp gets right is that it keeps its own conception of what and where a project is and as a result getting started is really straightforward. It just makes on-boarding from wherever a user happens to be much more friction-free.

Apart from that, I must say that I am really enjoying eglot thus far. I just love how well it slides into all the other packages which I'm already using rather than maintain its own parallel ecosystem of everything. Nice work!

Question: Before I decide on whether to go "bleeding edge. " How often do you release "stable" versions of eglot to ELPA?

[joaotavora]

rather than maintain its own parallel ecosystem of everything. Nice work!

That's the general idea yes. Glad you like it.

As far as retaining the ability to have your own concept of an eglot project, I think it's wise.

Yes, retaining this ability is important, but if "subprojects" had better support in project.el, maybe Eglot wouldn't need to exercise it as often, as it could just hook onto the inner project by default, the inner project presumably being what the hypothetical project-current would return either by default or when given a ':inner t' argument somehow. C-x f p and other tools would maybe default to the outer project, like they do now. Then maybe, in some (most?) cases, an Eglot specific concept wouldn't be required.

Not sure how vc-extra-markers fit into the picture, they don't seem very useful now (at least for my own megarepo over at $DAYJOB, where marker files are unfeasible).

Maybe all of the above or equivalent will happen, but I I wouldn't hold my breath.

How often do you release "stable" versions of eglot to ELPA?

Not very often I'm afraid. You can help me maintain this practice, by pinging me, testing and keeping the NEWS file up-to-date :-). Anyway the bleeding edge is what I use myself.

[cowboyd]

@joaotavora I've finally got around to trying out the bleeding edge and subprojects (I've been limping along and mostly been fine, but enough was enough)

I had to adapt it since it wasn't quite working. I had to unquote root and marker in the project definition... this seems to work?

(defun @cowboyd/deno-lsp-or-typescript-language-server (&optional interactive project)	 
  (let ((marker (and (eq (car project) 'eglot--project)
                   (plist-get (cddr project) :marker))))
	   (cond ((string= marker "package.json")
		  '("typescript-language-server" "--stdio"))
		 ((string= marker "deno.json")
		  '("deno" "lsp" :initializationOptions '(:enable t :lint t)))
		 (interactive
		  (message "Can't guess server invocation for '%s'" (project-root project))
		  nil))))

(defun @cowboyd/js-project-find-function (dir)
  (let* ((marker nil)
         (root (catch 'done
		 (locate-dominating-file
                  default-directory
                  (lambda (d)
                    (dolist (f '("package.json" "deno.json"))
                      (when (file-exists-p (expand-file-name f d))
			(setq marker f)
			(throw 'done (file-name-parent-directory (expand-file-name f d))))))))))

    (when root
      ;; Use experimental eglot--project project type which has some
      ;; space for user-defined props, `:marker' in this case
      `(eglot--project ,root :marker ,marker))))

(add-hook 'project-find-functions '@cowboyd/js-project-find-function)
(add-to-list 'eglot-server-programs
             `((js-mode (typescript-ts-base-mode :language-id "typescript"))
	       . @cowboyd/deno-lsp-or-typescript-language-server))

Also, note that I had to rejigger the let* to throw the root directory since it was never getting set before.

So far, so good. I'll let you know if anything changes.

[cowboyd]

Everything seems great, however whenever I am in a (eglot--project) I'm not able to switch out of it using (project-switch-project) which is pretty core to my workflow. Basically what happens is that when I select a project to switch to, it just returns be back to the eglot project

[cowboyd]

Another thing I've noticed is that there is a very significant performance degradation when finding files with (project-find-file) in the larger javascript monorepo that is all under a single project. Sometimes it can be several seconds between when (project-find-file) is invoked and when results are shown.

[dgutov]

a very significant performance degradation

I suppose that's due to your project definition being applied generally, and not just for eglot (when it asks for it).

A general-purpose project backend needs to have an optimized project-files implementation; quite a bit of effort went into the one that the project-vc backend has.

[joaotavora]

I suppose that's due to your project definition being applied generally, and not just for eglot (when it asks for it).

That's a possibility yes, seems like the eglot-lsp-context isn't being taken in consideration anywhere (not in my examples, nor in @cowboyd's)

A general-purpose project backend needs to have an optimized project-files implementation; quite a bit of effort went into the one that the project-vc backend has.

As far as I can tell, it's fast for small projects, but rather useless for larger ones. A better performing implementation for larger repos should use external-completion.el, not attempt to transfer all the possible file names into the Emacs address space and find some specialized external tool that keeps an index (like everything.exe on Windows, locate or fdfind on linux, or some custom-made tool just for Emacs). Much like symbol completions, you'll lose styles but you'll gain heaps in responsiveness. Just an idea.

[dgutov]

project-file-completion-table was the "canonical" generic at some point, see e.g. commit 55ec674f5090. IIRC that made certain things more difficult, and induced more copying (thus making things uniformly slower for implementations that don't use external storage).

Basically, if you want to show file completions even for empty input, this is where the problem starts. Could it be solved with a combination of a streaming interface and external filtering? Maybe.

[joaotavora]

I had to adapt it since it wasn't quite working. I had to unquote root and marker in the project definition... this seems to work?

unquoting is obvisouly needed yes. I was going to swear I tested that code, but this means I haven't.

Everything seems great, however whenever I am in a (eglot--project) I'm not able to switch out of it using (project-switch-project) which is pretty core to my workflow. Basically what happens is that when I select a project to switch to, it just returns be back to the eglot project

This would need a repro, so we can edebug it.

Another thing I've noticed is that there is a very significant performance degradation when finding files with (project-find-file) in the larger javascript monorepo that is all under a single project. Sometimes it can be several seconds between when (project-find-file) is invoked and when results are shown.

Does this happen without any subprojects at all? Or only when you plug in your solution?

If the former, I can tell you I also work with a (veeery large 600 K files, larger than yours? 🤡 ) and M-x project-find-file is positively useless there, even with the Git stuff it leverages. A better alternative would to use something like Everything.exe for windows (https://www.voidtools.com/downloads/) (or, in linux, maybe something equivalent like locatedb) and external-completion.el for responsiveness. I'm using this quick hack to contact an Everything.exe server: https://github.com/joaotavora/eel. It's fairly performant, and above all, responsive.

It's not plugged into project.el's C-x f though, so I just bind a key. No idea if it can even be plugged, probably yes. project.el seems very VC/Git-centric, but who knows maybe you can just feed it a completion table.

[dgutov]

I also work with a (veeery large 600 K files, larger than yours? 🤡 ) and M-x project-find-file is positively useless there

There is a certain threshold for the project size where the delay becomes too much, that depends on the person and the machine, though. With a 200K project I'm testing, project-find-file takes about 1 second, which is not great but might be considered manageable by some. There is still some optimization space left, but after that, I guess it's back to caching the file list, like Projectile does. Or external databases like in your approach.

[joaotavora]

I guess it's back to caching the file list

You run into cache invalidation problems. And unless you do things carefully and laboriously in C, you'll waste a lot of space and leave performance on the table. Supporting styles will be hard. External databases specialized for this purpose can use all the CPU and have optimized data structures.

[dgutov]

And unless you do things carefully and laboriously in C, you'll waste a lot of space and leave performance on the table

Moving some hot path into C is usually an option, the main question is how to organize the logic on the high level.

External databases specialized for this purpose can use all the CPU and have optimized data structures

How do those do cache invalidation?

[joaotavora]

Moving some hot path into C is usually an option, the main question is how to organize the logic on the high level.

If that hot path includes anything related to Lisp completion styles, you've lost a big chunk of what makes having it in Emacs worth it. And leveraging SMP and/or SIMD as I suspect these super-fast trendy rust alternatives do is still a big challenge.

How do those do cache invalidation?

no idea, the point is it's not my problem. as far as I can tell they work pretty well.

[joaotavora]

For what it's worth, I've recently bumped into a similar problem, and came up with this refinement of a previous solution in this discussion:

(defvar eglot-lsp-context)
(defvar project-find-functions)
(require 'cl-lib)

(defun joaot/eglot-project-find-function (dir)
  (when eglot-lsp-context
    (let* ((marker nil)
           ;; We use "dominating files" here, but we could use some
           ;; other criteria, like an Elisp data structure, or hack in
           ;; dir-locals for DIR (maybe with
           ;; `hack-dir-local-variables-non-file-buffer').
           (root (locate-dominating-file
                  dir
                  (lambda (d)
                    ;; Here, we search for these markers regardless of
                    ;; other state such current major mode.  We could
                    ;; fix that easily, but that's out of scope for this small example.
                    (cl-loop for f in '("pyrightconfig.json" "pyproject.toml")
                             when (file-exists-p (expand-file-name f d))
			     do (setq marker f)
                             and return t)))))
      (when root
        ;; Use the experimental eglot--project project type in latest
        ;; Eglot which has some space for user-defined props,
        ;; `:marker' in this case.  One could make use of it, say, in a
        ;; functional entry to `eglot-server-programs', to decide
        ;; which server to run based on the marker or the type of
        ;; the project.
        `(eglot--project ,root :marker ,marker)))))

(add-to-list 'project-find-functions
             #'joaot/eglot-project-find-function)

It's just a small iteration on what was already said above, just corrected to use eglot-lsp-context, use dir instead of the default-directory , and simplify the throw/catch (shouldn't really be needed if locate-dominating-file is doing it's job correctly).

[cowboyd]

Fwiw, I did a similar thing that used tsp-context and in addition to fixing the project context problems, it also solved the performance issues. I thought I had reported back on this, but apparently not.

I'll see if I can adopt what you have here.