Forbidden reentrant call of Tramp

[xor-xor]

Steps to reproduce:

1. Log into some remote Linux machine with clangd installed (but this may be server-agnostic).
2. Create a simple foobar "project" with the snippet below:

mkdir -p foobar/foo foobar/bar
echo "int main(){}" > foobar/foo/foo.cc
echo "int main(){}" > foobar/bar/bar.cc
git init foobar

3. From within Emacs running on your local machine, C-x C-f to foo.cc via TRAMP, e.g. /ssh:my-remote-machine:foobar/foo/foo.cc.
4. Wait until [eglot] Connected! Server `clangd' now managing `c++-mode' buffers in project `foobar'.
5. C-x C-f to bar.cc.
6. Observe the error Forbidden reentrant call of Tramp.

BTW, this doesn't happen when files foo.cc and bar.cc are in the same dir, e.g. foobar/foo.cc and foobar/bar.cc.

Details:

• Server used: clangd
• Emacs version: 27.1 (with TRAMP 2.5.2.2 installed from ELPA via package.el)
• Operating system: macOS (locally), Linux (remotely)
• Eglot version: current master @ 8dc5180
• Eglot installation method: git
• Using Doom: No

LSP transcript - M-x eglot-events-buffer (mandatory unless Emacs inoperable)

[internal] Thu Mar  3 19:34:41 2022:
(:message "Running language server: (sh -c stty raw > /dev/null; clangd --clang-tidy\\=0)")
[stderr] I[18:34:41.402] Ubuntu clangd version 13.0.0-2
[stderr] I[18:34:41.403] Features: linux
[stderr] I[18:34:41.403] PID: 2073
[stderr] I[18:34:41.403] Working directory: /home/xor-xor/foobar
[stderr] I[18:34:41.403] argv[0]: clangd
[stderr] I[18:34:41.403] argv[1]: --clang-tidy=0
[stderr] I[18:34:41.403] Starting LSP over stdin/stdout
[client-request] (id:1) Thu Mar  3 19:34:41 2022:
(:jsonrpc "2.0" :id 1 :method "initialize" :params
	  (:processId nil :rootPath "/home/xor-xor/foobar/" :rootUri "file:///home/xor-xor/foobar" :initializationOptions #s(hash-table size 65 test eql rehash-size 1.5 rehash-threshold 0.8125 data
																			      ())
		      :capabilities
		      (:workspace
		       (:applyEdit t :executeCommand
				   (:dynamicRegistration :json-false)
				   :workspaceEdit
				   (:documentChanges :json-false)
				   :didChangeWatchedFiles
				   (:dynamicRegistration t)
				   :symbol
				   (:dynamicRegistration :json-false)
				   :configuration t)
		       :textDocument
		       (:synchronization
			(:dynamicRegistration :json-false :willSave t :willSaveWaitUntil t :didSave t)
			:completion
			(:dynamicRegistration :json-false :completionItem
					      (:snippetSupport :json-false :deprecatedSupport t :tagSupport
							       (:valueSet
								[1]))
					      :contextSupport t)
			:hover
			(:dynamicRegistration :json-false :contentFormat
					      ["markdown" "plaintext"])
			:signatureHelp
			(:dynamicRegistration :json-false :signatureInformation
					      (:parameterInformation
					       (:labelOffsetSupport t)
					       :activeParameterSupport t))
			:references
			(:dynamicRegistration :json-false)
			:definition
			(:dynamicRegistration :json-false :linkSupport t)
			:declaration
			(:dynamicRegistration :json-false :linkSupport t)
			:implementation
			(:dynamicRegistration :json-false :linkSupport t)
			:typeDefinition
			(:dynamicRegistration :json-false :linkSupport t)
			:documentSymbol
			(:dynamicRegistration :json-false :hierarchicalDocumentSymbolSupport t :symbolKind
					      (:valueSet
					       [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26]))
			:documentHighlight
			(:dynamicRegistration :json-false)
			:codeAction
			(:dynamicRegistration :json-false :codeActionLiteralSupport
					      (:codeActionKind
					       (:valueSet
						["quickfix" "refactor" "refactor.extract" "refactor.inline" "refactor.rewrite" "source" "source.organizeImports"]))
					      :isPreferredSupport t)
			:formatting
			(:dynamicRegistration :json-false)
			:rangeFormatting
			(:dynamicRegistration :json-false)
			:rename
			(:dynamicRegistration :json-false)
			:publishDiagnostics
			(:relatedInformation :json-false :codeDescriptionSupport :json-false :tagSupport
					     (:valueSet
					      [1 2])))
		       :experimental #s(hash-table size 65 test eql rehash-size 1.5 rehash-threshold 0.8125 data
						   ()))))
[stderr] I[18:34:41.448] <-- initialize(1)
[stderr] I[18:34:41.449] --> reply:initialize(1) 0 ms
[server-reply] (id:1) Thu Mar  3 19:34:41 2022:
(:id 1 :jsonrpc "2.0" :result
     (:capabilities
      (:astProvider t :callHierarchyProvider t :codeActionProvider
		    (:codeActionKinds
		     ["quickfix" "refactor" "info"])
		    :compilationDatabase
		    (:automaticReload t)
		    :completionProvider
		    (:allCommitCharacters
		     [" " "	" "(" ")" "[" "]" "{" "}" "<" ">" ":" ";" "," "+" "-" "/" "*" "%" "^" "&" "#" "?" "." "=" "\"" "'" "|"]
		     :resolveProvider :json-false :triggerCharacters
		     ["." "<" ">" ":" "\"" "/"])
		    :declarationProvider t :definitionProvider t :documentFormattingProvider t :documentHighlightProvider t :documentLinkProvider
		    (:resolveProvider :json-false)
		    :documentOnTypeFormattingProvider
		    (:firstTriggerCharacter "\n" :moreTriggerCharacter
					    [])
		    :documentRangeFormattingProvider t :documentSymbolProvider t :executeCommandProvider
		    (:commands
		     ["clangd.applyFix" "clangd.applyTweak"])
		    :hoverProvider t :implementationProvider t :memoryUsageProvider t :referencesProvider t :renameProvider t :selectionRangeProvider t :semanticTokensProvider
		    (:full
		     (:delta t)
		     :legend
		     (:tokenModifiers
		      ["declaration" "deprecated" "deduced" "readonly" "static" "abstract" "dependentName" "defaultLibrary" "functionScope" "classScope" "fileScope" "globalScope"]
		      :tokenTypes
		      ["variable" "variable" "parameter" "function" "method" "function" "property" "variable" "class" "interface" "enum" "enumMember" "type" "type" "unknown" "namespace" "typeParameter" "concept" "type" "macro" "comment"])
		     :range :json-false)
		    :signatureHelpProvider
		    (:triggerCharacters
		     ["(" ","])
		    :textDocumentSync
		    (:change 2 :openClose t :save t)
		    :typeHierarchyProvider t :workspaceSymbolProvider t)
      :serverInfo
      (:name "clangd" :version "Ubuntu clangd version 13.0.0-2 linux x86_64-pc-linux-gnu")))
[client-notification] Thu Mar  3 19:34:41 2022:
(:jsonrpc "2.0" :method "initialized" :params #s(hash-table size 65 test eql rehash-size 1.5 rehash-threshold 0.8125 data
							    ()))
[client-notification] Thu Mar  3 19:34:41 2022:
(:jsonrpc "2.0" :method "textDocument/didOpen" :params
	  (:textDocument
	   (:uri "file:///home/xor-xor/foobar/foo/foo.cc" :version 0 :languageId "c++" :text "int main(){}\n")))
[client-notification] Thu Mar  3 19:34:41 2022:
(:jsonrpc "2.0" :method "workspace/didChangeConfiguration" :params
	  (:settings nil))
[stderr] I[18:34:41.466] <-- initialized
[stderr] I[18:34:41.658] <-- textDocument/didOpen
[stderr] I[18:34:41.659] Failed to find compilation database for /home/xor-xor/foobar/foo/foo.cc
[stderr] I[18:34:41.660] ASTWorker building file /home/xor-xor/foobar/foo/foo.cc version 0 with command clangd fallback
[stderr] [/home/xor-xor/foobar/foo]
[stderr] /usr/lib/llvm-13/bin/clang -resource-dir=/usr/lib/llvm-13/lib/clang/13.0.0 -- /home/xor-xor/foobar/foo/foo.cc
[stderr] I[18:34:41.661] <-- workspace/didChangeConfiguration
[stderr] I[18:34:41.674] --> textDocument/publishDiagnostics
[server-notification] Thu Mar  3 19:34:41 2022:
(:jsonrpc "2.0" :method "textDocument/publishDiagnostics" :params
	  (:diagnostics
	   []
	   :uri "file:///home/xor-xor/foobar/foo/foo.cc" :version 0))
[client-notification] Thu Mar  3 19:34:56 2022:
(:jsonrpc "2.0" :method "textDocument/didOpen" :params
	  (:textDocument
	   (:uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0 :languageId "c++" :text "int main(){}\n")))
[stderr] I[18:34:56.902] <-- textDocument/didOpen
[stderr] I[18:34:56.902] Failed to find compilation database for /home/xor-xor/foobar/bar/bar.cc
[stderr] I[18:34:56.902] ASTWorker building file /home/xor-xor/foobar/bar/bar.cc version 0 with command clangd fallback
[stderr] [/home/xor-xor/foobar/bar]
[stderr] /usr/lib/llvm-13/bin/clang -resource-dir=/usr/lib/llvm-13/lib/clang/13.0.0 -- /home/xor-xor/foobar/bar/bar.cc
[stderr] I[18:34:56.915] --> textDocument/publishDiagnostics
[server-notification] Thu Mar  3 19:34:57 2022:
(:jsonrpc "2.0" :method "textDocument/publishDiagnostics" :params
	  (:diagnostics
	   []
	   :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0))

Backtrace (mandatory, unless no error message seen or heard):

(hostname redacted out)

Debugger entered--Lisp error: (remote-file-error "Forbidden reentrant call of Tramp")
  signal(remote-file-error ("Forbidden reentrant call of Tramp"))
  tramp-error(#<process *tramp/ssh my-host.somewhere*> remote-file-error "Forbidden reentrant call of Tramp")
  tramp-send-string((tramp-file-name "ssh" nil nil "my-host.somewhere..." nil "/home/xor-xor/foobar/bar/bar.cc" nil) "\\readlink --canonicalize-missing /home/xor-xor...")
  tramp-send-command((tramp-file-name "ssh" nil nil "my-host.somewhere..." nil "/home/xor-xor/foobar/bar/bar.cc" nil) "\\readlink --canonicalize-missing /home/xor-xor...")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil "my-host.somewhere..." nil "/home/xor-xor/foobar/bar/bar.cc" nil) "\\readlink --canonicalize-missing /home/xor-xor...")
  tramp-sh-handle-file-truename("/ssh:my-host.somewhere...")
  apply(tramp-sh-handle-file-truename "/ssh:my-host.somewhere...")
  tramp-vc-file-name-handler(file-truename "/ssh:my-host.somewhere...")
  file-truename("/ssh:my-host.somewhere...")
  find-buffer-visiting("/ssh:my-host.somewhere...")
  (and t (find-buffer-visiting (eglot--uri-to-path uri)))
  (let* ((buffer (and t (find-buffer-visiting (eglot--uri-to-path uri))))) (if buffer (save-current-buffer (set-buffer buffer) (let* ((--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< (setq --cl-idx-- (1+ --cl-idx--)) (length --cl-vec--)) (setq diag-spec (aref --cl-vec-- --cl-idx--)) (setq diags (nconc diags (list ...)))) (cond (eglot--current-flymake-report-fn (eglot--report-to-flymake diags)) (t (setq eglot--unreported-diagnostics (cons t diags)))) nil)) (let* ((path (expand-file-name (eglot--uri-to-path uri))) (--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< (setq --cl-idx-- (1+ --cl-idx--)) (length --cl-vec--)) (setq diag-spec (aref --cl-vec-- --cl-idx--)) (setq diags (nconc diags (list (let ... ...))))) (setq flymake-list-only-diagnostics (assoc-delete-all path flymake-list-only-diagnostics #'string=)) (setq flymake-list-only-diagnostics (cons (cons path diags) flymake-list-only-diagnostics)) nil)))
  (progn (let* ((buffer (and t (find-buffer-visiting (eglot--uri-to-path uri))))) (if buffer (save-current-buffer (set-buffer buffer) (let* ((--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< (setq --cl-idx-- ...) (length --cl-vec--)) (setq diag-spec (aref --cl-vec-- --cl-idx--)) (setq diags (nconc diags ...))) (cond (eglot--current-flymake-report-fn (eglot--report-to-flymake diags)) (t (setq eglot--unreported-diagnostics ...))) nil)) (let* ((path (expand-file-name (eglot--uri-to-path uri))) (--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< (setq --cl-idx-- (1+ --cl-idx--)) (length --cl-vec--)) (setq diag-spec (aref --cl-vec-- --cl-idx--)) (setq diags (nconc diags (list ...)))) (setq flymake-list-only-diagnostics (assoc-delete-all path flymake-list-only-diagnostics #'string=)) (setq flymake-list-only-diagnostics (cons (cons path diags) flymake-list-only-diagnostics)) nil))))
  (let* ((--cl-eglot--diag-type-- #'(lambda (sev) (cond ((null sev) 'eglot-error) ((<= sev 1) 'eglot-error) ((= sev 2) 'eglot-warning) (t 'eglot-note))))) (progn (let* ((buffer (and t (find-buffer-visiting (eglot--uri-to-path uri))))) (if buffer (save-current-buffer (set-buffer buffer) (let* ((--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< ... ...) (setq diag-spec ...) (setq diags ...)) (cond (eglot--current-flymake-report-fn ...) (t ...)) nil)) (let* ((path (expand-file-name ...)) (--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< (setq --cl-idx-- ...) (length --cl-vec--)) (setq diag-spec (aref --cl-vec-- --cl-idx--)) (setq diags (nconc diags ...))) (setq flymake-list-only-diagnostics (assoc-delete-all path flymake-list-only-diagnostics #'string=)) (setq flymake-list-only-diagnostics (cons (cons path diags) flymake-list-only-diagnostics)) nil)))))
  (let* ((uri (car (cdr (plist-member --cl-rest-- ':uri)))) (diagnostics (car (cdr (plist-member --cl-rest-- ':diagnostics))))) (let* ((--cl-eglot--diag-type-- #'(lambda (sev) (cond (... ...) (... ...) (... ...) (t ...))))) (progn (let* ((buffer (and t (find-buffer-visiting ...)))) (if buffer (save-current-buffer (set-buffer buffer) (let* (... ... ... ...) (while ... ... ...) (cond ... ...) nil)) (let* ((path ...) (--cl-vec-- diagnostics) (--cl-idx-- -1) (diag-spec nil) (diags nil)) (while (< ... ...) (setq diag-spec ...) (setq diags ...)) (setq flymake-list-only-diagnostics (assoc-delete-all path flymake-list-only-diagnostics ...)) (setq flymake-list-only-diagnostics (cons ... flymake-list-only-diagnostics)) nil))))))
  (progn (let* ((uri (car (cdr (plist-member --cl-rest-- ':uri)))) (diagnostics (car (cdr (plist-member --cl-rest-- ':diagnostics))))) (let* ((--cl-eglot--diag-type-- #'(lambda (sev) (cond ... ... ... ...)))) (progn (let* ((buffer (and t ...))) (if buffer (save-current-buffer (set-buffer buffer) (let* ... ... ... nil)) (let* (... ... ... ... ...) (while ... ... ...) (setq flymake-list-only-diagnostics ...) (setq flymake-list-only-diagnostics ...) nil)))))))
  (closure (revert-buffer-preserve-modes eglot--managed-mode eglot-lsp-context company-tooltip-align-annotations company-backends markdown-fontify-code-blocks-natively t) (_server _method &rest --cl-rest--) "Handle notification publishDiagnostics.\n\n(fn SERVE..." (progn (let* ((uri (car (cdr ...))) (diagnostics (car (cdr ...)))) (let* ((--cl-eglot--diag-type-- #'...)) (progn (let* (...) (if buffer ... ...)))))))(#<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> textDocument/publishDiagnostics :diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0)
  apply((closure (revert-buffer-preserve-modes eglot--managed-mode eglot-lsp-context company-tooltip-align-annotations company-backends markdown-fontify-code-blocks-natively t) (_server _method &rest --cl-rest--) "Handle notification publishDiagnostics.\n\n(fn SERVE..." (progn (let* ((uri (car (cdr ...))) (diagnostics (car (cdr ...)))) (let* ((--cl-eglot--diag-type-- #'...)) (progn (let* (...) (if buffer ... ...))))))) #<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> textDocument/publishDiagnostics (:diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0))
  eglot-handle-notification(#<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> textDocument/publishDiagnostics :diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0)
  apply(eglot-handle-notification #<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> textDocument/publishDiagnostics (:diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0))
  (let ((eglot--cached-server server)) (apply fn server method (append params nil)))
  (closure ((fn . eglot-handle-notification) (initargs :process (closure ((contact "clangd" "--clang-tidy=0") (server-info "sh" "-c" "stty raw > /dev/null; clangd --clang-tidy\\=0") (autostart-inferior-process) (readable-name . "EGLOT (foobar/c++-mode)") (nickname . "foobar") (language-id . "c++") (contact "clangd" "--clang-tidy=0") (class . eglot-lsp-server) (project vc . "/ssh:my-host.somewhere...") (managed-major-mode . c++-mode) eglot--managed-mode eglot-lsp-context company-tooltip-align-annotations company-backends markdown-fontify-code-blocks-natively t) nil (let ((default-directory default-directory)) (make-process :name readable-name :command (setq server-info ...) :connection-type 'pipe :coding 'utf-8-emacs-unix :noquery t :stderr (get-buffer-create ...) :file-handler t)))) (contact "clangd" "--clang-tidy=0") (server-info "sh" "-c" "stty raw > /dev/null; clangd --clang-tidy\\=0") (autostart-inferior-process) (readable-name . "EGLOT (foobar/c++-mode)") (nickname . "foobar") (language-id . "c++") (contact "clangd" "--clang-tidy=0") (class . eglot-lsp-server) (project vc . "/ssh:my-host.somewhere...") (managed-major-mode . c++-mode) eglot--managed-mode eglot-lsp-context company-tooltip-align-annotations company-backends markdown-fontify-code-blocks-natively t) (server method params) (let ((eglot--cached-server server)) (apply fn server method (append params nil))))(#<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> textDocument/publishDiagnostics (:diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0))
  jsonrpc-connection-receive(#<eglot-lsp-server eglot-lsp-server-1fe17809a9a8> (:jsonrpc "2.0" :method "textDocument/publishDiagnostics" :params (:diagnostics [] :uri "file:///home/xor-xor/foobar/bar/bar.cc" :version 0)))
  jsonrpc--process-filter(#<process EGLOT (foobar/c++-mode)> "Content-Length: 158\15\n\15\n{\"jsonrpc\":\"2.0\",\"method\":\"...")
  process-send-string(#<process *tramp/ssh my-host.somewhere*> "tramp_vc_registered_read_file_names <<'cc634870bf3...")
  tramp-send-string((tramp-file-name "ssh" nil nil "my-host.somewhere..." nil "/home/xor-xor/foobar/bar/bar.cc" nil) "tramp_vc_registered_read_file_names <<'cc634870bf3...")
  tramp-send-command((tramp-file-name "ssh" nil nil "my-host.somewhere..." nil "/home/xor-xor/foobar/bar/bar.cc" nil) "tramp_vc_registered_read_file_names <<'cc634870bf3...")
  tramp-sh-handle-vc-registered("/ssh:my-host.somewhere...")
  apply(tramp-sh-handle-vc-registered "/ssh:my-host.somewhere...")
  tramp-sh-file-name-handler(vc-registered "/ssh:my-host.somewhere...")
  apply(tramp-sh-file-name-handler vc-registered "/ssh:my-host.somewhere...")
  tramp-file-name-handler(vc-registered "/ssh:my-host.somewhere...")
  vc-registered("/ssh:my-host.somewhere...")
  vc-backend("/ssh:my-host.somewhere...")
  vc-refresh-state()
  run-hooks(find-file-hook)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer bar.cc> "/ssh:my-host.somewhere..." nil nil "/ssh:my-host.somewhere..." (1299068 (-1 . 1)))
  find-file-noselect("/ssh:my-host.somewhere..." nil nil t)
  find-file("/ssh:my-host.somewhere..." t)
  funcall-interactively(find-file "/ssh:my-host.somewhere..." t)
  call-interactively(find-file nil nil)
  command-execute(find-file)

Minimal configuration (mandatory)

# Emacs started with:

$ emacs -Q -f package-initialize -L /path/to/git-cloned/eglot -l eglot.el 

;;; config evaluated via *scratch*: 
 
(setq debug-on-error t)

(with-eval-after-load 'cc-mode
  (require 'eglot)
  (add-to-list 'eglot-server-programs '((c++-mode) . ("clangd" "--clang-tidy=0")))
  (add-hook 'c++-mode-hook 'eglot-ensure))

[joaotavora]

Sorry for the long delay in looking at this. This need someone with TRAMP capabilities/expertise to debug.

[goranmoomin]

This bug seems to be locking up Emacs with a sufficiently big repository: the aforementioned "opening a new file in a different directory" deadlocks quite reliably on the vc operations to find the project root (with a pure Emacs 28.1 emacs -Q.)

I've found that project.el 0.5.4 (some random version) and tramp 2.4.4.4 (the version that was bundled with Emacs 27) doesn't deadlock that much (it still does randomly), but upgrading either of them to the latest version quite reliably deadlocks. I didn't try bisecting between the versions though. The version of Eglot didn't matter, so I think it's either that project.el or tramp has a bug or that the updates exposed a bug in Eglot.

I'm reliably able to deadlock on the same place in different repositories (e.g. a clone of the AFLplusplus repository, or an empty repository with just one commit that imports the whole Linux kernel sources) with a ssh tramp session connected to a local linux VM. I'd gladly help debugging this issue.

[jeslie0]

I’m not in a position to carry out an extensive test right now, but I found there to be next to zero errors of this form when using tramp and eglot through rsh. I can try some more extensive testing later.

[jeslie0]

I have found that most of the errors are related to ssh using ControlMaster. It is sufficient for me to set evaluate (setq tramp-use-ssh-controlmaster-options nil), or (setq tramp-controlmaster-options "-o ControlMaster=auto -o ControlPersiste=no") (removing "-o ControlPath=tramp.%%C") to massively reduce the number of Forbidden Reentrant errors.

This also explains why @goranmoomin didn't have tramp freezing up in Emacs 27 - one these variables was added in Emacs 28.1. Also, it explains why I had no issues using the RSH or RCP tramp methods.

Changing the variables above is a temporary fix. From my limited understanding of ControlMaster and SSH, I think the problem is that Eglot is using the same connection as other Emacs operations (for me, tramp would almost always hang when saving a buffer). This means that whenever Eglot is doing something, it will block other remote operations. A solution could be to force Eglot to open a new channel to talk to the remote language server (I think this is what Magit does).

[joaotavora]

As I don't use TRAMP, (and don't use ssh with "ControlMaster" -- at least I think so) I only got a general idea of what you meant in this post. But I thank you very much for the analysis.

A solution could be to force Eglot to open a new channel to talk to the remote language server (I think this is what Magit does).

Sounds ok, but how is that performed in practice?

[jeslie0]

A good question! I'm also new to the internals of Tramp, so I don't have an answer just yet. I will keep looking and either comment a solution here, or put in a PR.

My idea is to get Eglot to spawn a new SSH connection to the remote machine, where it sends and receives information from the language server. Using the ControlMaster setting (which is an SSH setting - Tramp defaults to using it in Emacs 28), causes all processes to be sent through the same SSH connection.

My Emacs-fu has space to improve. Do you know if it's possible to set a variable locally for Eglot? If so, the solution might be as simple as changing tramp-ssh-controlmaster-options locally in Eglot. I think we just need to change the ControlPath in the SSH connection that Eglot uses. That should make it spawn a new connection.

[joaotavora]

My Emacs-fu has space to improve. Do you know if it's possible to set a variable locally for Eglot? If so, the solution might be as simple as changing tramp-ssh-controlmaster-options locally in Eglot. I think we just need to change the ControlPath in the SSH connection that Eglot uses. That should make it spawn a new connection.

A local variable in Elisp isn't the best choice here, at least not if we can avoid it, but a dynamic binding may work yes.

We could bind that variable around the place where Eglot makes the process connection to the remote machine. Eglot doesn't invoke TRAMP directly, but it's started as a consequence of M-x eglot on remote paths.

Maybe you can play with this 100% untested patch:

diff --git a/eglot.el b/eglot.el
index 2e332c470f..860fcd061f 100644
--- a/eglot.el
+++ b/eglot.el
@@ -1108,6 +1108,8 @@ Each function is passed the server as an argument")
 (defvar-local eglot--cached-server nil
   "A cached reference to the current EGLOT server.")
 
+(defvar tramp-ssh-controlmaster-options) ;; forward declare
+
 (defun eglot--connect (managed-major-mode project class contact language-id)
   "Connect to MANAGED-MAJOR-MODE, LANGUAGE-ID, PROJECT, CLASS and CONTACT.
 This docstring appeases checkdoc, that's all."
@@ -1142,7 +1144,9 @@ This docstring appeases checkdoc, that's all."
                         (contact (cl-subseq contact 0 probe)))
                    `(:process
                      ,(lambda ()
-                        (let ((default-directory default-directory))
+                        (let ((default-directory default-directory)
+                              (tramp-ssh-controlmaster-options
+                               "-o ControlMaster=auto -o ControlPersist=no"))
                           (make-process
                            :name readable-name
                            :command (setq server-info (eglot--cmd contact))

[jeslie0]

Thank you! I will try and have a play today and report back with what I find.

[jeslie0]

This does cause to Eglot to use a new channel when connecting over SSH! On my tests, I had 0 locks with this patch. I think we also need to add in tramp-use-ssh-controlmaster-options t to ensure that Eglot uses this particular ControlMaster setting, rather than one from the users ssh config file.

There are a few small issues to be sorted out. First, the variable tramp-use-ssh-controlmaster-options was added in Emacs 28.1, so the solution will need to take into account the version of Emacs being used. Secondly, ControlMaster isn't in every SSH implementation (I don't think it exists on Windows). The solution will need to only set a ControlMaster setting when appropriate.

I will have a look at fixing these issues tonight. Thank you for your help!

[joaotavora]

The forward declaration and binding of a variable that is not in a specifc Emacs version is usually unproblematic, so don't worry about Emacs 27. The Windows problem might be real, but it should be up to tramp to decide whether to honor that variable or not.

[jeslie0]

Thankfully, the solution is to force SSH to not use ControlMaster, so it should still work for windows.

[jeslie0]

I made a PR with this fix. I changed it slightly - it now will always not use ControlMaster, even if the user specified it in their ssh config. There also shouldn't be a problem with Windows - it still accepts a ControlMaster option.