projectile-find-file hanging with TRAMP

[btoconnor]

Expected behavior

Running projectile-find-file runs over TRAMP successfully.

I am running on a fairly large project, and I am fairly certain I am doing something wrong. I've enabled file caching, turned off dynamic modeline. However, still whenever I run projectile-find-file, emacs hangs. I've got around 85k files in the repository. I've timed git ls-files -zco and on the command line that takes around 8s. However, running it in emacs with projectile, I've let this run for ~20ish minutes and nothing completes.

I couldn't figure out how to get a profile / trace going to showcase where the time is being spent when the projectile command is running. I got a backtrace, but it didn't appear helpful (to me) (remote host name and rootpath obscured below):

Debugger entered--Lisp error: (quit)
  signal(quit nil)
  tramp-file-name-handler(process-file "/bin/bash" nil t nil "-c" "git ls-files -zco --exclude-standard")
  apply(tramp-file-name-handler process-file "/bin/bash" nil t nil ("-c" "git ls-files -zco --exclude-standard"))
  process-file("/bin/bash" nil t nil "-c" "git ls-files -zco --exclude-standard")
  shell-command-to-string("git ls-files -zco --exclude-standard")
  projectile-files-via-ext-command("/ssh:host:rootpath" "git ls-files -zco --exclude-standard")
  projectile-dir-files-alien("/ssh:host:rootpath")
  projectile-project-files("/ssh:host:rootpath")
  projectile--find-file(nil)
  projectile-find-file(nil)
  funcall-interactively(projectile-find-file nil)
  call-interactively(projectile-find-file record nil)
  command-execute(projectile-find-file record)
  execute-extended-command(nil "projectile-find-file" "projectile-find-f")
  funcall-interactively(execute-extended-command nil "projectile-find-file" "projectile-find-f")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Output from time command:

real	0m8.016s
user	0m0.110s
sys	0m0.219s

This might not be projectile at all, so if someone could point me to how to generate a profile as seen here: #1232 (comment) I think I could start to narrow things down. I can also provide my configuration settings as well if that's helpful.

Actual behavior

When I run projectile-find-file over a TRAMP session, it hangs for minutes. I have not received a result yet, and the CPU goes to 100%

Steps to reproduce the problem

Running projectile-find-file on a remote project over ssh.
This is extremely important! Providing us with a reliable way to reproduce
a problem will expedite its solution.

Environment & Version information

Projectile version information

20191204.1451

Emacs version

26.3

Operating system

OSX

[btoconnor]

I was able to figure out (I think) how to produce that profiler report:

So it seems like tramp-check-for-regexp is the offending function here.

[btoconnor]

Also worth pointing out that I changed projectile-indexing-method to native. That made it so emacs was no longer hanging indefinitely at 100% cpu, and I saw a little spinner rotating around while it indexed the the files. Unfortunately, I let it run for around 15 minutes and it never completed.

I was able to get emacs up and running on the remote host to see how it operated there - projectile indexed the project in around a second.

[Silex]

Does it work on small repositories? Does TRAMP work normally if you disable projectile?

[btoconnor]

Apologies, should have included that in the initial report. TRAMP works normally even without projectile disabled - If I issue C-x C-f I can open and browse files normally and it's fine.

Additionally, it does work on smaller repositories. I have a project with around ~450 files in git, and projectile + TRAMP works fine on that project. Issuing projectile-find-file takes less than a second to return.

[Silex]

Ok, I just noticed this: I've got around 85k files in the repository

That means that shell-command-to-string("git ls-files -zco --exclude-standard") actually tries to return 85k strings concatened together, which is probably a huuuuge string.

If we look at your trace, we see this:

(defun tramp-wait-for-output (proc &optional timeout)
  "Wait for output from remote command."
  (unless (buffer-live-p (process-buffer proc))
    (delete-process proc)
    (tramp-error proc 'file-error "Process `%s' not available, try again" proc))
  (with-current-buffer (process-buffer proc)
    (let* (;; Initially, `tramp-end-of-output' is "#$ ".  There might
	   ;; be leading escape sequences, which must be ignored.
	   ;; Busyboxes built with the EDITING_ASK_TERMINAL config
	   ;; option send also escape sequences, which must be
	   ;; ignored.
	   (regexp (format "[^#$\n]*%s\\(%s\\)?\r?$"
			   (regexp-quote tramp-end-of-output)
			   tramp-device-escape-sequence-regexp))
	   ;; Sometimes, the commands do not return a newline but a
	   ;; null byte before the shell prompt, for example "git
	   ;; ls-files -c -z ...".
	   (regexp1 (format "\\(^\\|\000\\)%s" regexp))
	   (found (tramp-wait-for-regexp proc timeout regexp1)))
      .... snip ...

So, basically what happens is that tramp tries to find regexp1 over and over in the buffer while receiving this huge string. I'd probably be done different and wait until it read it all and then try to match the regex, but that'd be a bug report to the emacs mailing list. Maybe that was even fixed in the latest TRAMP version.

So, it looks like a TRAMP "bug", that said maybe projectile could use something else than shell-command-to-string to gather the file list, that does not look very efficient. This would allow to add a limit of the number of files returned, avoiding lists that are too bigs (e.g limit to max 1000 files), but that's to be implemented and I don't know how easy it'd be.

Does projectile even work locally on this repo? 85k files seems too much for any completion system...

[btoconnor]

Thanks for taking a look at this - it definitely works locally - it's not the snappiest thing, but I can run projectile-find-file and the list prompt appears almost instantly (I do have caching enabled at this point). When I type a few characters, I'll get a list of files that match within a second or two. I'm sure there's some improvements that can be had there as well, but it's not top of mind here at the moment :)

Not sure if it's relevant, but using ido in my setup.

[Silex]

Thanks for taking a look at this - it definitely works locally - it's not the snappiest thing, but I can run projectile-find-file and the list prompt appears almost instantly (I do have caching enabled at this point)

Yeah, it just occured to me that locally shell-command-to-string does not use TRAMP at all, so that's why.

Anyway, as I said the best you can do here is make a bug report to TRAMP/Emacs. We'll see what @bbatsov has to say but I doubt changing projectile to modify this shell-command-to-string as some loop-and-parse-strings-as-they-come-while-counting-and-aborting-the-loop-if-n-equals-1000 is an easy task (tho it might fix projectile for huuuge repositories in the process).

[Silex]

Oh, maybe you could try to install the latest TRAMP version to check if that was fixed before doing a bug report. If you don't plan on doing a bug report tell me I might do one myself.

[btoconnor]

Is there a guide I can read somewhere about updating to the latest TRAMP? Is it on MELPA? Not familiar with how to update a package that came bundled with Emacs itself.

I'm not opposed to opening up a bug report with TRAMP, but truthfully I have a very very very limited understanding of how any of this works (I just found out how to do the profiler for this in the comment above).

[Silex]

Oh well, no there's no guide. To test latest TRAMP you have to get the source from the savannah repositories, and to make the bug report well... let's just say it's complicated (Emacs dev still lives in the CSV era reguarding its workflow).

I'll try to make a bug report and report here.

[btoconnor]

That's very much appreciated, thank you! Let me know if there's anyway I can help out going forward.

[Silex]

@btoconnor: just to confirm that I understood the issue right, can you do this:

• naviguate to your repository over TRAMP, at the project's root directory.
• press M-: (or M-x eval-expression) and type this: (shell-command-to-string "git ls-files -zco --exclude-standard").
• confirm that it takes forever.

[Silex]

@btoconnor: also, is this a public repository? This could help reproduce.

If not maybe the linux kernel could be one to test with... tho I can't wrap my head around this 85K files, why so much? why not split it in smaller repositories?

[btoconnor]

Can confirm running (shell-command-to-string "git ls-files -zco --exclude-standard") over TRAMP takes forever and pegs the CPU at 100%.

The repo is very much not public. Unfortunately in this case, that's just the way our project works - it's a monolithic repo.

[Silex]

@btoconnor: good news I got an answer (you can follow at https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00377.html). Can you test with the patch at https://lists.gnu.org/archive/html/emacs-devel/2019-12/txtrcGfaTvbOT.txt and report?

[btoconnor]

Whoa thanks! That was way faster than I was expecting. I'm traveling today and will need to figure out how to do all this, but I should be able to find time in the next week. I'll post back here with my results, but the exchange on the list sounds very promising.

Thanks for moving this forward for me!

[Silex]

Btw, I didn't know but latest TRAMP is always available on ELPA, so it's easy to install the latest version.

[Silex]

Ok apparently based on tests from Michael the patch should really fix your problem, I think we can close this as it does not concern projectile anymore. But please still add a comment later about your tests so I can report to the ML that your problem was fixed.

[btoconnor]

Was just able to compile emacs (there was probably a faster way, but it worked for the job) from https://github.com/emacs-mirror/emacs which I noticed had the patch in it.

It worked! Projectile was able to fetch the files within about 10 seconds, and now the files appear to be cached.

Thanks for your help!

[Silex]

Great news! Thanks 👍