projectile-mode makes editior extremally laggy and slow

[jsobiecki]

I'm using projectile for a while, and I'm extremaly happy with that.

But recently, after package update something went wrong. When projectile-mode is enabled, all my actions in editor (like typing, and navigating in file) started to be very slow, practically making normal work impossible.

When I disable projectile-mode, then all issues magically disappear and I can continue my work. I compared two projects (both .git based).

1. My config files (several lisp files) - edition of all files is slow as hell
2. Drupal 8 projects (massive amount of php files) - the same.

I believe, that this wasn't the case in a past, but when I recently upgraded to newest (20171009.84) version, it started to crawl.

[Silex]

Try cleaning up the cache (C-p i)

[rreckel]

I have got the exact same problem.
M-x projectile-invalidate-cache
didn't help. Sorry

I try to find a workaround. If you have one, plz post.

[rreckel]

Here is the profile log:

- command-execute                                               35500  65%
  - call-interactively                                           35500  65%
   - byte-code                                                   30924  56%
    - helm-M-x-read-extended-command                             30924  56%
     - helm-comp-read                                            30924  56%
      - helm                                                     30923  56%
       - apply                                                   30923  56%
        - helm                                                   30923  56%
         - apply                                                 30923  56%
          - helm-internal                                        30923  56%
           - helm-read-pattern-maybe                             28503  52%
            - read-from-minibuffer                               28490  52%
             - redisplay_internal (C function)                   28422  52%
              - eval                                             28422  52%
               - format                                          28422  52%
                - projectile-project-type                        28422  52%
                 - projectile-detect-project-type                28422  52%
                  - cl-find-if                                   28422  52%
                   - apply                                       28422  52%
                    - cl-find                                    28422  52%
                     - apply                                     28422  52%
                      - cl-position                              28422  52%
                       - cl--position                            28422  52%
                        - #<compiled 0x40411bc1>                 28422  52%
                         - projectile-go-project-p               28366  52%
                          - projectile-current-pr...              28332  52%
                           - cl-mapcan                           28332  52%
                            - apply                              28332  52%
                             - cl-mapcar                         28332  52%
                              - mapcar                           28332  52%
                               - projectile-dir-f...              28332  52%
                                - projectile-dir-...              20937  38%
                                 - mapcar                        18370  33%
                                  - #<compiled 0x...              18370  33%
                                   - file-relativ...              11729  21%
                                      file-remote-p                 56   0%
                                      #<compiled ...                 23   0%
                                      string-pref...                  3   0%
                                 - projectile-get...               2275   4%
                                  - projectile-fi...               1768   3%
                                   - shell-comman...                878   1%
                                    - process-file                 873   1%
                                     - apply                       873   1%
                                      - call-proc...                 15   0%
                                         beacon--...                  6   0%
                                    - #<compiled ...                  3   0%
                                       kill-buffer                   3   0%
                                     split-string                  747   1%
                                  - projectile-ge...                499   0%
                                   - projectile-g...                499   0%
                                    - projectile-...                499   0%
                                     - projectile...                453   0%
                                      - shell-com...                434   0%
                                       - process-...                274   0%
                                        - apply                    269   0%
                                         - call-p...                  7   0%
                                          - ucs-n...                  7   0%
                                             gene...                  7   0%
                                       - #<compil...                154   0%
                                        - kill-bu...                150   0%
                                         - replac...                  6   0%
                                            unrec...                  4   0%
                                           helm-k...                  2   0%
                                        split-str...                  1   0%
                                     - projectile...                 40   0%
                                      - projectil...                 40   0%
                                       - projecti...                 34   0%
                                        - file-ex...                  5   0%
                                         - ucs-no...                  4   0%
                                          - inser...                  2   0%
                                             beac...                  2   0%
                                            ucs-n...                  1   0%
                                            gener...                  1   0%
                                     - cl-remove-...                  1   0%
                                        apply                        1   0%
                                  - projectile-ge...                  2   0%
                                   - projectile-p...                  2   0%
                                      projectile-...                  2   0%
                                  - projectile-pr...                  1   0%
                                     projectile-f...                  1   0%
                                - projectile-adju...               7388  13%
                                 - projectile-rem...               7080  13%
                                  - cl-remove-if                  6832  12%
                                   - apply                        6832  12%
                                    - cl-remove                   6824  12%
                                     - #<compiled...               6813  12%
                                      - cl-some                    828   1%
                                         #<compil...                700   1%
                                       - #<compil...                 94   0%
                                          string-...                 15   0%

[mriehl]

You should try this workaround: brocode/emacs_subconfigs@831d654

As I understand it a new feature is that projectile tries to determine the project type by going through heuristics. For golang projects the heuristic is "go through all files in the project and see if it ends with .go". Honestly I think that is a pretty terrible idea because going through all files in the project is always a bad idea (esp. if it's just to find out if the project is golang or not!).
The workaround I linked removes the project type from the modeline so it's not computed anymore.

There was #1182 open but it was fixed in the sense that projectile does not try to determine the project type when there is no project (it would go through all files in your $HOME then). But if your project is huge you will feel the pain.

(note: the format string in that workaround commit is broken, you want Projectile[%s] instead)

[bbatsov]

Hmm, I guess I underestimated this - I'll try to take a look at it soon, but in the meantime everyone can just set the modeline to nil or something like old value:

(setq projectile-mode-line
         '(:eval (format " Projectile[%s]"
                        (projectile-project-name))))

[bbatsov]

Looking at the profiling data it seems something is wrong with the project type caching, otherwise I can't imagine what would be causing this slowdown.

(defun projectile-detect-project-type ()
  "Detect the type of the current project."
  (let ((project-type (cl-find-if
                       (lambda (project-type)
                         (let ((marker (plist-get (gethash project-type projectile-project-types) 'marker-files)))
                           (if (listp marker)
                               (and (projectile-verify-files marker) project-type)
                             (and (funcall marker) project-type))))
                       (projectile-hash-keys projectile-project-types))))
    (when project-type
      (puthash (projectile-project-root) project-type projectile-project-type-cache))
    project-type))

(defun projectile-project-type ()
  "Determine the project's type based on its structure."
  (if projectile-project-type
      projectile-project-type
    (let ((project-root (ignore-errors (projectile-project-root))))
      (if project-root
          (or (gethash project-root projectile-project-type-cache)
              (projectile-detect-project-type)
              'generic)
        'generic))))

That's all the relevant code and it's pretty simple. At least right away nothing seems wrong with respect to caching here.

[bbatsov]

For what is worth in my tests I don't see any lag.

[jsobiecki]

@bbatsov @mriehl - Workaround works for me, thanks!

[rreckel]

Thx a lot.
The workaround worked for me too!

So, if I understand correctly: Every time the cursor moves, the mode line is updated, and so every time projectile tries to determine the project type, which is very slow (to determine go projects).

If there is a problem with my projectile cache, how can I remove/invalidate it?
As I said M-x projectile-invalidate-cache just tells me that it removed 0 items from the recentf list, and does not help in any way.

Here is a second part of the profile, that could confirm the slowness in determining go projects:

- ...                                                            9193  36%
    Automatic GC                                                  8877  35%
  - line-move-visual                                               316   1%
   - posn-at-point                                                 316   1%
    - eval                                                         316   1%
     - format                                                      316   1%
      - projectile-project-type                                    316   1%
       - projectile-detect-project-type                            316   1%
        - cl-find-if                                               316   1%
         - apply                                                   316   1%
          - cl-find                                                316   1%
           - apply                                                 316   1%
            - cl-position                                          316   1%
             - cl--position                                        316   1%
              - #<compiled 0x416538b3>                             316   1%
               - projectile-go-project-p                           316   1%
                - cl-some                                          316   1%
                 - #<compiled 0x4165386d>                          316   1%
                  - file-name-extension                            315   1%
                     file-name-sans-versions                        30   0%
                     #<compiled 0x4152e21f>                          2   0%

[vkazanov]

Same issue here, and the workaround helps.

[mriehl]

@bbatsov I think it's just the golang detection that is super inefficient.

  (cl-some
   (lambda (file)
     (string= (file-name-extension file) "go"))
(projectile-current-project-files)))

On linux (using strace -yy -f for syscall tracing) this translates in a crapton of fstat, fcntl, getdents, openat and close syscalls for each folder in the project. If you have a .npm or lots of directories this is super super slow.

In the elisp profiles projectile-go-project-p 28366 52% seems to be a dead giveaway.

I'm not sure why file-name-extension translates into so many syscalls. But a workaround could be to do something that isn't O(n) with the number of files/directories in the project for golang detection.
Like the rustlang detection that is more or less O(1) since it looks for Cargo.toml.
For instance we could be using $GOPATH and just see if the project cwd is in that path. This would result in false negatives ($GOPATH not set but go files in the project) but at the same time since golang is super opinionated you can't go build or go test if you're not in the $GOPATH sooooo...

The caching seems to be working for me (if I let the initial project type detection complete, which takes upwards of 5 minutes (!) then it's using the cached type). Also the -nw emacs seems to be doing this in the background (?) so it can stack a lot of projectile-go-project calls before it actually stores a cached type.

[maginemu]

Same issue and I reached same workaround.
I'm using projectile on C# project, so I think it's not just golang detection problem.

 - command-execute                                                2119  74%
  - call-interactively                                            2119  74%
   - funcall-interactively                                        1973  69%
    - next-line                                                   1969  69%
     - line-move                                                  1969  69%
      - line-move-partial                                         1969  69%
       - pos-visible-in-window-p                                  1243  43%
        - eval                                                    1243  43%
         - format                                                 1241  43%
          - projectile-project-type                               1241  43%
           - projectile-detect-project-type                       1241  43%
            - cl-find-if                                          1241  43%
             - apply                                              1241  43%
              - cl-find                                           1241  43%
               - apply                                            1241  43%
                - cl-position                                     1241  43%
                 - cl--position                                   1241  43%
                  - #<compiled 0x41474245>                        1241  43%
                   - projectile-verify-files                      1027  36%
                    - cl-every                                    1027  36%
                     - projectile-verify-file                     1027  36%
                        file-expand-wildcards                     1027  36%
                   + projectile-go-project-p                       180   6%
                   + projectile-cabal-project-p                     34   1%

[bbatsov]

@mriehl Yeah, that's very inefficient indeed, but on the other hand once the type is detected it is cached. However, on a bigger project the initial delay might be quite significant. If someone has a better way to detect go projects - I'll all ears. :-)

[bbatsov]

I've tested this again locally - for I see that project types are properly cached and retried. Some of you should instrument projectile-project-type with edebug to see why the caching is not working for you.

[rreckel]

While edebugging, I noticed that my directory .emac.d gets detected as 'generic.
As it seems, this means that projectile-detect-project-type doesn't put the project-root into projectile-project-type-cache. (I checked and projectile-project-type-cache was empty.

This can perhaps be the cause why editing my init.el is so extremely slow.....

Hope this helps a bit.
Let me know if you need more info.

[pcmantz]

Also had this issue and disabled the modeline component. Emacs was nearly unusable.

Does this callback need to happen every time the cursor moves? I would imagine that if it fired on switching buffers that would be sufficient.

[bbatsov]

@rreckel Ah, that's probably it! I'm assuming a lot of people had issues outside projects.

@pcmantz Yeah, that's how it works by default - I wish there were ways to tune this behaviour. It caused so much pain, that I'm seriously considering making this static just on find-file, but that has downsides as well (e.g. when you're doing things like switch-project the modeline won't update).

[blallau]

affected too :( , i'll try the workaround, regards

[idfellows]

I'm also affected on a small-ish php project (<5000 loc). This was causing 7-10s of lag between cursor moves, and persisted between Emacs restarts. The workaround works for me though!

[blallau]

Workaround works for me, thanks

[nickenchev]

I'm experiencing the same high latency, but only noticed it when editing an elisp file. My C++ projects weren't affected, or were affected much less (that I didn't notice).

The workaround @bbatsov provided seems to have fixed the issue.

Thanks

[pronobis]

I'm affected as well. Mostly the problem appears when editing elisp files.

[john-carter]

Let me make a wild arsed guess here....

Those of us who are experiencing this have something mounted that imposes a delay when queried.

Usual culprits are...

• network drives like cifs/samba

• or some sort of hardrive that is powered down if unused for awhile and takes awhile to spin up.

My guess is @bbatsov is not experiencing this because none of the directories between his current work directory and / have one of these mounts.

A simple fix would be stop looking for vc directories if you hit /home/* or /

Alas, I suspect this won't be bullet proof, so I suspect it would be better to turn off this feature by default.

[pronobis]

That could be the case indeed. However, the lagging never stops and makes working with emacs pretty much impossible. As for making the feature optional, I totally agree. I don't even see the information displayed for projectile in the modeline until I expand the list (which I never do). I value performance and responsiveness much more.

[carlosgeos]

Same as @nickenchev, C++ projects are OK. Python, elisp, etc are not.

[linnik]

I've met this issue while trying to edit elisp files for Prelude (~/.emacs.d/personal/). All of my python projects are fine. Workaround works for me too.

[D4N]

The workaround also works for me. Before it local (on a SSD) and remote Python projects were completely unusable. However I should add that the machine I tested it on is very old (2008-ish).

[codesuki]

Just experienced this after updating. Couldn't even scroll properly. In any buffer (yaml, list-packages, ...) where projectile was enabled. Removing the type from the mode-line worked.

[MiloDavis]

I got this issue with org-mode files. The workaround has fixed the issue for me.

[ghost]

Is a fix being worked on for this? Perhaps the new feature should be customizable so it can be more easily disabled?

[ysmolski]

It's rather a problem, though workaround has worked. I wish that got a permanent fix. Thanks!

[zingbretsenucb]

Just adding another voice in here--I stopped using emacs for a few days because it became basically unusable. bbatsov's mode line workaround has resolved the issue for me.

[valignatev]

I'll drop my word here also:
It works well on easily distinguishable project types like python. But it lags heavily on my home folder which is under git too. I guess it's quite typical case if you check the comments mentioning this issue :)

Modeline workaround works great though, thanks a lot @bbatsov

[bbatsov]

Yeah, the problem is the caching of the generic project type. I'm sorry for the pain this small change caused to so many of you. I'll try to find some time to look if I can fix this or I'll simply revert it.

On Mon, Oct 23, 2017 at 06:16 Valentin Ignatyev ***@***.***> wrote: I'll drop my word here also: It works well on easily distinguishable project types like python. But it lags heavily on my home folder which is under git too. I guess it's quite typical case if you check the comments mentioning this issue :) Modeline workaround works great though, thanks a lot @bbatsov <https://github.com/bbatsov> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1183 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGVykm0xQcfipjdqKNk_Znb1zPdtR5Rks5svBMegaJpZM4Pzlkr> .

-- Best Regards, Bozhidar Batsov http://www.batsov.com

[mgrbyte]

Just to note I'm also affected (editing anything it seems with projectile-mode enabled).

[dato]

Here’s an analysis that you might find useful. (And thank you for providing a workaround in the meantime.)

There are three elements that, combined, I think are enough to explain everybody’s observed behavior. I list them below. First, for this watching along, this is how project detection works:

1. To determine a project type the first time, projectile-detect-project-type just cycles through all known project types, until one reports success.

2. Verifying one type is typically very fast: just checking if a file, e.g. requirements.txt exists in the root directory (or sometimes a small number of them, e.g. Gemfile plus spec).

3. Once the correct project type is found, the result is cached in a local variable.

And here are the three elements that affect each of the above, making it slow in different scenarios:

1. By cycling through all known types, how many “attempts” are done varies depending on the order of the list of known types. Consider:

   Known types: Python, C++, Go, Java, Ruby.

2. Verifying whether a given project is Go is slow: it potentially visits the whole directory tree. It is very slow, in particular, for non-Go projects!

   Compare, using the list above:

   • Huge C++ project: fast to detect

   • Tiny Ruby project: fast to detect! Since it's tiny, the overhead of checking for Go is negligible.

   • Huge Java project: slow to detect the first time, because it recurses the tree in search for a single Go file. Faster afterwards, though: once the correct type was detected, it gets cached.

3. Negative results (i.e. when no type was assigned) are not cached. This explains why “generic” projects are permanently slow: it never hits the cache.

Some possible ameliorations to consider:

• pushing function-based project detection to the end of the list
• cache negative results (with care)
• use "*.go" as a wildcard, and assume there'll always be at least one Go file at top level

Long-term advice to make it faster.

• do not use file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.

• instead, modify projectile-detect-project-type to perform only one call to obtain the list of top-level file names and then, for each attempted project type:

  • use set intersection if the marker files contain no wildcards
  • use fnmatch if it does
  • pass on that list of top-level file names to types that use a function to detect themselves

I hope this helps.

Thank you for your work on Projectile, I use it everyday and it improved my use of Emacs a lot.

[juergenhoetzel]

I use this hacky workaround to cache the lighter for each filename:

(defvar projectile-lighter-cache (make-hash-table :test 'equal))

(setq projectile-mode-line
      '(:eval
	(or (gethash (buffer-file-name) projectile-lighter-cache)
	    (puthash (buffer-file-name)
		     (format " Projectile[%s(%s)]"
			     (projectile-project-name)
			     (projectile-project-type))
		     projectile-lighter-cache))))

[juergenhoetzel]

do not use file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.

Indeed file-exists-p is much faster:

ELISP> (benchmark 100 '(file-exists-p "test.el"))
"Elapsed time: 0.002275s"
ELISP> (benchmark 100 '(file-expand-wildcards "test.el"))
"Elapsed time: 0.327794s (0.101474s in 1 GCs)"

I wonder why file-expand-wildcards is used in the first place. The use of wildcards isn't documented in
projectile-register-project-type and wildcards are not used in the implemented project types. So I just redefined this function:

(defun projectile-verify-file (file)
  "Check whether FILE exists in the current project."
  (file-exists-p (projectile-expand-root file)))

which resulted in an remarkable speedup.

[bbatsov]

I've fixed the fixing for 'generic project types, so this should work ok within projects, although to fix it better it needs to use buffer-local caching for the project type. (similar to what we have for projectile-cached-project-name) As I don't have time for this I'll just remove the project type from the the default modeline. If someone liked it - they can just add it back themselves.

[bbatsov]

@dato Good points and very valid. I did a couple of tweaks to the project detection now. There are so many easy wins in Projectile here and there, unfortunately I rarely find time for it these days.