New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

projectile-mode makes editior extremally laggy and slow #1183

Closed
jsobiecki opened this Issue Oct 10, 2017 · 43 comments

Comments

Projects
None yet
@jsobiecki

jsobiecki commented Oct 10, 2017

I'm using projectile for a while, and I'm extremaly happy with that.

But recently, after package update something went wrong. When projectile-mode is enabled, all my actions in editor (like typing, and navigating in file) started to be very slow, practically making normal work impossible.

When I disable projectile-mode, then all issues magically disappear and I can continue my work. I compared two projects (both .git based).

  1. My config files (several lisp files) - edition of all files is slow as hell
  2. Drupal 8 projects (massive amount of php files) - the same.

I believe, that this wasn't the case in a past, but when I recently upgraded to newest (20171009.84) version, it started to crawl.

@Silex

This comment has been minimized.

Contributor

Silex commented Oct 10, 2017

Try cleaning up the cache (C-p i)

@rreckel

This comment has been minimized.

rreckel commented Oct 10, 2017

I have got the exact same problem.
M-x projectile-invalidate-cache
didn't help. Sorry

I try to find a workaround. If you have one, plz post.

@rreckel

This comment has been minimized.

rreckel commented Oct 10, 2017

Here is the profile log:

- command-execute                                               35500  65%
  - call-interactively                                           35500  65%
   - byte-code                                                   30924  56%
    - helm-M-x-read-extended-command                             30924  56%
     - helm-comp-read                                            30924  56%
      - helm                                                     30923  56%
       - apply                                                   30923  56%
        - helm                                                   30923  56%
         - apply                                                 30923  56%
          - helm-internal                                        30923  56%
           - helm-read-pattern-maybe                             28503  52%
            - read-from-minibuffer                               28490  52%
             - redisplay_internal (C function)                   28422  52%
              - eval                                             28422  52%
               - format                                          28422  52%
                - projectile-project-type                        28422  52%
                 - projectile-detect-project-type                28422  52%
                  - cl-find-if                                   28422  52%
                   - apply                                       28422  52%
                    - cl-find                                    28422  52%
                     - apply                                     28422  52%
                      - cl-position                              28422  52%
                       - cl--position                            28422  52%
                        - #<compiled 0x40411bc1>                 28422  52%
                         - projectile-go-project-p               28366  52%
                          - projectile-current-pr...              28332  52%
                           - cl-mapcan                           28332  52%
                            - apply                              28332  52%
                             - cl-mapcar                         28332  52%
                              - mapcar                           28332  52%
                               - projectile-dir-f...              28332  52%
                                - projectile-dir-...              20937  38%
                                 - mapcar                        18370  33%
                                  - #<compiled 0x...              18370  33%
                                   - file-relativ...              11729  21%
                                      file-remote-p                 56   0%
                                      #<compiled ...                 23   0%
                                      string-pref...                  3   0%
                                 - projectile-get...               2275   4%
                                  - projectile-fi...               1768   3%
                                   - shell-comman...                878   1%
                                    - process-file                 873   1%
                                     - apply                       873   1%
                                      - call-proc...                 15   0%
                                         beacon--...                  6   0%
                                    - #<compiled ...                  3   0%
                                       kill-buffer                   3   0%
                                     split-string                  747   1%
                                  - projectile-ge...                499   0%
                                   - projectile-g...                499   0%
                                    - projectile-...                499   0%
                                     - projectile...                453   0%
                                      - shell-com...                434   0%
                                       - process-...                274   0%
                                        - apply                    269   0%
                                         - call-p...                  7   0%
                                          - ucs-n...                  7   0%
                                             gene...                  7   0%
                                       - #<compil...                154   0%
                                        - kill-bu...                150   0%
                                         - replac...                  6   0%
                                            unrec...                  4   0%
                                           helm-k...                  2   0%
                                        split-str...                  1   0%
                                     - projectile...                 40   0%
                                      - projectil...                 40   0%
                                       - projecti...                 34   0%
                                        - file-ex...                  5   0%
                                         - ucs-no...                  4   0%
                                          - inser...                  2   0%
                                             beac...                  2   0%
                                            ucs-n...                  1   0%
                                            gener...                  1   0%
                                     - cl-remove-...                  1   0%
                                        apply                        1   0%
                                  - projectile-ge...                  2   0%
                                   - projectile-p...                  2   0%
                                      projectile-...                  2   0%
                                  - projectile-pr...                  1   0%
                                     projectile-f...                  1   0%
                                - projectile-adju...               7388  13%
                                 - projectile-rem...               7080  13%
                                  - cl-remove-if                  6832  12%
                                   - apply                        6832  12%
                                    - cl-remove                   6824  12%
                                     - #<compiled...               6813  12%
                                      - cl-some                    828   1%
                                         #<compil...                700   1%
                                       - #<compil...                 94   0%
                                          string-...                 15   0%
@mriehl

This comment has been minimized.

mriehl commented Oct 10, 2017

You should try this workaround: brocode/emacs_subconfigs@831d654

As I understand it a new feature is that projectile tries to determine the project type by going through heuristics. For golang projects the heuristic is "go through all files in the project and see if it ends with .go". Honestly I think that is a pretty terrible idea because going through all files in the project is always a bad idea (esp. if it's just to find out if the project is golang or not!).
The workaround I linked removes the project type from the modeline so it's not computed anymore.

There was #1182 open but it was fixed in the sense that projectile does not try to determine the project type when there is no project (it would go through all files in your $HOME then). But if your project is huge you will feel the pain.

(note: the format string in that workaround commit is broken, you want Projectile[%s] instead)

@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 10, 2017

Hmm, I guess I underestimated this - I'll try to take a look at it soon, but in the meantime everyone can just set the modeline to nil or something like old value:

(setq projectile-mode-line
         '(:eval (format " Projectile[%s]"
                        (projectile-project-name))))
@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 10, 2017

Looking at the profiling data it seems something is wrong with the project type caching, otherwise I can't imagine what would be causing this slowdown.

(defun projectile-detect-project-type ()
  "Detect the type of the current project."
  (let ((project-type (cl-find-if
                       (lambda (project-type)
                         (let ((marker (plist-get (gethash project-type projectile-project-types) 'marker-files)))
                           (if (listp marker)
                               (and (projectile-verify-files marker) project-type)
                             (and (funcall marker) project-type))))
                       (projectile-hash-keys projectile-project-types))))
    (when project-type
      (puthash (projectile-project-root) project-type projectile-project-type-cache))
    project-type))

(defun projectile-project-type ()
  "Determine the project's type based on its structure."
  (if projectile-project-type
      projectile-project-type
    (let ((project-root (ignore-errors (projectile-project-root))))
      (if project-root
          (or (gethash project-root projectile-project-type-cache)
              (projectile-detect-project-type)
              'generic)
        'generic))))

That's all the relevant code and it's pretty simple. At least right away nothing seems wrong with respect to caching here.

svend added a commit to svend/dot-emacsd that referenced this issue Oct 10, 2017

@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 10, 2017

For what is worth in my tests I don't see any lag.

@jsobiecki

This comment has been minimized.

jsobiecki commented Oct 10, 2017

@bbatsov @mriehl - Workaround works for me, thanks!

@rreckel

This comment has been minimized.

rreckel commented Oct 11, 2017

Thx a lot.
The workaround worked for me too!

So, if I understand correctly: Every time the cursor moves, the mode line is updated, and so every time projectile tries to determine the project type, which is very slow (to determine go projects).

If there is a problem with my projectile cache, how can I remove/invalidate it?
As I said M-x projectile-invalidate-cache just tells me that it removed 0 items from the recentf list, and does not help in any way.

Here is a second part of the profile, that could confirm the slowness in determining go projects:

- ...                                                            9193  36%
    Automatic GC                                                  8877  35%
  - line-move-visual                                               316   1%
   - posn-at-point                                                 316   1%
    - eval                                                         316   1%
     - format                                                      316   1%
      - projectile-project-type                                    316   1%
       - projectile-detect-project-type                            316   1%
        - cl-find-if                                               316   1%
         - apply                                                   316   1%
          - cl-find                                                316   1%
           - apply                                                 316   1%
            - cl-position                                          316   1%
             - cl--position                                        316   1%
              - #<compiled 0x416538b3>                             316   1%
               - projectile-go-project-p                           316   1%
                - cl-some                                          316   1%
                 - #<compiled 0x4165386d>                          316   1%
                  - file-name-extension                            315   1%
                     file-name-sans-versions                        30   0%
                     #<compiled 0x4152e21f>                          2   0%
@vkazanov

This comment has been minimized.

vkazanov commented Oct 11, 2017

Same issue here, and the workaround helps.

@mriehl

This comment has been minimized.

mriehl commented Oct 11, 2017

@bbatsov I think it's just the golang detection that is super inefficient.

  (cl-some
   (lambda (file)
     (string= (file-name-extension file) "go"))
(projectile-current-project-files)))

On linux (using strace -yy -f for syscall tracing) this translates in a crapton of fstat, fcntl, getdents, openat and close syscalls for each folder in the project. If you have a .npm or lots of directories this is super super slow.

In the elisp profiles projectile-go-project-p 28366 52% seems to be a dead giveaway.

I'm not sure why file-name-extension translates into so many syscalls. But a workaround could be to do something that isn't O(n) with the number of files/directories in the project for golang detection.
Like the rustlang detection that is more or less O(1) since it looks for Cargo.toml.
For instance we could be using $GOPATH and just see if the project cwd is in that path. This would result in false negatives ($GOPATH not set but go files in the project) but at the same time since golang is super opinionated you can't go build or go test if you're not in the $GOPATH sooooo...

The caching seems to be working for me (if I let the initial project type detection complete, which takes upwards of 5 minutes (!) then it's using the cached type). Also the -nw emacs seems to be doing this in the background (?) so it can stack a lot of projectile-go-project calls before it actually stores a cached type.

@maginemu

This comment has been minimized.

maginemu commented Oct 11, 2017

Same issue and I reached same workaround.
I'm using projectile on C# project, so I think it's not just golang detection problem.

 - command-execute                                                2119  74%
  - call-interactively                                            2119  74%
   - funcall-interactively                                        1973  69%
    - next-line                                                   1969  69%
     - line-move                                                  1969  69%
      - line-move-partial                                         1969  69%
       - pos-visible-in-window-p                                  1243  43%
        - eval                                                    1243  43%
         - format                                                 1241  43%
          - projectile-project-type                               1241  43%
           - projectile-detect-project-type                       1241  43%
            - cl-find-if                                          1241  43%
             - apply                                              1241  43%
              - cl-find                                           1241  43%
               - apply                                            1241  43%
                - cl-position                                     1241  43%
                 - cl--position                                   1241  43%
                  - #<compiled 0x41474245>                        1241  43%
                   - projectile-verify-files                      1027  36%
                    - cl-every                                    1027  36%
                     - projectile-verify-file                     1027  36%
                        file-expand-wildcards                     1027  36%
                   + projectile-go-project-p                       180   6%
                   + projectile-cabal-project-p                     34   1%
@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 12, 2017

@mriehl Yeah, that's very inefficient indeed, but on the other hand once the type is detected it is cached. However, on a bigger project the initial delay might be quite significant. If someone has a better way to detect go projects - I'll all ears. :-)

@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 12, 2017

I've tested this again locally - for I see that project types are properly cached and retried. Some of you should instrument projectile-project-type with edebug to see why the caching is not working for you.

@rreckel

This comment has been minimized.

rreckel commented Oct 12, 2017

While edebugging, I noticed that my directory .emac.d gets detected as 'generic.
As it seems, this means that projectile-detect-project-type doesn't put the project-root into projectile-project-type-cache. (I checked and projectile-project-type-cache was empty.

This can perhaps be the cause why editing my init.el is so extremely slow.....

Hope this helps a bit.
Let me know if you need more info.

@pcmantz

This comment has been minimized.

pcmantz commented Oct 12, 2017

Also had this issue and disabled the modeline component. Emacs was nearly unusable.

Does this callback need to happen every time the cursor moves? I would imagine that if it fired on switching buffers that would be sufficient.

@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 12, 2017

@rreckel Ah, that's probably it! I'm assuming a lot of people had issues outside projects.

@pcmantz Yeah, that's how it works by default - I wish there were ways to tune this behaviour. It caused so much pain, that I'm seriously considering making this static just on find-file, but that has downsides as well (e.g. when you're doing things like switch-project the modeline won't update).

@blallau

This comment has been minimized.

blallau commented Oct 13, 2017

affected too :( , i'll try the workaround, regards

@idfellows

This comment has been minimized.

idfellows commented Oct 13, 2017

I'm also affected on a small-ish php project (<5000 loc). This was causing 7-10s of lag between cursor moves, and persisted between Emacs restarts. The workaround works for me though!

@blallau

This comment has been minimized.

blallau commented Oct 13, 2017

Workaround works for me, thanks

@nickenchev

This comment has been minimized.

nickenchev commented Oct 16, 2017

I'm experiencing the same high latency, but only noticed it when editing an elisp file. My C++ projects weren't affected, or were affected much less (that I didn't notice).

The workaround @bbatsov provided seems to have fixed the issue.

Thanks

Superbil added a commit to Superbil/emacs.d that referenced this issue Oct 17, 2017

@pronobis

This comment has been minimized.

pronobis commented Oct 18, 2017

I'm affected as well. Mostly the problem appears when editing elisp files.

@john-carter

This comment has been minimized.

john-carter commented Oct 18, 2017

Let me make a wild arsed guess here....

Those of us who are experiencing this have something mounted that imposes a delay when queried.

Usual culprits are...

  • network drives like cifs/samba

  • or some sort of hardrive that is powered down if unused for awhile and takes awhile to spin up.

My guess is @bbatsov is not experiencing this because none of the directories between his current work directory and / have one of these mounts.

A simple fix would be stop looking for vc directories if you hit /home/* or /

Alas, I suspect this won't be bullet proof, so I suspect it would be better to turn off this feature by default.

@pronobis

This comment has been minimized.

pronobis commented Oct 18, 2017

That could be the case indeed. However, the lagging never stops and makes working with emacs pretty much impossible. As for making the feature optional, I totally agree. I don't even see the information displayed for projectile in the modeline until I expand the list (which I never do). I value performance and responsiveness much more.

@carlosgeos

This comment has been minimized.

carlosgeos commented Oct 18, 2017

Same as @nickenchev, C++ projects are OK. Python, elisp, etc are not.

@linnik

This comment has been minimized.

linnik commented Oct 18, 2017

I've met this issue while trying to edit elisp files for Prelude (~/.emacs.d/personal/). All of my python projects are fine. Workaround works for me too.

carlosgeos added a commit to carlosgeos/dot that referenced this issue Oct 18, 2017

workaround for slow emacs
see bbatsov/projectile#1183
if the system is mac, set clang as checker for flycheck
@D4N

This comment has been minimized.

D4N commented Oct 18, 2017

The workaround also works for me. Before it local (on a SSD) and remote Python projects were completely unusable. However I should add that the machine I tested it on is very old (2008-ish).

@thiagowfx

This comment has been minimized.

thiagowfx commented Oct 18, 2017

I am experiencing this issue as well. I realized it was super slow at the time I was editing my personal emacs configs. The workaround suggested by @bbatsov also worked for me.

@codesuki

This comment has been minimized.

codesuki commented Oct 19, 2017

Just experienced this after updating. Couldn't even scroll properly. In any buffer (yaml, list-packages, ...) where projectile was enabled. Removing the type from the mode-line worked.

@MiloDavis

This comment has been minimized.

MiloDavis commented Oct 19, 2017

I got this issue with org-mode files. The workaround has fixed the issue for me.

@wkmanire

This comment has been minimized.

wkmanire commented Oct 20, 2017

Is a fix being worked on for this? Perhaps the new feature should be customizable so it can be more easily disabled?

@ysmolsky

This comment has been minimized.

ysmolsky commented Oct 20, 2017

It's rather a problem, though workaround has worked. I wish that got a permanent fix. Thanks!

jackson15j pushed a commit to jackson15j/dot_emacs that referenced this issue Oct 20, 2017

@zingbretsenucb

This comment has been minimized.

zingbretsenucb commented Oct 20, 2017

Just adding another voice in here--I stopped using emacs for a few days because it became basically unusable. bbatsov's mode line workaround has resolved the issue for me.

flamingbear added a commit to flamingbear/emacs-config that referenced this issue Oct 21, 2017

wat-aro added a commit to wat-aro/dotfiles that referenced this issue Oct 22, 2017

Fix projectile setting
重くなったので設定を追加
cf: bbatsov/projectile#1183

PhilippeJara added a commit to PhilippeJara/main-emacs-config that referenced this issue Oct 22, 2017

valignatev added a commit to valignatev/dotfiles that referenced this issue Oct 23, 2017

@valignatev

This comment has been minimized.

valignatev commented Oct 23, 2017

I'll drop my word here also:
It works well on easily distinguishable project types like python. But it lags heavily on my home folder which is under git too. I guess it's quite typical case if you check the comments mentioning this issue :)

Modeline workaround works great though, thanks a lot @bbatsov

@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 23, 2017

sivakov512 added a commit to sivakov512/.emacs.d that referenced this issue Oct 23, 2017

@mgrbyte

This comment has been minimized.

mgrbyte commented Oct 24, 2017

Just to note I'm also affected (editing anything it seems with projectile-mode enabled).

dfrkp pushed a commit to dfrkp/emacs.d that referenced this issue Oct 26, 2017

andreyk0 pushed a commit to andreyk0/home_dot_files that referenced this issue Oct 26, 2017

fabacino added a commit to fabacino/emacs.d that referenced this issue Oct 27, 2017

@dato

This comment has been minimized.

dato commented Oct 28, 2017

Here’s an analysis that you might find useful. (And thank you for providing a workaround in the meantime.)

There are three elements that, combined, I think are enough to explain everybody’s observed behavior. I list them below. First, for this watching along, this is how project detection works:

  1. To determine a project type the first time, projectile-detect-project-type just cycles through all known project types, until one reports success.

  2. Verifying one type is typically very fast: just checking if a file, e.g. requirements.txt exists in the root directory (or sometimes a small number of them, e.g. Gemfile plus spec).

  3. Once the correct project type is found, the result is cached in a local variable.

And here are the three elements that affect each of the above, making it slow in different scenarios:

  1. By cycling through all known types, how many “attempts” are done varies depending on the order of the list of known types. Consider:

    Known types: Python, C++, Go, Java, Ruby.

  2. Verifying whether a given project is Go is slow: it potentially visits the whole directory tree. It is very slow, in particular, for non-Go projects!

    Compare, using the list above:

    • Huge C++ project: fast to detect

    • Tiny Ruby project: fast to detect! Since it's tiny, the overhead of checking for Go is negligible.

    • Huge Java project: slow to detect the first time, because it recurses the tree in search for a single Go file. Faster afterwards, though: once the correct type was detected, it gets cached.

  3. Negative results (i.e. when no type was assigned) are not cached. This explains why “generic” projects are permanently slow: it never hits the cache.

Some possible ameliorations to consider:

  • pushing function-based project detection to the end of the list
  • cache negative results (with care)
  • use "*.go" as a wildcard, and assume there'll always be at least one Go file at top level

Long-term advice to make it faster.

  • do not use file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.

  • instead, modify projectile-detect-project-type to perform only one call to obtain the list of top-level file names and then, for each attempted project type:

    • use set intersection if the marker files contain no wildcards
    • use fnmatch if it does
    • pass on that list of top-level file names to types that use a function to detect themselves

I hope this helps.

Thank you for your work on Projectile, I use it everyday and it improved my use of Emacs a lot.

@juergenhoetzel

This comment has been minimized.

Contributor

juergenhoetzel commented Oct 31, 2017

I use this hacky workaround to cache the lighter for each filename:

(defvar projectile-lighter-cache (make-hash-table :test 'equal))

(setq projectile-mode-line
      '(:eval
	(or (gethash (buffer-file-name) projectile-lighter-cache)
	    (puthash (buffer-file-name)
		     (format " Projectile[%s(%s)]"
			     (projectile-project-name)
			     (projectile-project-type))
		     projectile-lighter-cache))))
@juergenhoetzel

This comment has been minimized.

Contributor

juergenhoetzel commented Oct 31, 2017

do not use file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.

Indeed file-exists-p is much faster:

ELISP> (benchmark 100 '(file-exists-p "test.el"))
"Elapsed time: 0.002275s"
ELISP> (benchmark 100 '(file-expand-wildcards "test.el"))
"Elapsed time: 0.327794s (0.101474s in 1 GCs)"

I wonder why file-expand-wildcards is used in the first place. The use of wildcards isn't documented in
projectile-register-project-type and wildcards are not used in the implemented project types. So I just redefined this function:

(defun projectile-verify-file (file)
  "Check whether FILE exists in the current project."
  (file-exists-p (projectile-expand-root file)))

which resulted in an remarkable speedup.

bbatsov added a commit that referenced this issue Oct 31, 2017

[#1183] Properly cache the 'generic project type
Before this chance if the project type was 'generic there was project
type caching.

For the issue to be solved completely we'll need to introduce
something similar to the projectile-cached-project-name, though.
@bbatsov

This comment has been minimized.

Owner

bbatsov commented Oct 31, 2017

I've fixed the fixing for 'generic project types, so this should work ok within projects, although to fix it better it needs to use buffer-local caching for the project type. (similar to what we have for projectile-cached-project-name) As I don't have time for this I'll just remove the project type from the the default modeline. If someone liked it - they can just add it back themselves.

@bbatsov bbatsov closed this in 7892f64 Oct 31, 2017

bbatsov added a commit that referenced this issue Jan 18, 2018

[#1183] Speed up projectile-verify-file
It was implemented in terms of wildcard expansion just to handle
cabal projects. Now the cabal project detection simply uses a
different function.

bbatsov added a commit that referenced this issue Jan 18, 2018

[#1183] Tweak a bit the project type detection order
By putting the slowest checks last we make sure that people won't have
to experience them unless all other fast type checks have failed.

See the discussion on the ticket for more details.
@bbatsov

This comment has been minimized.

Owner

bbatsov commented Jan 18, 2018

@dato Good points and very valid. I did a couple of tweaks to the project detection now. There are so many easy wins in Projectile here and there, unfortunately I rarely find time for it these days.

bbatsov added a commit that referenced this issue Jan 18, 2018

[#1183] Tweak further the project type detection order
Push some (subjectively) more common project types earlier in the
resolution order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment