New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swiper takes over a second to start on very long files #416

Closed
Malabarba opened this Issue Mar 3, 2016 · 17 comments

Comments

Projects
None yet
9 participants
@Malabarba

Malabarba commented Mar 3, 2016

This happens before I even start to type anything. Everytime I invoke swiper on a very large file (11k lines), it takes about 2 seconds to until the prompt shows up.
Maybe you're already aware of this. And maybe it's not something you can fix. But I just thought I'd let you know.

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Mar 4, 2016

Owner

Swiper traverses the whole buffer to collect the lines. This can be a lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start up.

Another known issue is with visual-line-mode - it can slow down things to a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k lines. Perhaps also try counsel-grep - an async swiper alternative for huge files.

Owner

abo-abo commented Mar 4, 2016

Swiper traverses the whole buffer to collect the lines. This can be a lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start up.

Another known issue is with visual-line-mode - it can slow down things to a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k lines. Perhaps also try counsel-grep - an async swiper alternative for huge files.

@Malabarba

This comment has been minimized.

Show comment
Hide comment
@Malabarba

Malabarba Mar 4, 2016

I suggest trying with minimal config - 2s sounds way too long for 11k
lines.

Or maybe my laptop is just slower than yours. :-)

Anyway, I've only just enabled swiper. I'll just keep using it for now and
see whether I experience that on any other files.

Thanks for the package

On Fri, Mar 4, 2016 at 5:01 AM Oleh Krehel notifications@github.com wrote:

Swiper traverses the whole buffer to collect the lines. This can be a
lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start
up.

Another known issue is with visual-line-mode - it can slow down things to
a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k
lines. Perhaps also try counsel-grep - an async swiper alternative for
huge files.


Reply to this email directly or view it on GitHub
#416 (comment).

Malabarba commented Mar 4, 2016

I suggest trying with minimal config - 2s sounds way too long for 11k
lines.

Or maybe my laptop is just slower than yours. :-)

Anyway, I've only just enabled swiper. I'll just keep using it for now and
see whether I experience that on any other files.

Thanks for the package

On Fri, Mar 4, 2016 at 5:01 AM Oleh Krehel notifications@github.com wrote:

Swiper traverses the whole buffer to collect the lines. This can be a
lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start
up.

Another known issue is with visual-line-mode - it can slow down things to
a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k
lines. Perhaps also try counsel-grep - an async swiper alternative for
huge files.


Reply to this email directly or view it on GitHub
#416 (comment).

@xendk

This comment has been minimized.

Show comment
Hide comment
@xendk

xendk Mar 10, 2016

Something else is also a factor. It takes >2 secs on paradox-list-packages buffer, and there's only ~4200 lines. It also looses the font-lock coloring, which it doesn't for a regular php-mode buffer, so maybe there's a relation there.

xendk commented Mar 10, 2016

Something else is also a factor. It takes >2 secs on paradox-list-packages buffer, and there's only ~4200 lines. It also looses the font-lock coloring, which it doesn't for a regular php-mode buffer, so maybe there's a relation there.

@novoid

This comment has been minimized.

Show comment
Hide comment
@novoid

novoid Apr 9, 2016

I have the same issue with several of my large Org-mode files. On my 💻 intel i5 with GNU Emacs 24.4.1 on Debian GNU/Linux and swiper-20160124.429 (from elpa):

number of lines of buffer seconds until search query entry is possible
48000 20
30000 8
14000 1
6700 1

So on my side, it is clearly not a second as mentioned in the title of this issue. It's so slow, that it's basically useless to me. 😦

However, I do like Swiper very much. Therefore, as a workaround, I plan to switch my search method according to the number of lines of the current buffer. As an Elisp amateur, I let you know when I've got something working here.

Best case - of course - would be to find the reason for this performance lag. To shed some objective light on this issue, I did a profiler report for the time between invoking Swiper and the display of the search buffer before I start typing my search query:

Function CPU Samples %
-- call-interactively 34706 98%
--- swiper 34618 98%
---- apply 34618 98%
----- #<compiled 0x156eb5d> 34618 98%
------ swiper--ivy 34618 98%
------- swiper--candidates 32825 93%
-------- replace-regexp-in-string 88 0%
--------- apply 24 0%
--------+ funcall 8 0%
------+ ivy-read 196 0%
--+ byte-code 49 0%
--+ minibuffer-complete 31 0%
--+ execute-extended-command 8 0%
-+ ... 452 1%
-+ yas--post-command-handler 16 0%
-+ timer-event-handler 6 0%
-+ redisplay_internal (C function) 4 0%

So, swiper--candidates takes up 93 percent of my CPU cycles when "launching" a search process.

Let me know when I can help you finding more information on this issue.

Thanks for your great work! 👍

novoid commented Apr 9, 2016

I have the same issue with several of my large Org-mode files. On my 💻 intel i5 with GNU Emacs 24.4.1 on Debian GNU/Linux and swiper-20160124.429 (from elpa):

number of lines of buffer seconds until search query entry is possible
48000 20
30000 8
14000 1
6700 1

So on my side, it is clearly not a second as mentioned in the title of this issue. It's so slow, that it's basically useless to me. 😦

However, I do like Swiper very much. Therefore, as a workaround, I plan to switch my search method according to the number of lines of the current buffer. As an Elisp amateur, I let you know when I've got something working here.

Best case - of course - would be to find the reason for this performance lag. To shed some objective light on this issue, I did a profiler report for the time between invoking Swiper and the display of the search buffer before I start typing my search query:

Function CPU Samples %
-- call-interactively 34706 98%
--- swiper 34618 98%
---- apply 34618 98%
----- #<compiled 0x156eb5d> 34618 98%
------ swiper--ivy 34618 98%
------- swiper--candidates 32825 93%
-------- replace-regexp-in-string 88 0%
--------- apply 24 0%
--------+ funcall 8 0%
------+ ivy-read 196 0%
--+ byte-code 49 0%
--+ minibuffer-complete 31 0%
--+ execute-extended-command 8 0%
-+ ... 452 1%
-+ yas--post-command-handler 16 0%
-+ timer-event-handler 6 0%
-+ redisplay_internal (C function) 4 0%

So, swiper--candidates takes up 93 percent of my CPU cycles when "launching" a search process.

Let me know when I can help you finding more information on this issue.

Thanks for your great work! 👍

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Apr 9, 2016

Owner

@novoid

See the function from my config. It combines the best for swiper and counsel-grep, depending on buffer-size.

(global-set-key "\C-s" 'ora-swiper)

(defun ora-swiper ()
  (interactive)
  (if (and (buffer-file-name)
           (not (ignore-errors
                  (file-remote-p (buffer-file-name))))
           (if (eq major-mode 'org-mode)
               (> (buffer-size) 60000)
             (> (buffer-size) 300000)))
      (progn
        (save-buffer)
        (counsel-grep))
    (swiper--ivy (swiper--candidates))))

I've been using it for weeks now, it's really convenient. The start-up time for counsel-grep is 0.000001s. Also I recently made it match 40 times faster by removing grep's --ignore-case flag. The difference was shocking.

For a 60MB file consisting of org.el copied over and over until the whole file is almost two million lines long, counsel-grep:

  • Starts up in 0.0001s.
  • Finds 448 matches for input simple in 0.082926135s.
  • Finds 495552 matches for input org in 10.495491079s.

So I suggest you try the code above, let me know how it goes.

Thanks for your great work!

You're welcome, of course. I'm glad people like it.

Owner

abo-abo commented Apr 9, 2016

@novoid

See the function from my config. It combines the best for swiper and counsel-grep, depending on buffer-size.

(global-set-key "\C-s" 'ora-swiper)

(defun ora-swiper ()
  (interactive)
  (if (and (buffer-file-name)
           (not (ignore-errors
                  (file-remote-p (buffer-file-name))))
           (if (eq major-mode 'org-mode)
               (> (buffer-size) 60000)
             (> (buffer-size) 300000)))
      (progn
        (save-buffer)
        (counsel-grep))
    (swiper--ivy (swiper--candidates))))

I've been using it for weeks now, it's really convenient. The start-up time for counsel-grep is 0.000001s. Also I recently made it match 40 times faster by removing grep's --ignore-case flag. The difference was shocking.

For a 60MB file consisting of org.el copied over and over until the whole file is almost two million lines long, counsel-grep:

  • Starts up in 0.0001s.
  • Finds 448 matches for input simple in 0.082926135s.
  • Finds 495552 matches for input org in 10.495491079s.

So I suggest you try the code above, let me know how it goes.

Thanks for your great work!

You're welcome, of course. I'm glad people like it.

@novoid

This comment has been minimized.

Show comment
Hide comment
@novoid

novoid Apr 9, 2016

@abo-abo

Thanks for the ultra-fast response!

I do get issues with your code snippet though:

(swiper--ivy (swiper--candidates)) results in (wrong-type-argument stringp (#(" followed by a string containing the whole buffer.

counsel-grep results in Symbol's function definition is void: ivy-set-display-transformer. I found #404 but could not resolve the issue yet.

Remark: "I found #404" is really funny - pun not indented.

novoid commented Apr 9, 2016

@abo-abo

Thanks for the ultra-fast response!

I do get issues with your code snippet though:

(swiper--ivy (swiper--candidates)) results in (wrong-type-argument stringp (#(" followed by a string containing the whole buffer.

counsel-grep results in Symbol's function definition is void: ivy-set-display-transformer. I found #404 but could not resolve the issue yet.

Remark: "I found #404" is really funny - pun not indented.

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Apr 9, 2016

Owner

Looks like you're using an outdated version of swiper. Possibly from GELPA instead of MELPA. Update and try again.

I'll try to release 0.8.0 in GELPA sometime this month. Can't have a 4 month out of date version around.

Owner

abo-abo commented Apr 9, 2016

Looks like you're using an outdated version of swiper. Possibly from GELPA instead of MELPA. Update and try again.

I'll try to release 0.8.0 in GELPA sometime this month. Can't have a 4 month out of date version around.

@novoid

This comment has been minimized.

Show comment
Hide comment
@novoid

novoid Apr 9, 2016

@abo-abo

To my astonishment, I figured out that due to a misconfiguration I had very old packages all over. So after a long sessions of upgrading all packages, counsel and swiper and your code snippet are working great.

I summarized my today's journey on my blog.

Thanks for your great help!

novoid commented Apr 9, 2016

@abo-abo

To my astonishment, I figured out that due to a misconfiguration I had very old packages all over. So after a long sessions of upgrading all packages, counsel and swiper and your code snippet are working great.

I summarized my today's journey on my blog.

Thanks for your great help!

manuel-uberti pushed a commit to manuel-uberti/.emacs.d that referenced this issue Apr 10, 2016

manuel-uberti
Add custom command for Swiper
Choose between `swiper--ivy' and `counsel-grep' based upon file size.

See: abo-abo/swiper#416 (comment)
@manuel-uberti

This comment has been minimized.

Show comment
Hide comment
@manuel-uberti

manuel-uberti Apr 10, 2016

Contributor

Thanks for the custom function, really awesome.

Contributor

manuel-uberti commented Apr 10, 2016

Thanks for the custom function, really awesome.

jonathanchu added a commit to jonathanchu/dotemacs that referenced this issue Apr 14, 2016

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo May 3, 2016

Owner

To wrap this up, the new function counsel-grep-or-swiper solves the problems with the initial delay. It will use swiper for small files and buffers without a file, and counsel-grep for large ones. counsel-grep has negligible start-up time, and can handle files with millions of lines.

Owner

abo-abo commented May 3, 2016

To wrap this up, the new function counsel-grep-or-swiper solves the problems with the initial delay. It will use swiper for small files and buffers without a file, and counsel-grep for large ones. counsel-grep has negligible start-up time, and can handle files with millions of lines.

@abo-abo abo-abo closed this May 3, 2016

@technician77

This comment has been minimized.

Show comment
Hide comment
@technician77

technician77 Jul 17, 2017

I'm no coder therefore this might be nonsense but wouldn't it be possible to show swiper input line directly while results are loaded asynchronously in the background?

technician77 commented Jul 17, 2017

I'm no coder therefore this might be nonsense but wouldn't it be possible to show swiper input line directly while results are loaded asynchronously in the background?

@Pitometsu

This comment has been minimized.

Show comment
Hide comment
@Pitometsu

Pitometsu Nov 23, 2017

Same for me: with ~2k lines – ~3 seconds delay. But I use visual-line-mode. Is there any workaround or way to speed it up? @abo-abo would be glad any help.

Pitometsu commented Nov 23, 2017

Same for me: with ~2k lines – ~3 seconds delay. But I use visual-line-mode. Is there any workaround or way to speed it up? @abo-abo would be glad any help.

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Nov 23, 2017

Owner

@Pitometsu Like it says above, counsel-grep-or-swiper is the recommended solution.ν

Owner

abo-abo commented Nov 23, 2017

@Pitometsu Like it says above, counsel-grep-or-swiper is the recommended solution.ν

@guilhermemtr

This comment has been minimized.

Show comment
Hide comment
@guilhermemtr

guilhermemtr Jan 4, 2018

@abo-abo what if the buffer is not a file? It is also taking me like 2 seconds when I try to do a search on
paradox-list-packages buffer.

guilhermemtr commented Jan 4, 2018

@abo-abo what if the buffer is not a file? It is also taking me like 2 seconds when I try to do a search on
paradox-list-packages buffer.

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Jan 4, 2018

Owner

Here's what I get in the package buffer (4143 lines):

(benchmark-run 1 (swiper--candidates))
;; => (0.517214548 5 0.4373610800000005)

0.5s start-up time is bearable to me. PRs to improve the speed of swiper--candidates are welcome, but it will require a fundamental change in the functionality of swiper to get the startup below 0.1s. Most of the time is spent on calculating the line numbers; if we decide we don't want line numbers, it can be much faster.

Owner

abo-abo commented Jan 4, 2018

Here's what I get in the package buffer (4143 lines):

(benchmark-run 1 (swiper--candidates))
;; => (0.517214548 5 0.4373610800000005)

0.5s start-up time is bearable to me. PRs to improve the speed of swiper--candidates are welcome, but it will require a fundamental change in the functionality of swiper to get the startup below 0.1s. Most of the time is spent on calculating the line numbers; if we decide we don't want line numbers, it can be much faster.

@drrlvn

This comment has been minimized.

Show comment
Hide comment
@drrlvn

drrlvn Jan 4, 2018

Contributor

Maybe the line number calculation can be made configurable so that it could be disabled for better performance.

Contributor

drrlvn commented Jan 4, 2018

Maybe the line number calculation can be made configurable so that it could be disabled for better performance.

@abo-abo

This comment has been minimized.

Show comment
Hide comment
@abo-abo

abo-abo Jan 4, 2018

Owner

Having investigated it a bit further, it seems swiper can't be made much faster with a synchronized collection. With the same packages buffer:

(benchmark-run 1 (split-string (buffer-string) "\n"))
;; => (0.242485285 2 0.19264026999999828)

So only twice as fast start-up. Not good enough to be worth the effort, especially considering the there are bigger buffers.

Here's my draft implementation of an async swiper, if anyone wants to try it:

(defun ivy-isearch-function (s)
  (when (> (length s) 0)
    (let ((re (setq ivy--old-re (ivy--regex-plus s)))
          res)
      (with-ivy-window
        (goto-char (point-min))
        (while (re-search-forward re nil t)
          (push
           (propertize
            (buffer-substring-no-properties
             (line-beginning-position)
             (line-end-position))
            'pos (match-end 0))
           res))
        (nreverse res)))))

(defun ivy-isearch-action (x)
  (let ((pt (and x (get-text-property 0 'pos x))))
    (when pt
      (goto-char pt))))

(defun ivy--isearch-update-input ()
  (when ivy--old-re
    (swiper--cleanup)
    (with-ivy-window
      (ivy-isearch-action (ivy-state-current ivy-last))
      (swiper--add-overlays
       ivy--old-re
       (window-start)
       (window-end (selected-window) t)))))

(defun ivy-isearch ()
  (interactive)
  (ivy-read "Swiper: " #'ivy-isearch-function
            :dynamic-collection t
            :action #'ivy-isearch-action
            :update-fn #'ivy--isearch-update-input
            :unwind #'swiper--cleanup
            :preselect (buffer-substring-no-properties
                        (line-beginning-position)
                        (line-end-position))))

It still needs some refinement.

Owner

abo-abo commented Jan 4, 2018

Having investigated it a bit further, it seems swiper can't be made much faster with a synchronized collection. With the same packages buffer:

(benchmark-run 1 (split-string (buffer-string) "\n"))
;; => (0.242485285 2 0.19264026999999828)

So only twice as fast start-up. Not good enough to be worth the effort, especially considering the there are bigger buffers.

Here's my draft implementation of an async swiper, if anyone wants to try it:

(defun ivy-isearch-function (s)
  (when (> (length s) 0)
    (let ((re (setq ivy--old-re (ivy--regex-plus s)))
          res)
      (with-ivy-window
        (goto-char (point-min))
        (while (re-search-forward re nil t)
          (push
           (propertize
            (buffer-substring-no-properties
             (line-beginning-position)
             (line-end-position))
            'pos (match-end 0))
           res))
        (nreverse res)))))

(defun ivy-isearch-action (x)
  (let ((pt (and x (get-text-property 0 'pos x))))
    (when pt
      (goto-char pt))))

(defun ivy--isearch-update-input ()
  (when ivy--old-re
    (swiper--cleanup)
    (with-ivy-window
      (ivy-isearch-action (ivy-state-current ivy-last))
      (swiper--add-overlays
       ivy--old-re
       (window-start)
       (window-end (selected-window) t)))))

(defun ivy-isearch ()
  (interactive)
  (ivy-read "Swiper: " #'ivy-isearch-function
            :dynamic-collection t
            :action #'ivy-isearch-action
            :update-fn #'ivy--isearch-update-input
            :unwind #'swiper--cleanup
            :preselect (buffer-substring-no-properties
                        (line-beginning-position)
                        (line-end-position))))

It still needs some refinement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment