Swiper takes over a second to start on very long files

[Malabarba]

This happens before I even start to type anything. Everytime I invoke swiper on a very large file (11k lines), it takes about 2 seconds to until the prompt shows up.
Maybe you're already aware of this. And maybe it's not something you can fix. But I just thought I'd let you know.

[abo-abo]

Swiper traverses the whole buffer to collect the lines. This can be a lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start up.

Another known issue is with visual-line-mode - it can slow down things to a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k lines. Perhaps also try counsel-grep - an async swiper alternative for huge files.

[Malabarba]

I suggest trying with minimal config - 2s sounds way too long for 11k
lines.

Or maybe my laptop is just slower than yours. :-)

Anyway, I've only just enabled swiper. I'll just keep using it for now and
see whether I experience that on any other files.

Thanks for the package

On Fri, Mar 4, 2016 at 5:01 AM Oleh Krehel notifications@github.com wrote:

Swiper traverses the whole buffer to collect the lines. This can be a
lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start
up.

Another known issue is with visual-line-mode - it can slow down things to
a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k
lines. Perhaps also try counsel-grep - an async swiper alternative for
huge files.

—
Reply to this email directly or view it on GitHub
#416 (comment).

[xendk]

Something else is also a factor. It takes >2 secs on paradox-list-packages buffer, and there's only ~4200 lines. It also looses the font-lock coloring, which it doesn't for a regular php-mode buffer, so maybe there's a relation there.

[novoid]

I have the same issue with several of my large Org-mode files. On my 💻 intel i5 with GNU Emacs 24.4.1 on Debian GNU/Linux and swiper-20160124.429 (from elpa):

number of lines of buffer	seconds until search query entry is possible
48000	20
30000	8
14000	1
6700	1

So on my side, it is clearly not a second as mentioned in the title of this issue. It's so slow, that it's basically useless to me. 😦

However, I do like Swiper very much. Therefore, as a workaround, I plan to switch my search method according to the number of lines of the current buffer. As an Elisp amateur, I let you know when I've got something working here.

Best case - of course - would be to find the reason for this performance lag. To shed some objective light on this issue, I did a profiler report for the time between invoking Swiper and the display of the search buffer before I start typing my search query:

Function	CPU Samples	%
-- call-interactively	34706	98%
--- swiper	34618	98%
---- apply	34618	98%
----- #<compiled 0x156eb5d>	34618	98%
------ swiper--ivy	34618	98%
------- swiper--candidates	32825	93%
-------- replace-regexp-in-string	88	0%
--------- apply	24	0%
--------+ funcall	8	0%
------+ ivy-read	196	0%
--+ byte-code	49	0%
--+ minibuffer-complete	31	0%
--+ execute-extended-command	8	0%
-+ ...	452	1%
-+ yas--post-command-handler	16	0%
-+ timer-event-handler	6	0%
-+ redisplay_internal (C function)	4	0%

So, swiper--candidates takes up 93 percent of my CPU cycles when "launching" a search process.

Let me know when I can help you finding more information on this issue.

Thanks for your great work! 👍

[abo-abo]

@novoid

See the function from my config. It combines the best for swiper and counsel-grep, depending on buffer-size.

(global-set-key "\C-s" 'ora-swiper)

(defun ora-swiper ()
  (interactive)
  (if (and (buffer-file-name)
           (not (ignore-errors
                  (file-remote-p (buffer-file-name))))
           (if (eq major-mode 'org-mode)
               (> (buffer-size) 60000)
             (> (buffer-size) 300000)))
      (progn
        (save-buffer)
        (counsel-grep))
    (swiper--ivy (swiper--candidates))))

I've been using it for weeks now, it's really convenient. The start-up time for counsel-grep is 0.000001s. Also I recently made it match 40 times faster by removing grep's --ignore-case flag. The difference was shocking.

For a 60MB file consisting of org.el copied over and over until the whole file is almost two million lines long, counsel-grep:

• Starts up in 0.0001s.
• Finds 448 matches for input simple in 0.082926135s.
• Finds 495552 matches for input org in 10.495491079s.

So I suggest you try the code above, let me know how it goes.

Thanks for your great work!

You're welcome, of course. I'm glad people like it.

[novoid]

@abo-abo

Thanks for the ultra-fast response!

I do get issues with your code snippet though:

(swiper--ivy (swiper--candidates)) results in (wrong-type-argument stringp (#(" followed by a string containing the whole buffer.

counsel-grep results in Symbol's function definition is void: ivy-set-display-transformer. I found #404 but could not resolve the issue yet.

Remark: "I found #404" is really funny - pun not indented.

[abo-abo]

Looks like you're using an outdated version of swiper. Possibly from GELPA instead of MELPA. Update and try again.

I'll try to release 0.8.0 in GELPA sometime this month. Can't have a 4 month out of date version around.

[novoid]

@abo-abo

To my astonishment, I figured out that due to a misconfiguration I had very old packages all over. So after a long sessions of upgrading all packages, counsel and swiper and your code snippet are working great.

I summarized my today's journey on my blog.

Thanks for your great help!

[manuel-uberti]

Thanks for the custom function, really awesome.

[abo-abo]

To wrap this up, the new function counsel-grep-or-swiper solves the problems with the initial delay. It will use swiper for small files and buffers without a file, and counsel-grep for large ones. counsel-grep has negligible start-up time, and can handle files with millions of lines.

[technician77]

I'm no coder therefore this might be nonsense but wouldn't it be possible to show swiper input line directly while results are loaded asynchronously in the background?

[Pitometsu]

Same for me: with ~2k lines – ~3 seconds delay. But I use visual-line-mode. Is there any workaround or way to speed it up? @abo-abo would be glad any help.

[abo-abo]

@Pitometsu Like it says above, counsel-grep-or-swiper is the recommended solution.ν

[guilhermemtr]

@abo-abo what if the buffer is not a file? It is also taking me like 2 seconds when I try to do a search on
paradox-list-packages buffer.

[abo-abo]

Here's what I get in the package buffer (4143 lines):

(benchmark-run 1 (swiper--candidates))
;; => (0.517214548 5 0.4373610800000005)

0.5s start-up time is bearable to me. PRs to improve the speed of swiper--candidates are welcome, but it will require a fundamental change in the functionality of swiper to get the startup below 0.1s. Most of the time is spent on calculating the line numbers; if we decide we don't want line numbers, it can be much faster.

[drrlvn]

Maybe the line number calculation can be made configurable so that it could be disabled for better performance.

[abo-abo]

Having investigated it a bit further, it seems swiper can't be made much faster with a synchronized collection. With the same packages buffer:

(benchmark-run 1 (split-string (buffer-string) "\n"))
;; => (0.242485285 2 0.19264026999999828)

So only twice as fast start-up. Not good enough to be worth the effort, especially considering the there are bigger buffers.

Here's my draft implementation of an async swiper, if anyone wants to try it:

(defun ivy-isearch-function (s)
  (when (> (length s) 0)
    (let ((re (setq ivy--old-re (ivy--regex-plus s)))
          res)
      (with-ivy-window
        (goto-char (point-min))
        (while (re-search-forward re nil t)
          (push
           (propertize
            (buffer-substring-no-properties
             (line-beginning-position)
             (line-end-position))
            'pos (match-end 0))
           res))
        (nreverse res)))))

(defun ivy-isearch-action (x)
  (let ((pt (and x (get-text-property 0 'pos x))))
    (when pt
      (goto-char pt))))

(defun ivy--isearch-update-input ()
  (when ivy--old-re
    (swiper--cleanup)
    (with-ivy-window
      (ivy-isearch-action (ivy-state-current ivy-last))
      (swiper--add-overlays
       ivy--old-re
       (window-start)
       (window-end (selected-window) t)))))

(defun ivy-isearch ()
  (interactive)
  (ivy-read "Swiper: " #'ivy-isearch-function
            :dynamic-collection t
            :action #'ivy-isearch-action
            :update-fn #'ivy--isearch-update-input
            :unwind #'swiper--cleanup
            :preselect (buffer-substring-no-properties
                        (line-beginning-position)
                        (line-end-position))))

It still needs some refinement.

[pengpengxp]

Having investigated it a bit further, it seems swiper can't be made much faster with a synchronized collection. With the same packages buffer:

(benchmark-run 1 (split-string (buffer-string) "\n"))
;; => (0.242485285 2 0.19264026999999828)

So only twice as fast start-up. Not good enough to be worth the effort, especially considering the there are bigger buffers.

Here's my draft implementation of an async swiper, if anyone wants to try it:

(defun ivy-isearch-function (s)
  (when (> (length s) 0)
    (let ((re (setq ivy--old-re (ivy--regex-plus s)))
          res)
      (with-ivy-window
        (goto-char (point-min))
        (while (re-search-forward re nil t)
          (push
           (propertize
            (buffer-substring-no-properties
             (line-beginning-position)
             (line-end-position))
            'pos (match-end 0))
           res))
        (nreverse res)))))

(defun ivy-isearch-action (x)
  (let ((pt (and x (get-text-property 0 'pos x))))
    (when pt
      (goto-char pt))))

(defun ivy--isearch-update-input ()
  (when ivy--old-re
    (swiper--cleanup)
    (with-ivy-window
      (ivy-isearch-action (ivy-state-current ivy-last))
      (swiper--add-overlays
       ivy--old-re
       (window-start)
       (window-end (selected-window) t)))))

(defun ivy-isearch ()
  (interactive)
  (ivy-read "Swiper: " #'ivy-isearch-function
            :dynamic-collection t
            :action #'ivy-isearch-action
            :update-fn #'ivy--isearch-update-input
            :unwind #'swiper--cleanup
            :preselect (buffer-substring-no-properties
                        (line-beginning-position)
                        (line-end-position))))

It still needs some refinement.

Will you refine ivy-isearch' function and put it into the next release? Or, in other words, current counsel-grep-or-swiper' is just ok?

[abo-abo]

@pengpengxp

Will you refine ivy-isearch' function and put it into the next release? Or, in other words, current counsel-grep-or-swiper' is just ok?

I'll look into refining it at some point. Right now I have a backlog of 90 unread open issues.

[otadmor]

Hi,
Look at this:
https://github.com/otadmor/emacs-conf/blob/master/emacs.d/lisp/swiper-async.el
[cross]Its not done, but working basically.
As I see, I have problems with history and with pressing yank-word twice. [/cross]
done

[abo-abo]

@otadmor Well done. The fuzzy search didn't yet work for me. In one test example, an Org file with 5292 lines, the performance is much slower than counsel-grep: takes around 2-5 seconds for the final results to refresh. Nonetheless, it's impressive, good job.

[otadmor]

Thanks.
Maybe I should send to the build in forward search the re builder- result? I'm not using the fuzzy search much so a list of "use cases" with which I could test this would help a lot.
Also, could you send me the problematic file so I could debug it? Does it takes long for the first result to show up or for all of the results to show up?

[otadmor]

Ivy, ignore order and fuzzy (after the \n fix) re-builder are working. I also added a plain text and regularexpression re-builders. I run ivy-rotate-preferred-builders to switch between the re builders.

If someone has a file which makes it slow ill be glad to debug it and fix.

[otadmor]

I have updated swiper-async to decide when to use grep (similar to counsel-grep-or-swiper). The difference is it does not saves the buffer if it different from the file and fallback to search-forward, supports "online" updating of the buffer while using swiper-async (fallback to the search-forward) and fallback to ivy-filter when there are atleast 3 chars in the input.

[abo-abo]

Just for reference, this issue is solved by using swiper-isearch.

[Alexander-Shukaev]

See #2390 (comment) and #1908. In short, my proposal is to teach swiper to use grep even for file-less buffers, where it could just pipe the contents of a file-less buffer to grep. The big advantage now is that grep is invoked asynchronously (see how counsel-grep-function calls counsel--async-command), so unlike the current swiper, it won't freeze your Emacs while collecting matches. Right now I have to use counsel-grep-or-swiper, which is configured in such a way that it always uses grep apart from file-less buffers, where the aforementioned feature could help.