Swiper takes over a second to start on very long files #416

Closed
Malabarba opened this Issue Mar 3, 2016 · 10 comments

Projects

None yet

5 participants

@Malabarba

This happens before I even start to type anything. Everytime I invoke swiper on a very large file (11k lines), it takes about 2 seconds to until the prompt shows up.
Maybe you're already aware of this. And maybe it's not something you can fix. But I just thought I'd let you know.

@abo-abo
Owner
abo-abo commented Mar 4, 2016

Swiper traverses the whole buffer to collect the lines. This can be a lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start up.

Another known issue is with visual-line-mode - it can slow down things to a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k lines. Perhaps also try counsel-grep - an async swiper alternative for huge files.

@Malabarba

I suggest trying with minimal config - 2s sounds way too long for 11k
lines.

Or maybe my laptop is just slower than yours. :-)

Anyway, I've only just enabled swiper. I'll just keep using it for now and
see whether I experience that on any other files.

Thanks for the package

On Fri, Mar 4, 2016 at 5:01 AM Oleh Krehel notifications@github.com wrote:

Swiper traverses the whole buffer to collect the lines. This can be a
lengthy operation for large buffers.

However, having just tested org.el (25k lines), it takes < 0.5s to start
up.

Another known issue is with visual-line-mode - it can slow down things to
a crawl for large files.

I suggest trying with minimal config - 2s sounds way too long for 11k
lines. Perhaps also try counsel-grep - an async swiper alternative for
huge files.


Reply to this email directly or view it on GitHub
#416 (comment).

@xendk
xendk commented Mar 10, 2016

Something else is also a factor. It takes >2 secs on paradox-list-packages buffer, and there's only ~4200 lines. It also looses the font-lock coloring, which it doesn't for a regular php-mode buffer, so maybe there's a relation there.

@novoid
novoid commented Apr 9, 2016

I have the same issue with several of my large Org-mode files. On my 💻 intel i5 with GNU Emacs 24.4.1 on Debian GNU/Linux and swiper-20160124.429 (from elpa):

number of lines of buffer seconds until search query entry is possible
48000 20
30000 8
14000 1
6700 1

So on my side, it is clearly not a second as mentioned in the title of this issue. It's so slow, that it's basically useless to me. 😦

However, I do like Swiper very much. Therefore, as a workaround, I plan to switch my search method according to the number of lines of the current buffer. As an Elisp amateur, I let you know when I've got something working here.

Best case - of course - would be to find the reason for this performance lag. To shed some objective light on this issue, I did a profiler report for the time between invoking Swiper and the display of the search buffer before I start typing my search query:

Function CPU Samples %
-- call-interactively 34706 98%
--- swiper 34618 98%
---- apply 34618 98%
----- #<compiled 0x156eb5d> 34618 98%
------ swiper--ivy 34618 98%
------- swiper--candidates 32825 93%
-------- replace-regexp-in-string 88 0%
--------- apply 24 0%
--------+ funcall 8 0%
------+ ivy-read 196 0%
--+ byte-code 49 0%
--+ minibuffer-complete 31 0%
--+ execute-extended-command 8 0%
-+ ... 452 1%
-+ yas--post-command-handler 16 0%
-+ timer-event-handler 6 0%
-+ redisplay_internal (C function) 4 0%

So, swiper--candidates takes up 93 percent of my CPU cycles when "launching" a search process.

Let me know when I can help you finding more information on this issue.

Thanks for your great work! 👍

@abo-abo
Owner
abo-abo commented Apr 9, 2016

@novoid

See the function from my config. It combines the best for swiper and counsel-grep, depending on buffer-size.

(global-set-key "\C-s" 'ora-swiper)

(defun ora-swiper ()
  (interactive)
  (if (and (buffer-file-name)
           (not (ignore-errors
                  (file-remote-p (buffer-file-name))))
           (if (eq major-mode 'org-mode)
               (> (buffer-size) 60000)
             (> (buffer-size) 300000)))
      (progn
        (save-buffer)
        (counsel-grep))
    (swiper--ivy (swiper--candidates))))

I've been using it for weeks now, it's really convenient. The start-up time for counsel-grep is 0.000001s. Also I recently made it match 40 times faster by removing grep's --ignore-case flag. The difference was shocking.

For a 60MB file consisting of org.el copied over and over until the whole file is almost two million lines long, counsel-grep:

  • Starts up in 0.0001s.
  • Finds 448 matches for input simple in 0.082926135s.
  • Finds 495552 matches for input org in 10.495491079s.

So I suggest you try the code above, let me know how it goes.

Thanks for your great work!

You're welcome, of course. I'm glad people like it.

@novoid
novoid commented Apr 9, 2016

@abo-abo

Thanks for the ultra-fast response!

I do get issues with your code snippet though:

(swiper--ivy (swiper--candidates)) results in (wrong-type-argument stringp (#(" followed by a string containing the whole buffer.

counsel-grep results in Symbol's function definition is void: ivy-set-display-transformer. I found #404 but could not resolve the issue yet.

Remark: "I found #404" is really funny - pun not indented.

@abo-abo
Owner
abo-abo commented Apr 9, 2016

Looks like you're using an outdated version of swiper. Possibly from GELPA instead of MELPA. Update and try again.

I'll try to release 0.8.0 in GELPA sometime this month. Can't have a 4 month out of date version around.

@novoid
novoid commented Apr 9, 2016

@abo-abo

To my astonishment, I figured out that due to a misconfiguration I had very old packages all over. So after a long sessions of upgrading all packages, counsel and swiper and your code snippet are working great.

I summarized my today's journey on my blog.

Thanks for your great help!

@manuel-uberti manuel-uberti added a commit to manuel-uberti/.emacs.d that referenced this issue Apr 10, 2016
@manuel-uberti manuel-uberti Add custom command for Swiper
Choose between `swiper--ivy' and `counsel-grep' based upon file size.

See: abo-abo/swiper#416 (comment)
a1c3ee1
@manuel-uberti
Contributor

Thanks for the custom function, really awesome.

@jonathanchu jonathanchu added a commit to jonathanchu/dotemacs that referenced this issue Apr 14, 2016
@jonathanchu jonathanchu Use different methods of search for large files
This helps speed up searching in large files. Taken directly from:

abo-abo/swiper#416 (comment)
http://karl-voit.at/2016/04/09/chosing-emacs-search-method/
c694709
@abo-abo
Owner
abo-abo commented May 3, 2016

To wrap this up, the new function counsel-grep-or-swiper solves the problems with the initial delay. It will use swiper for small files and buffers without a file, and counsel-grep for large ones. counsel-grep has negligible start-up time, and can handle files with millions of lines.

@abo-abo abo-abo closed this May 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment