Slow \includegraphic completion when existing a lot of files.

[jiangyinzuo]

Description

My project file structure is:

datasets/
fig1.pdf
fig2.pdf
...

datasets/ contains a lot of files that are not figures.

I want to ignore datasets/ directory when completing \includegraphics commands. What should I do?

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Do you use a latexmkrc file?

No

VimtexInfo

System info:
  OS: Ubuntu 24.10
  Vim version: NVIM v0.11.0-dev-c7d4a77
  Has clientserver: true
  Servername: /run/user/0//nvim.72964.0

VimTeX project: main
  base: main.tex
  root: /mnt/d/doc2/PROJECTS/gpudb-project-ictdb/graph-ann
  tex: /mnt/d/doc2/PROJECTS/gpudb-project-ictdb/graph-ann/main.tex
  main parser: recursive search
  document class: IEEEtran
  packages: algorithm2e amsbsy amsfonts amsgen amsmath amsopn amssymb amstext array atbegshi atbegshi-ltx atveryend atveryend-ltx bigintcalc bitset cite epstopdf-base etoolbox gettitlestring graphics graphicx grfext hycolor hyperref ifoddpage iftex ifthen infwarerr intcalc keyval kvdefinekeys kvoptions kvsetkeys ltxcmds multirow nameref pdfescape pdftexcmds refcount relsize rerunfilecheck stringenc subfigure tabularx textcomp trig uniquecounter url xcolor xspace
  source files:
    main.tex
    abstract.tex
    1-intro.tex
    2-algos.tex
    3-setup.tex
    4-eval.tex
    5-conclude.tex
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: General
  qf method: LaTeX logfile

[jiangyinzuo]

globpath(b:vimtex.root, ...) in autoload/vimtex/complete.vim is slow. Maybe we can customize the glob path like globpath("`fd`", ...) ?

[lervag]

This is the first time I've heard someone complain about performance of the file completion for includegraphics. I could look into it, but it is hard without having a reproducible example.

Are you able to build a simple example and a step-by-step guide for how to reproduce the experienced lag?

[lervag]

globpath(b:vimtex.root, ...) in autoload/vimtex/complete.vim is slow.

I've found it to be quite fast myself, although I might have never really tested it in any crazy large folders.

Maybe we can customize the glob path like globpath("fd", ...) ?

That's not trivial and probably not so robust. I think, before looking at solutions here, I want to find a reliable way to reproduce the lag.

[jiangyinzuo]

I use WSL and put my tex project under /mnt/d/.... I found that accessing directories under /mnt/d/... in WSL can be very slow.

I created a folder /mnt/d/test and touched 10000 files under /mnt/d/test:

#!/bin/bash
for ((i = 1; i <= 10000; i++)); do
    touch ${i}.txt
done

Then, I observed that
ls linux command under /mnt/d/test or vim command :echo globpath('.', '**/*') is very slow, but fd linux command is fast.

So I think it is a performance problem about WSL filesystem, rather than vimtex, see also microsoft/WSL#4197

[lervag]

Yes, I think this is a performance problem in Windows WSL, and I don't really want to add the complexity here in VimTeX to account for that.

Notice that, adding fd would require options and more complicated code to account for those who don't have that tool installed. I would need to add tests as well.

Now, if you were to suggest an updated with a PR, then I might consider to add it after guiding it into something that seems robust enough.

[Zulleyy3]

I can replicate this on Linux as I have accidentally created a stress test for this:
I generated a lot of graphics and tables and put them into a folder in the latex root to select and include from while writing.

It's a total of 39389 files across 20748 subdirectories and typing "\includedirectory" or "\input" or trying to edit a path freezes Neovim for multiple seconds.

[lervag]

I can replicate this on Linux as I have accidentally created a stress test for this: I generated a lot of graphics and tables and put them into a folder in the latex root to select and include from while writing.

It's a total of 39389 files across 20748 subdirectories and typing "\includedirectory" or "\input" or trying to edit a path freezes Neovim for multiple seconds.

Huh, ok; but is this really a realistic scenario? I mean, why would you ever have so many files and subdirectories - and if you do, why can't you just move the latex source to a separate independent directory?

I'm not trying to be difficult, but I'm also not very interested in adding complexity to the project for extremely uncommon edge cases.

[Zulleyy3]

The generated graphics (generating is the reason why there is so many of them) were initially in a different directory and I moved them into the latex source directory, because it would be nice to include them quickly as I sort through them. I thought it would be a nice workflow initially, so I naturally ended up in that situation 😊

I think it's fair to not want to add complexity over this, especially if it involves external commands that might not be available on all platforms.

I have searched the documentation for it yet (maybe it is available), but would it be possible to add incremental completion (as an option) rather than listing all files at once?
So in the case of this issue typing \includegraphics{data would only suggest "datasets/" and once that is completed begin adding onto that with files within that directory. That could avoid slowdowns in cases like mine and the case of the initial poster.

It's not a big deal not to. I can't tell if this would be difficult to implement or even desirable for most users.

Thank you for your work.

[lervag]

I have searched the documentation for it yet (maybe it is available), but would it be possible to add incremental completion (as an option) rather than listing all files at once? So in the case of this issue typing \includegraphics{data would only suggest "datasets/" and once that is completed begin adding onto that with files within that directory. That could avoid slowdowns in cases like mine and the case of the initial poster.

The issue with this, IMHO, is that it would break the workflow of most people - myself included. I believe most people would have O(10) image files. Crazy people might have O(100) image files in a document (say, less than 1000). The current implementation would still work well with that.

Further, the current behaviour takes the current input and uses it to filter the files as a regex. I.e., let's say you type \includegraphics{\w*\.p<c-x><c-o>; then VimTeX will first compute the list of candidates, then filter it with the supplied regex. I believe most people don't really use this feature, but again, it's a breaking change.

Now, let's say I add an option for to keep the current default but to allow a "nested" search. Yes, perhaps it would solve the current issue for you, but then someone else would have the exact same issue as you except their files are in a flat structure. This solution would not be adequate for them.

Notice, though: you may actually come far here if you instead rely on the built-in file completion. That is, instead of \includegraphics{<c-x><c-o>, you can use \includegraphics{<c-x><c-f> - it would start completing file names. Yes, it will complete non-image files as well, but it may still be close to what you want.

If you are using autocomplete plugins, they may allow you to configure this behaviour as well, although it will probably require some clever config.

It's not a big deal not to. I can't tell if this would be difficult to implement or even desirable for most users.

It's not very hard to implement, but I'm very skeptical of the value it brings compared to the cost.

Thank you for your work.

Glad you enjoy it!