[Feature request] Speeding up query performance of rdm

[ngkaho1234]

Please mark appropriate

• rtags (rdm/rc/rp)
• Emacs Lisp
  • ac-rtags.el
  • company-rtags.el
  • helm-rtags.el
  • flycheck-rtags.el
  • ivy-rtags.el
  • rtags.el

Feature request

The query performance of RTags working on large projects such as Linux kernel is quite slow compared to some existing solutions like Eclipse... For example, the first query of a symbol name (--find-symbols) right after the daemon's being started takes about 5 minutes if the source code and RTags datafiles are all located on a local filesystem on HDD. I don't know if there are any solution we can work on to improve the query performance of RTags on spinning drive.

[Andersbakken]

The database is generally too large to fit in memory so a slow drive is likely to cause problems for most solutions. I've considered building some sort of index in memory when indexing is finished but this could have adverse effects on projects where you make a lot of changes. I haven't done a lot of profiling on the source code that handles find-symbols but 5 minutes is clearly not acceptable. This is after the indexing has finished I assume?

[ngkaho1234]

Yes. A query operation was issued after indexing had finished and without Linux page cache being warmed up, that operation took 5 minutes to walk ~20000 data files.
I did an experiment on adding a persistent database for mapping SYMBOL_NAMES -> File ID. Although the time required for the first indexing query to finish greatly reduced from 5 minutes to 2 seconds, the database entry format used still need to be redesigned for clarity and functionality (and Regex support/Case sensitivity toggle won't work under the current indexing scheme). Here is the link: ngkaho1234@b3e9f41

Also, here is the compile commands dot json of linux-4.11.9 (the compilation configuration file is taken from ArchLinux's core/linux): https://gist.github.com/1fb10ea5e08b72ebc578d55c0cc98834

[sciamano]

What would you suggest to do the profiling? I also see issues with the query performance for a large project: the indexing take some time, but that is fine (eclipse and friends are also slow there), but then the query never ends (maybe I was not patient enough...). And in my setup there is no HDD involved, I have just symlinked ~/.cache/rtags to /dev/shm/rtags, so the index is in fact in-memory (and there are dozens of GB of RAM).

[MaskRay]

The "head-of-line blocking" also bothers me. It easily takes me more than 6 seconds to do the first query.

% du -sh ~/.cache/rtags/_home_ray_Dev_Bin_radare2_/
264M    /home/ray/.cache/rtags/_home_ray_Dev_Bin_radare2_/