list: optimize -R/--recursive

[efiop]

http://bench.dvc.org/

[rlamy]

A significant part of the recent performance regression comes from 4965995, specifically this line:

dvc/dvc/repo/ls.py

Line 88 in 3c3cbee

if fs.isdvc(info["name"], recursive=True) or not dvc_only:

We could swipe that issue under the rug by swapping the arguments of the or, but looking at the rest of the code in _ls(), that fs.isdvc() seems redundant.

But the main issue seems to be that there are a lot of redundant path operations, splitting and recombining them multiple times in _DvcFileSystem.info() and similar methods.

[efiop]

📉

[rlamy]

After #8241 is merged, there will still be a few easy targets for optimisation:

• _DvcFileSystem.ls() shouldn't call ._info() but rather directly gather the relevant info from the underlying fs/dvc_fs, using .ls(.., detail=True).
• _info() ends up calling _update() on every file that is passed to it, it should probably only do that for directories. (Note that the impact of this is compounded by the previous point).
• In dvc.repo.ls._ls(), the following lines take 30% of the run-time just to convert filenames into paths:

  dvc/dvc/repo/ls.py

  Lines 73 to 76 in 8432de5

  	for name, entry in files.items():
  	entry_fs_path = fs.path.join(root, name)
  	relparts = fs.path.relparts(entry_fs_path, fs_path)
  	name = os.path.join(*relparts)

[efiop]

Turns out I accidentally re-discovered and fixed all three points from @rlamy 's research ^

Fixed by #8779 and #8780 . Closing.

Kudos @rlamy 🙏

[efiop]

For the record, this is how it looks now