Mount w/large snapshot fails

[akrabu]

Output of restic version

restic 0.15.1 compiled with go1.19.5 on darwin/arm64

How did you run restic exactly?

restic mount ~/mnt <specific snapshot-ID> (other IDs work, just not this one)

What backend/server/service did you use to store the repository?

Rclone:OneDrive

Expected behavior

The snapshot to mount properly

Actual behavior

Acts as if mounted, won't list the directory contents in the terminal or in Finder. Eventually gets killed by Zsh.

Steps to reproduce the behavior

Mount a snapshot with a LOT of tiny files in each directory, wait for it to say it's ready, try to use it. Terminal / Finder freezes. Zsh eventually kills it.

Do you have any idea what may have caused this?

Either high RAM usage or some bug with lots (and I do mean LOTS) of tiny files in one folder. Listing the directories straight off the source drive can take 2-3 minutes as it is.

Do you have an idea how to solve the issue?

Throw more RAM at it (I can't test this, I have one of those fancy new iMacs that would be a pain to open up). Other than that, not sure.

Did restic help you today? Did it make you happy in any way?

Best backup app out there!

[greatroar]

How much is lots, hundreds of millions? Restic constructs the directory structure in memory and keeps it there for at least as long as one process (Finder, ls) is keeping it open. I don't know how exactly much that costs, but a quick glance at the code suggests something on the order of a kilobyte per file, maybe a few.

It's more likely the index files that take up a lot of memory.

[akrabu]

Looks like this snapshot in particular is 38,206,383 files, 6.612TiB (restore-size). I just know it has a lot of tiny files in one folder and it takes minutes to enumerate on the disk itself. I don't have the disk now (the user has it), and since I can't mount it, there's no easy way to check specific folder counts.

[greatroar]

You can check how many files are in the dir with restic cat. It's a bit involved, but essentially you start with

restic cat snapshot <snapshotID>

then

restic cat blob <treeID>

from the tree ID shown in the snapshot's output, and go down the filesystem hierarchy using the subtree fields. The number of files in a directory is

restic cat blob <treeID> | jq '.nodes | length'

[MichaelEischer]

Looks like this is related to #2446, although with the extra problem that the fuse code in restic currently implements ReadDirAll() which returns all files in a directory at once. From a quick look at the fuse library we're using, providing an incremental ReadDir() implementation is currently not supported there.