Question - what is the expectation for how long a scan should take?

[Grendel76]

Hello all,

I have roughly 15tb of data on a network share that I access over my network using SMB. When I run a scan on Stash on my windows PC (Stash and my StashDB live on my windows client, my collection lives on the server), a scan of the linux based 15TB takes about 3 hours, give or take.

Is this a... normal timeframe? I have a guy on a forum who swears it should only take 4 seconds for a full scan of a 40tb collection, and the only reason my takes so long is my setup is broken.

Before I do a deep dive, I thought I would do a sanity check here.

Thanks!

[DogmaDragon]

4 seconds is unrealistic fiction.
The scanning task is CPU bound, so that will have an impact on time, but also other options you have enabled under the task can increase the time significantly. That said, 3 hours does not sound unreasonable to me, especially if you have pHash generation enabled.

[DogmaDragon]

@smith113-p just meant that any subsequent scans will be very quick if no new files are being added in which case it's possible to have 4 seconds. But it's not realistic if any new files are being added.

[smith113-p]

Yes -- @Grendel76 , are you saying that when you do a regular scan with no/few files added it still takes 3 hours?

[Grendel76]

Yes -- @Grendel76 , are you saying that when you do a regular scan with no/few files added it still takes 3 hours?

Yes. If I hit Scan right now, having added no files today, it will probably take 3 hours. I did in fact add no files at all today and after reading these replies, to test I hit scan:

8 minutes into the scan, it says 20%, estimates 44 minutes
10 minutes in, we're at 21% and it estimates an hour
15 minutes in, we're at 24% and it still estimates an hour

Since the tone of your posts seems to indicate this is maybe longer than expected, here are my library stats if it helps at all:

Scenes
Size: 14.7 TiB
Scenes: 36,372
Groups: 0
Duration: 1Y 9M 16h
Performers: 4,537

Images
Size: 77.7 GiB
Galleries: 297
Images: 98,854
Studios: 1,569
Tags: 1,873

The only options toggled on in Library are generate phashes and generate previews.

The Stash app lives on a windows PC, a 990 Pro m2 nvme SSD. I don't really know where the DB is located? It says stash-go.sqlite for path. Binary data storage type is Filesystem, and binary data filesystem path is blobs.

oh and lol I just saw this:

Parallel Scan/Generation
Number of parallel task for scan/generation
1

Is that wrong? I suddenly suspect we have a winner.

[DogmaDragon]

Parallel Scan/Generation
Number of parallel task for scan/generation
1

That will defintely have an impact. https://docs.stashapp.cc/in-app-manual/configuration/#parallel-scangeneration

[Grendel76]

Thank you! I don't know how it got set to 1, and I didn't know I had created a problem when I did it because... don't laugh, I didn't know it should be faster.

[Gykes]

Generate previews can take a while. MY recommendation for the initial scan is to generate phash only. This will get all of your scenes in then you can generate the rest in the background passively rather than just starting at a mostly blank screen.

You can set the number of parallel tasks to 0 and then Stash will utilize all that it can. Warning on this as it will more than likely pin your CPU until it is done.

Edit: seems like discussions dont update like regular comments do so looks like it was already answered lol

[smith113-p]

Hmm even with 1 I wouldn;t expect it to be that slow when the files haven't changed. Does Tools -> Logs show anything interesting?

[Grendel76]

Hmm even with 1 I wouldn;t expect it to be that slow when the files haven't changed. Does Tools -> Logs show anything interesting?

Not really, no. FFmpeg has a hard time with a few files but we're talking about a handful. I already looked for any phashes that were null in case I had a bunch of corrupt ones and removed those but again we're talking about just like a handful.

When I set the count to 0 it absolutely flies so I'm pretty sure that was everything. I'm cleaning up some big image directories that I don't use anymore before scanning it again so it doesn't waste time on stuff I don't even want but I think that was it.