Slow scan for libraries in rclone

[Gomeyy]

Love everything about mango but these scan times are insane, I haven't seen anyone else posting about this so I would assume its something unique to me but I'm unsure whats causing it. If you need anything from me let me know, I'll get back to you when possible.

[hkalexling]

Hi there, thanks for reaching out. It would be great if you let me know a few things to help diagnose the problem.

• How many entries (or chapters) do you have in those 12 titles?
• Are you running Mango inside some container? (e.g., Docker, WSL, VM, etc.)
• On what kind of device are you running Mango? Can you let me know the CPU and RAM information?
• It might also help if you could set the log level to "debug" so Mango logs more detailed information

[Gomeyy]

Thanks for getting back to me so quick,

• There's 1056 different files in those chapters, using 59 GBs
• No container, using the binary
• A home server running Ubuntu 18.04.5. CPU: i7-7700k @ 4.2GHz RAM: 32 GBs DDR4 3200MHz
• Set the log level, will post a screenshot when its done

[Gomeyy]

[hkalexling]

Hmm, I still don't see how it takes so long. My only guess is that your library is quite large and it might be causing some issues here. My own library is 6GB and the scan takes 800ms. I will test it with a larger library and see if I can reproduce the issue.

[hkalexling]

I just tested it with a library of size 62GB, and the scan finished within one minute, and I ran it in Docker so there's the additional IO overhead.

Where did you get your manga files? Do you think you can send me a typical cbz file from your library so I can inspect and test? You can always email me at email@hkalexling.com. Thanks!

[Gomeyy]

Sent you an email

[hkalexling]

Thanks for the email! You mentioned in the email that your library is in a rclone directory and I think that's the culprit. When scanning, Mango attempts to read the first page of each archive to see if it's corrupted, so in your case it would download every archive. It also creates an info.json file in every subdirectory to store metadata, so there's the additional overhead.

Let me think about it and see if we can make the corruption check optional. As a temporary workaround, I would suggest downloading some of the titles to your server so Mango can scan locally.

[Gomeyy]

Yeah I figured that would be the cause, I have all my media on an rclone directory so I would like to leave it that way in the long run. Take your time on the fix, glad we found the issue though

[v1006945671]

I have the same problem, but I have a lot of comics, about 700g and 4000 copies. The scanning time is 50 minutes. But if you restart the container, it will be scanned again, and the scanning thumbnail will be stuck

[hkalexling]

Hi @v1006945671, just to confirm - are you putting your library in an rclone directory as well?

[v1006945671]

Hi @v1006945671, just to confirm - are you putting your library in an rclone directory as well?

No, I use synology docker and put comics in the library.

[hkalexling]

@v1006945671 I see. This issue is more about Mango's performance under rclone. I guess the title is a bit confusing so I am updating it now. Regarding your issue, I have never tested Mango with such a gigantic library, but I will see what I can do. Thanks for letting me know!

[v1006945671]

@v1006945671 I see. This issue is more about Mango's performance under rclone. I guess the title is a bit confusing so I am updating it now. Regarding your issue, I have never tested Mango with such a gigantic library, but I will see what I can do. Thanks for letting me know!

I don't speak English very well, but I will try my best to feedback bugs. Thanks.

[v1006945671]

Hi @v1006945671, just to confirm - are you putting your library in an rclone directory as well?

I don't know if it's a bug. If there are too many comics, the memory usage will increase unlimited. My machine has 6GB of memory, so I have to limit it.

[hkalexling]

@v1006945671 Are you saying that you are seeing a memory leak? If that's the case, it would be great if you could open a new issue because it's unrelated to the current thread.

[Leeingnyo]

I tested the library that has 338 entries (4.61 GB) in Google Drive, mounted on SSD by rclone (using own project api) with the Mango v0.24.0 (cache enabled).

• first scan : an hour...
• re-scan : 5 seconds max
• first access to /library : 23 seconds
• second access to /library : 5 seconds
• first and seconds access to /book/some-book : 2 seconds
• loading reader for a book that has 93 pages : 25 seconds
• loading images of above book : 15 seconds on average 1600x1097

It's improved but need to be improved more.

• slow first scan
• loading each pages

[hkalexling]

@Leeingnyo Thanks for testing this! The initial scan is slow on rclone because Mango would attempt to decompress each entry to make sure they are readable, which means every file will be downloaded during the first scan. Perhaps we can allow users to disable the check from the config file.

[hkalexling]

#215 would also be a nice workaround for this