Support for large libraries #186

gelato · 2021-04-09T11:06:14Z

It's a great and lightweight application but does not seem to work with large libraries (6500+ entries).
Loading of such library takes ages and often ends with 502 error page, it's virtually impossible to browse such libraries. Reader still works if you pick first entry from the library, read it till the end and use "Next" button at the end of the page.

hkalexling · 2021-04-09T12:05:28Z

Hi, thanks for reaching out, but please follow the issue template and provide more information.

Loading of such library

Can you define loading? Is the library scan being slow or is the web frontend not responding?

and often ends with 502 error page

Please provide steps to reproduce. What did you do before getting the error? Did you see other error messages in your console?

I tried a library with over 8k files and it works fine.

> find ~/mango/library -type f -iname '*.cbz' -o -iname '*.cbr' | wc -l
8575

There are a lot of factors that can contribute to the slowness you are experiencing. What's the size (file size in GB) of your library? Are you using an HDD? Is your library on a network-attached drive? Are the entries mostly RAR/CBR files?

It would be great if you could provide the extra information so I can reproduce it and pinpoint the issue.

gelato · 2021-04-09T14:15:40Z

Describe the bug
Very slow frontend response time when using large library (6500+ items) often ending with gateway timeout error (504)

To Reproduce
Steps to reproduce the behavior:

Go to Library
Click on library name
Wait
Wait more
More wait...
See error

Expected behavior
Library will open fast and provide view of all elements

Environment (please complete the following information):

OS: Ubuntu 18.04
Browser chrome, safari, firefox
Mango Version v0.21

Library has size of 150GB, it's on HDD(7.2k rpm one), not on nas, on bare filesystem. Entries are CBZ files.

2021/04/09 17:07:02 [error] 1748#1748: *160749 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.13, server: _, request: "GET /mango/book/6623ef5bcacb4c638d691ab354418fbb HTTP/1.1", upstream: "http://127.0.0.1:9003/book/6623ef5bcacb4c638d691ab354418fbb", host: "192.168.1.57", referrer: "https://192.168.1.57/mango/library"

I don't know if mango has any logging options - can't find them...

hkalexling · 2021-04-17T12:57:55Z

Sorry for the long delay!

Click on library name

Mango supports only one library. I assume you mean clicking on one of your manga in your library? May I know how many entries do you have in the said manga?

Library has size of 150GB, it's on HDD(7.2k rpm one)

This might be the reason why I am not experiencing the slowness. My library is smaller in size and I am using an SSD. Will try to test it on a HDD and see how it goes.

hkalexling · 2021-04-17T13:00:37Z

Are you putting all your 6k+ cbz files under one folder? If so that is probably the reason it's slow. Please try to organize your library as recommended in the README.

gelato · 2021-04-19T20:21:21Z

My library is already organized and still that's 6500 entries (i can't place different mangas in one folder to shorten the overall list of folders). I've increased nginx proxy timeout to 3000 and that provided me the ability to actually read something, but i still have to wait more than a minute before all entries are shown in the library when it loads.

JakeShirley · 2021-05-02T05:22:35Z

I am experiencing something similar. My collection is around ~1TB total and 1000s of directories. Trying to use Chunky's (iPad reader app) OPDS actually times out completely, where at the web just takes a minute to load.

hkalexling · 2021-05-04T07:28:12Z

@JakeShirley Wow, you've got a huge library! Currently, Mango stores metadata to the info.json files in your library, and thus loading a huge folder requires heavy disk I/O and a very long time. #148 should solve this, but many of the components would need to be rewritten so it could take a while.

I am currently working on an official react native app for Mango (still a private repository). I plan to return to #148 once the app has the basic functionalities ready.

AnnoyedDev · 2021-06-19T11:59:04Z

50 titles, 3 000 chapter for all of them (Webtoon) and its took a minute to scan it. everytime i start the application, instead of having the library back, mango seem to not check the database and i have to wait the scan is finished again to access it.

The applpication is very light, but definitely need optimization for even a very small library

(actually i just can't scan my full library even hour and hour waiting because its just do not scan fast enought) :/

EDIT: need a minuter to scan on a SSD and a high end computer. Impossible to run it on a slow or a low end computer neither on HDD.

Leeingnyo · 2021-08-28T17:52:28Z

For a reference, here are my environments:

59 files on a HDD at a WSL2 (~2.4GB) took 5 seconds to scan (server 1)
800 files on the same HDD at a WSL2 (~70GB) took 2 minutes to scan (server 2)
12,000 files on the same HDD at a WSL2 (~500GB) took 30 minutes to scan (server 2 after adding large libraries for a test)
500 files on a SSD (SATA3) at a native ubuntu (~17GB) took 500ms (development environment)

more detail, the Mango runs on Windows docker (WSL2 based engine, Windows installed on SATA3 SSD). the entries are on another HDD. I symlinked /mnt/d/path/to/library (windows' side files) to /mango/library (WSL2's side FS), which would cause slower scanning performance than WSL1 or native OS.

There are two slow factors, scanning files and loading pages.

I worked to improve to load pages and it seems good (not enough tested yet and some jobs left). It would resolve timeout errors, slow loading for a massive entries in one title.

We couldn't make it fast that scanning large library on HDDs because of its characteristics. However, it is necessary reading all files in the library at least once.

everytime i start the application, instead of having the library back, mango seem to not check the database and i have to wait the scan is finished again to access it.

I agree this is annoying! We could serialize scanned titles to a file, and restore them when the app restarted and inspect / update them one by one. or could skip scanning some files that has a same signature before

hkalexling · 2021-08-29T11:30:35Z

We could serialize scanned titles to a file, and restore them when the app restarted and inspect / update them one by one. or could skip scanning some files that has a same signature before

These are actually great idea! I think we can even combine the two to further speed things up. I can try this out next week, but @Leeingnyo let me know if you are interested in doing it yourself :D

Leeingnyo · 2021-08-30T04:23:26Z

@hkalexling haha I want to do but I think I'll be busy a few weeks. I would focus to finish improving loading pages (config cache size etc.).

hkalexling · 2021-08-30T05:07:15Z

@Leeingnyo No worries please take your time :P I will work on other issues first.

Leeingnyo · 2021-09-19T10:19:20Z

Hi guys! Could you try the pre-released version 0.24.0, please? It would be helpful! Let me know if your issue resolved

hkalexling · 2021-09-19T12:03:23Z

@zFerry98 it would be great if you could give rc/0.24.0 a try too! I remember you have a big library, and the new changes should help speed things up.

VegethB · 2021-09-19T12:35:18Z

@zFerry98 it would be great if you could give rc/0.24.0 a try too! I remember you have a big library, and the new changes should help speed things up.

has been in testing since last night 👍
I can no longer speak for Hyper-V + Docker, but on WSL1 it is

Log:

              _|      _|
              _|_|  _|_|    _|_|_|  _|_|_|      _|_|_|    _|_|
              _|  _|  _|  _|    _|  _|    _|  _|    _|  _|    _|
              _|      _|  _|    _|  _|    _|  _|    _|  _|    _|
              _|      _|    _|_|_|  _|    _|    _|_|_|    _|_|
                                                    _|
                                                _|_|


Mango - Manga Server and Web Reader. Version 0.23.0

[INFO]    2021/09/18 15:40:41 | Starting thumbnail generation
[INFO]    2021/09/18 15:40:41 | Thumbnail generation finished
[INFO]    2021/09/18 15:44:44 | Scanned 168 titles in 303231.2662ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[ERROR]   2021/09/18 15:54:42 | No part with name `file` found
[ERROR]   2021/09/18 15:55:59 | No part with name `file` found
[ERROR]   2021/09/18 15:59:33 | No part with name `file` found
[ERROR]   2021/09/18 15:59:44 | No part with name `file` found
[ERROR]   2021/09/18 15:59:57 | No part with name `file` found
[ERROR]   2021/09/18 16:02:40 | No part with name `file` found
[ERROR]   2021/09/18 16:02:52 | No part with name `file` found
[ERROR]   2021/09/18 16:03:04 | No part with name `file` found
[ERROR]   2021/09/18 16:03:17 | No part with name `file` found
[ERROR]   2021/09/18 16:03:26 | No part with name `file` found
[ERROR]   2021/09/18 16:03:36 | No part with name `file` found
[ERROR]   2021/09/18 16:03:47 | No part with name `file` found
[ERROR]   2021/09/18 16:03:57 | No part with name `file` found
[ERROR]   2021/09/18 16:04:08 | No part with name `file` found
[ERROR]   2021/09/18 16:04:18 | No part with name `file` found
[ERROR]   2021/09/18 16:04:28 | No part with name `file` found
[ERROR]   2021/09/18 16:04:39 | No part with name `file` found
[ERROR]   2021/09/18 16:04:52 | No part with name `file` found
[ERROR]   2021/09/18 16:05:01 | No part with name `file` found
[ERROR]   2021/09/18 16:05:14 | No part with name `file` found
[ERROR]   2021/09/18 16:05:29 | No part with name `file` found
[ERROR]   2021/09/18 16:05:42 | No part with name `file` found
[ERROR]   2021/09/18 16:05:55 | No part with name `file` found
[ERROR]   2021/09/18 16:06:08 | No part with name `file` found
[ERROR]   2021/09/18 16:06:23 | No part with name `file` found
[ERROR]   2021/09/18 16:06:32 | No part with name `file` found
[ERROR]   2021/09/18 16:06:45 | No part with name `file` found
[ERROR]   2021/09/18 16:06:56 | No part with name `file` found
[ERROR]   2021/09/18 16:07:13 | No part with name `file` found
[ERROR]   2021/09/18 16:07:24 | No part with name `file` found
[ERROR]   2021/09/18 16:07:36 | No part with name `file` found
[INFO]    2021/09/18 19:44:50 | Scanned 168 titles in 5405.8462ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/18 23:44:52 | Scanned 168 titles in 2057.6374ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/19 03:44:54 | Scanned 168 titles in 2120.7041ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/19 07:44:56 | Scanned 168 titles in 2119.1843ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/19 11:44:58 | Scanned 168 titles in 2167.9803ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/19 14:17:19 | Scanned 168 titles in 2161.8406ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
[INFO]    2021/09/19 14:17:26 | Scanned 168 titles in 2322.7872ms
Unhandled exception in spawn: Error opening file with mode 'w': '/home/uadmin/mango/library.yml.gz': No such file or directory (File::NotFoundError)
  from /usr/share/crystal/src/crystal/system/unix/file.cr:11:7 in 'open'
  from /usr/share/crystal/src/file.cr:112:5 in '->'
  from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???

[INFO] 2021/09/18 15:44:44 | Scanned 168 titles in 303231.2662ms first scan

[INFO] 2021/09/19 14:17:19 | Scanned 168 titles in 2161.8406ms manual scan from admin page (now)

[INFO] 2021/09/19 14:17:26 | Scanned 168 titles in 2322.7872ms manual scan from admin page (now)

I now notice these errors... 🤔

What have I done:

Off Mango;
Added the new build (Build #392);
Added to the configuration file the parameter: cache_enabled: true ;
Server started;

What I've noticed regardless of whether it's working or not:

Rocket-cannon in loading the library and the covers (even if sometimes they take a few seconds to appear);
Rocket-cannon in loading the library on Tachiyomi (ONLY FOR SERIES WITHOUT SUB-FOLDERS, those don't even load the cover). At the moment it does not load the series that have even only 1 subfolder (not even the covers);

Now I have to understand the library.yml.gz': No such file or directory error if it is due to some weird permission on WSL1 or if it is due to something else.
For the error "[ERROR] 2021/09/18 15:54:42 | No part with name 'file' found" it is also present from previous versions (I think from 21) but I have never tried to investigate because it is "random", I think it's due to some strange name in some file inside the .cbz.

For better or for worse he is already a progressone is version 👍🔝

PNG mango log

Leeingnyo · 2021-09-19T13:52:50Z

@zFerry98 Thanks for testing! You can set library_cache_path in the config to resolve that permission error, I think :)

VegethB · 2021-09-19T14:17:57Z

@hkalexling @Leeingnyo

👍🚀

Added the path of the file "library.yml.gz" in the configuration (you must also add the name + extension "library.yml.gz" otherwise it tells you that the path entered is a path and not a file 🙃).
The first run has been doing its mega scan for ages, after which it created the "library.yml.gz" file and subsequent scans are almost instantaneous.
I shut down the server, started ... 2 seconds and had loaded the library 🔝.
Otherwise it seems to work as usual.

hkalexling mentioned this issue Aug 5, 2021

[Question] speed up the loading of the home #209

Closed

Leeingnyo mentioned this issue Aug 30, 2021

Improve loading pages (library, titles) #222

Merged

hkalexling mentioned this issue Sep 12, 2021

[Feature Request] Tachiyomi Support #33

Open

Leeingnyo mentioned this issue Sep 12, 2021

Reuse scanned titles on boot and scanning #229

Merged

hkalexling mentioned this issue Sep 18, 2021

v0.24.0 #232

Merged

hkalexling closed this as completed in #232 Sep 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for large libraries #186

Support for large libraries #186

gelato commented Apr 9, 2021

hkalexling commented Apr 9, 2021

gelato commented Apr 9, 2021

hkalexling commented Apr 17, 2021

hkalexling commented Apr 17, 2021

gelato commented Apr 19, 2021

JakeShirley commented May 2, 2021

hkalexling commented May 4, 2021

AnnoyedDev commented Jun 19, 2021 •

edited

Loading

Leeingnyo commented Aug 28, 2021 •

edited

Loading

hkalexling commented Aug 29, 2021

Leeingnyo commented Aug 30, 2021 •

edited

Loading

hkalexling commented Aug 30, 2021

Leeingnyo commented Sep 19, 2021

hkalexling commented Sep 19, 2021

VegethB commented Sep 19, 2021

Leeingnyo commented Sep 19, 2021

VegethB commented Sep 19, 2021

Support for large libraries #186

Support for large libraries #186

Comments

gelato commented Apr 9, 2021

hkalexling commented Apr 9, 2021

gelato commented Apr 9, 2021

hkalexling commented Apr 17, 2021

hkalexling commented Apr 17, 2021

gelato commented Apr 19, 2021

JakeShirley commented May 2, 2021

hkalexling commented May 4, 2021

AnnoyedDev commented Jun 19, 2021 • edited Loading

Leeingnyo commented Aug 28, 2021 • edited Loading

hkalexling commented Aug 29, 2021

Leeingnyo commented Aug 30, 2021 • edited Loading

hkalexling commented Aug 30, 2021

Leeingnyo commented Sep 19, 2021

hkalexling commented Sep 19, 2021

VegethB commented Sep 19, 2021

What have I done:

What I've noticed regardless of whether it's working or not:

Leeingnyo commented Sep 19, 2021

VegethB commented Sep 19, 2021

AnnoyedDev commented Jun 19, 2021 •

edited

Loading

Leeingnyo commented Aug 28, 2021 •

edited

Loading

Leeingnyo commented Aug 30, 2021 •

edited

Loading