New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unusable for large collections #90

Closed
RandolfCarter opened this Issue Sep 10, 2013 · 27 comments

Comments

Projects
None yet
@RandolfCarter

RandolfCarter commented Sep 10, 2013

I know, the app is still in a preview state, but I wanted to give it a try; here's my report that alas, with a big music collection, the app is unusable at the moment.

On first access to the app, the page loading takes forever (actually it never finishes). I heard the hard drive being accessed so I thought I'd let it run. Then the access to the hard drive stopped suddenly. The server log then showed a segmentation fault in some child process; the apache and mysql process still showed high CPU usage. Only a restart of apache helped restore the ownCloud interface. And when trying to go again to the music app, the page loads forever, this time without any hard drive access, again locking up apache and mysql.

Please realize that people will have more than a few songs, and that therefore

  1. Progress report during scan is crucial
  2. The whole scan possibly cannot fit into one script run, or might be aborted in the middle, so there has to be a way to (ideally automatically) resume the scan
  3. The scan not finishing should not break the music app; and even more importantly, should not prevent any other ownCloud access by leaving apache/mysql running at high CPU cycles...

Environment:
Ubuntu Server 12.04
ownCloud 5.0.10
music: tried with latest master (65f4678) and "various-fixes" branch (d75a4ff)
Client: Firefox 23.0 running on Ubuntu 13.04

@butonic

This comment has been minimized.

Member

butonic commented Sep 10, 2013

@icewind1991 I had to find a solution for the 'scan a lot of files' problem when working on search_lucene.

IMHO The files app has the best scanning mechanism because it saves the scan status in the filecache table. Can we find a general solution for scanning mechanisms? How do we mark a file as scanned by search_lucene (I currently join my own status table with the filecache table ... but meh) or by music (or by pictures or by videos)? All of them could extract meta data. I personally would like to explore zend_lucene for searching through that ... (cc @andrewsbrown) ... but I may be biased. Perhaps we can come up with a more general search API that can use either the database or zend lucene ....

Anyway. I propose the following approach: we create a new onscan event/hook that apps that want to extract meta data can register for. Correct me if I am wrong but the files app currently determines the mime type and basic file meta data. music could listen for audio files and then use getid3 to extract more specific meta data. search_lucene likewise. Unfortunately, this naive implementations will cause synchronization to take longer because the file app scans the file on write causing an upload via webdav to wait for all meta extraction to be complete. Therefore we need to separate "file app"-scanning and meta data indexing from each other. This is where background processes come into play. I just don't know yet if adding a separate job for each file makes sense or if just adding a "reindex" job is more elegant. The problem with a "reindex" job is that you don't want to reindex all files but only the ones that are new. In search_lucene I use a status table to keep track of this. Music could use a flag in the songs table but then we are duplicating the mechanism. Single jobs will fill the jobs table quite quickly ...

@icewind1991 @karlitschek @kabum @DeepDiver1975 @bartv2 opinions? ideas? we need a more general solution for this?

@Tronic69

This comment has been minimized.

Tronic69 commented Dec 10, 2013

I can confirm, that it's not usable for large collections and for me it seems to scan the collection every time again, when I open the musics app.

I opened the app and then it shows "Loading ...", but that never changes (not even if an error occurs)!
While the music app (tells me it) is loading my whole owncloud instance is unusable. 23 seconds after opening the music app I tried to access the admin settings to check for an error in the logs, but nothing happened until a PHP error occured and the script was "killed":

[Tue Dec 10 08:12:15 2013] [error] [client 46.5.241.128] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 77 bytes) in /srv/www/owncloud6_RC3/apps/appframework/db/entity.php on line 58, referer: https://oc.vtux.de/index.php/apps/music/

As you can see in the log lines below after nearly 2 minutes (when script exhausted memory) the admin request was processed. So you can see that everything else on the owncloud instance was blocked.

I haven't checked the code, but I think it would be a better idea to do just a rescan for all files where filecache.fileid > music_tracks.file_id (for current user) and then in (background) scan for deleted files which never exists in filecache. This would be unique, too.

Regards,
Tom

[10/Dec/2013:08:10:04 +0100] "GET /cron.php HTTP/1.1" 200 20 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"
[10/Dec/2013:08:10:04 +0100] "GET /index.php/apps/music/api/artists?fulltree=true HTTP/1.1" 500 20 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"

[10/Dec/2013:08:12:16 +0100] "GET /data/htaccesstest.txt HTTP/1.0" 403 1860 "-" "-"
[10/Dec/2013:08:12:17 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 401 291 "-" "-"
[10/Dec/2013:08:10:27 +0100] "GET /index.php/settings/admin HTTP/1.1" 200 6294 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"

@RandolfCarter

This comment has been minimized.

RandolfCarter commented Dec 10, 2013

Any progress on the app in general? Will it be included in OC6? Or will there be no music app in OC6?

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Dec 10, 2013

@RandolfCarter It will not be included into the OC6 package, but can be installed from the app store. Therefore the app could be updated individually. I'm quite busy with university at the moment.

@giovanisp

This comment has been minimized.

giovanisp commented Mar 15, 2014

will this app ever be included back in the OC cloud core ?

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Mar 16, 2014

will this app ever be included back in the OC cloud core ?

cc @karlitschek

@karlitschek

This comment has been minimized.

Member

karlitschek commented Mar 16, 2014

@MorrisJobke We can ship it with the next release if you want. Or we distribute it via the appstore independently. Or we add the option to update shipped apps with versions from the appstore in owncloud 7. But this still have to be implemented. :-)

@LukasReschke

This comment has been minimized.

Member

LukasReschke commented Mar 16, 2014

@karlitschek Could we please discuss privately the process of inclusion of new applications? - Adding new applications to the packages without at least a short security review is something that we should avoid at all costs.

Even a simple grep (which has cost me one second) has shown one potential attack vector to me: JamesHeinrich/getID3@dc85490

The recent problems within the documents app has shown that this is something where we have really place to improve.

@leizh

This comment has been minimized.

Contributor

leizh commented Mar 18, 2014

Hi,

I have worked on my fork on some features to improve large collection support. I will explain them here.

  • #158 (already merged) :
    Register a command to rescan the music files from the owncloud console (wiki). This is usefull when the collection is too large to be scanned by the first request to the music app.
  • #159 (already merged) : Allows to search in the collection in the OwnCloud search area. To implement this, I have redirected the play requests to changes of the hash. I have removed the routing mechanism of angular because it implied to reload all the scopes.
  • leizh/owncloud collection-path (merged in #165) : Allows to filter the collection by a path specified in the user settings (because I don't want not well tagged tracks in my collection). In this branch, I have done huge changes to how the collection is retrieved. A new REST handler 'collection' has been created which retrieves the tree of all the collection in 3 sql statements (instead of 1 + nbArtists + nbAlbums + nbTracks) and with a minimal data amount. Results for 7000 tracks : 21 sec => 6 sec, 3 MB => 1.5 MB.
  • leizh/owncloud collapsed-artists (opened PR #166 - triaging state): Show/hide the albums by click on the artist name. I wanted animations so I upgraded angular to the last stable version (1.2.14) and included the animate-module.

At this point, I stopped to create branches for each features because it was very difficult to merge.

  • 65445ca (merged 6a35f48...601e299): I have more than one image in my album directories. This commit allow to select in first place the images named "front_" and "cover_". I haven't took the time to create a setting entry for this.
  • 1317f93 (merged de85928): I have remove the bindings to the current position. This caused huge lags on the web interface. I supposed that angular tested all the binding updates every second.

Not related to large collections :

I will stop to work on this app for a while. Feel free to merge this features.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Mar 18, 2014

@leizh You are so awesome! I will try to merge as many as possible. Thanks.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Mar 18, 2014

@leizh How can I give you credits for this changes? Feel free to open a PR for https://github.com/owncloud/music/blob/master/AUTHORS

@leizh

This comment has been minimized.

Contributor

leizh commented Mar 18, 2014

(I have updated my comment while you're wrote yours).

This is not very important to me. You can add a "Leizh leizh@free.fr" line in AUTHOR if you want.

Take the time to review the commits, some can not please you and some require settings entries (collapsed-authors and the album cover filenames for example).

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Mar 19, 2014

I added checkmarks to your comment to mark merged stuff

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Mar 20, 2014

@leizh 7 out of the 8 changes are merged into master. The 8th is the collapsing of the artists and maybe needs a polishing but I prepared a PR for this.

@leizh

This comment has been minimized.

Contributor

leizh commented Mar 20, 2014

Thanks for the merging of all of this.

@MorrisJobke MorrisJobke added this to the 0.3 milestone Mar 24, 2014

@HLFH

This comment has been minimized.

HLFH commented Nov 18, 2014

@MorrisJobke Any updates? :)

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Nov 18, 2014

Sadly not ... I currently have not enough time for this :(

@HLFH

This comment has been minimized.

HLFH commented Jul 8, 2015

@MorrisJobke @butonic Is there a way to merge https://github.com/libasys/audios and ownCloud Music? ownCloud Music has a working Ampache backend but is unusable for large collections, Audios ownCloud plugin is usable for large collections but does not have any Ampache backend.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Jul 8, 2015

We plan to join forces.

@pellaeon

This comment has been minimized.

Member

pellaeon commented Nov 18, 2015

@MorrisJobke Any updates? How do you plan to solve problems with large collection? It's really a pain for me, so I want to help.
Currently I find 2 bottlenecks, one is /api/collection and another /api/scan?dry=0, both seems to be caused by slow database queries. Perhaps it could be solved by memcache?

@arpee

This comment has been minimized.

arpee commented Mar 23, 2016

Would also like to bump on this issue.

I get unresponsive pages since the application finished scanning my collection.
(Stuck with a loading gif.)

@tetzlav

This comment has been minimized.

tetzlav commented May 4, 2016

+1
unusable slow with 10000+ songs... :(

@HLFH

This comment has been minimized.

HLFH commented May 4, 2016

@tetzlav You should use Plex or Emby for multimedia (movies, tv shows, music) as @MorrisJobke does. For this moment, it's the better way.

@tenleftfingers

This comment has been minimized.

tenleftfingers commented Jul 4, 2016

You do not need 10k songs to slow down this. I have 3k and it's unusable. Press play on a song and wait minutes to play. UI takes minutes to complete scanning (which it does too often). Gap of several minutes between songs. No way to browse by album. Would love to see this app spread it's wings.

@bee272

This comment has been minimized.

bee272 commented Jul 16, 2016

Great app but....
+1
unusable with 10000+ songs... :(

@paulijar

This comment has been minimized.

Collaborator

paulijar commented May 31, 2017

The PR #564 improving the performance especially with large collections was merged yesterday. If you are willing to test with development version, I would be interested to hear how it works now with large real-life music collections and what is the practical collection size limit for usability. I would recommend using Chrome browser with large music collections.

@paulijar

This comment has been minimized.

Collaborator

paulijar commented Jul 29, 2017

The Music app version 0.4.1 containing #564 has been released to ownCloud marketplace and Nextcloud app store. There's always room for improvement on performance, but IMHO, the app is now quite usable at least with several thousands of tracks. Obviously, it will still probably become unusable if you add e.g. 100k tracks. I'm closing now this very old issue. Feel free to report a new issue if you still find it unusable with your music collection.

@paulijar paulijar closed this Jul 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment