-
Notifications
You must be signed in to change notification settings - Fork 199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unusable for large collections #90
Comments
@icewind1991 I had to find a solution for the 'scan a lot of files' problem when working on search_lucene. IMHO The files app has the best scanning mechanism because it saves the scan status in the filecache table. Can we find a general solution for scanning mechanisms? How do we mark a file as scanned by search_lucene (I currently join my own status table with the filecache table ... but meh) or by music (or by pictures or by videos)? All of them could extract meta data. I personally would like to explore zend_lucene for searching through that ... (cc @andrewsbrown) ... but I may be biased. Perhaps we can come up with a more general search API that can use either the database or zend lucene .... Anyway. I propose the following approach: we create a new onscan event/hook that apps that want to extract meta data can register for. Correct me if I am wrong but the files app currently determines the mime type and basic file meta data. music could listen for audio files and then use getid3 to extract more specific meta data. search_lucene likewise. Unfortunately, this naive implementations will cause synchronization to take longer because the file app scans the file on write causing an upload via webdav to wait for all meta extraction to be complete. Therefore we need to separate "file app"-scanning and meta data indexing from each other. This is where background processes come into play. I just don't know yet if adding a separate job for each file makes sense or if just adding a "reindex" job is more elegant. The problem with a "reindex" job is that you don't want to reindex all files but only the ones that are new. In search_lucene I use a status table to keep track of this. Music could use a flag in the songs table but then we are duplicating the mechanism. Single jobs will fill the jobs table quite quickly ... @icewind1991 @karlitschek @kabum @DeepDiver1975 @bartv2 opinions? ideas? we need a more general solution for this? |
I can confirm, that it's not usable for large collections and for me it seems to scan the collection every time again, when I open the musics app. I opened the app and then it shows "Loading ...", but that never changes (not even if an error occurs)! [Tue Dec 10 08:12:15 2013] [error] [client 46.5.241.128] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 77 bytes) in /srv/www/owncloud6_RC3/apps/appframework/db/entity.php on line 58, referer: https://oc.vtux.de/index.php/apps/music/ As you can see in the log lines below after nearly 2 minutes (when script exhausted memory) the admin request was processed. So you can see that everything else on the owncloud instance was blocked. I haven't checked the code, but I think it would be a better idea to do just a rescan for all files where filecache.fileid > music_tracks.file_id (for current user) and then in (background) scan for deleted files which never exists in filecache. This would be unique, too. Regards, [10/Dec/2013:08:10:04 +0100] "GET /cron.php HTTP/1.1" 200 20 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" [10/Dec/2013:08:12:16 +0100] "GET /data/htaccesstest.txt HTTP/1.0" 403 1860 "-" "-" |
Any progress on the app in general? Will it be included in OC6? Or will there be no music app in OC6? |
@RandolfCarter It will not be included into the OC6 package, but can be installed from the app store. Therefore the app could be updated individually. I'm quite busy with university at the moment. |
will this app ever be included back in the OC cloud core ? |
cc @karlitschek |
@MorrisJobke We can ship it with the next release if you want. Or we distribute it via the appstore independently. Or we add the option to update shipped apps with versions from the appstore in owncloud 7. But this still have to be implemented. :-) |
@karlitschek Could we please discuss privately the process of inclusion of new applications? - Adding new applications to the packages without at least a short security review is something that we should avoid at all costs. Even a simple grep (which has cost me one second) has shown one potential attack vector to me: JamesHeinrich/getID3@dc85490 The recent problems within the documents app has shown that this is something where we have really place to improve. |
Hi, I have worked on my fork on some features to improve large collection support. I will explain them here.
At this point, I stopped to create branches for each features because it was very difficult to merge.
Not related to large collections :
I will stop to work on this app for a while. Feel free to merge this features. |
@leizh You are so awesome! I will try to merge as many as possible. Thanks. |
@leizh How can I give you credits for this changes? Feel free to open a PR for https://github.com/owncloud/music/blob/master/AUTHORS |
(I have updated my comment while you're wrote yours). This is not very important to me. You can add a "Leizh leizh@free.fr" line in AUTHOR if you want. Take the time to review the commits, some can not please you and some require settings entries (collapsed-authors and the album cover filenames for example). |
I added checkmarks to your comment to mark merged stuff |
@leizh 7 out of the 8 changes are merged into master. The 8th is the collapsing of the artists and maybe needs a polishing but I prepared a PR for this. |
Thanks for the merging of all of this. |
@MorrisJobke Any updates? :) |
Sadly not ... I currently have not enough time for this :( |
@MorrisJobke @butonic Is there a way to merge https://github.com/libasys/audios and ownCloud Music? ownCloud Music has a working Ampache backend but is unusable for large collections, Audios ownCloud plugin is usable for large collections but does not have any Ampache backend. |
We plan to join forces. |
@MorrisJobke Any updates? How do you plan to solve problems with large collection? It's really a pain for me, so I want to help. |
Would also like to bump on this issue. I get unresponsive pages since the application finished scanning my collection. |
+1 |
@tetzlav You should use Plex or Emby for multimedia (movies, tv shows, music) as @MorrisJobke does. For this moment, it's the better way. |
You do not need 10k songs to slow down this. I have 3k and it's unusable. Press play on a song and wait minutes to play. UI takes minutes to complete scanning (which it does too often). Gap of several minutes between songs. No way to browse by album. Would love to see this app spread it's wings. |
Great app but.... |
The PR #564 improving the performance especially with large collections was merged yesterday. If you are willing to test with development version, I would be interested to hear how it works now with large real-life music collections and what is the practical collection size limit for usability. I would recommend using Chrome browser with large music collections. |
The Music app version 0.4.1 containing #564 has been released to ownCloud marketplace and Nextcloud app store. There's always room for improvement on performance, but IMHO, the app is now quite usable at least with several thousands of tracks. Obviously, it will still probably become unusable if you add e.g. 100k tracks. I'm closing now this very old issue. Feel free to report a new issue if you still find it unusable with your music collection. |
I know, the app is still in a preview state, but I wanted to give it a try; here's my report that alas, with a big music collection, the app is unusable at the moment.
On first access to the app, the page loading takes forever (actually it never finishes). I heard the hard drive being accessed so I thought I'd let it run. Then the access to the hard drive stopped suddenly. The server log then showed a segmentation fault in some child process; the apache and mysql process still showed high CPU usage. Only a restart of apache helped restore the ownCloud interface. And when trying to go again to the music app, the page loads forever, this time without any hard drive access, again locking up apache and mysql.
Please realize that people will have more than a few songs, and that therefore
Environment:
Ubuntu Server 12.04
ownCloud 5.0.10
music: tried with latest master (65f4678) and "various-fixes" branch (d75a4ff)
Client: Firefox 23.0 running on Ubuntu 13.04
The text was updated successfully, but these errors were encountered: