80% of cpu #645
Comments
|
Its not really doing anything too CPU intensive ever. Mine uses like 2% of Not sure about the auto shutdown either..... let me think about what might
|
|
Here log when heaphone used 80% of cpu : 25-mai-2012 11:24:41 - INFO :: MainThread : Headphones is already up-to-date. 25-mai-2012 16:50:11 - INFO :: MainThread : Headphones is already up-to-date. At 16:50 I restart it. |
|
Okay, so its getting stuck while doing the library scan. It shouldn't be
|
|
Hi, |
|
The actual log file might provide more clues Headphonesdir/logs/headphones.log
|
|
This is headphones.log. Here you can see I restart headphones : root@_:/home/www/vhosts/_************/logs# vim headphones.log 01-juin-2012 09:53:45 - INFO :: MainThread : Headphones is already up-to-date. |
|
The problem with the CPU is back Top : The last log entries : The probleme is here when headphones rescan library. Maybe I make a mistake somewhere in install? |
|
Hello, I also have the same issue. When Headphones auto updates (wanted albums or library), it raise the CPU load to 98% and stays at that level, making everything run very slowly. I must go to the web interface and click on he restart link, and CPU load get back to ~20% again. |
|
It's gotta be getting stuck somewhere, but not producing any error messages. Maybe can you tr running it from the terminal to see if there are any exceptions during the library scan? On Jun 19, 2012, at 11:02 AM, fullbrightreply@reply.github.com wrote:
|
|
I make your test and I force refresh of music but the CPU and memory is ok so it's not when headphones refresh music... |
|
Hello, Thanks for the tests. I'll do further tests and see where this is from. Le 22 juin 2012 23:27, "zoic21" < I make your test and I force refresh of music but the CPU and memory is ok Reply to this email directly or view it on GitHub: |
|
I am getting this also, 80-85% Yesterday i heard cpu fan go into overdrive, looked at running proccess and headphones was as 165% restarted it and it was good again for a bit but still does go up to 80-85% I have 4 GB memory and 2 x 3ghz running on Openmediavault(Debian) and music collection is around 66GB I ran headphones in a terminal and top in another, Updated artists 1-2% (9% max) Scan music directory, bang 83% I love headphones but this is making me think of uninstalling as its causing whole system to slow down. |
|
I also still the problem pending a resolution of the problem I stopped headphones ... |
|
I want to help resolve this so will keep using it for now but without any error report i think it will be hard for anyone to track down what is causing this to happen. What system are you using to run headphones? Other users with this problem should post what system they are using as it may be a system issue and will also help eliminate probable causes. |
|
Are you guys running it from source? I'd like to narrow this down and fix
|
|
I am, git clone then i remove the .git folder to allow updates. Thanks for having a go at getting this sorted. |
|
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 192% thats too high to leave running, will have to disable it for now but will be happy to test any changes you suggest. |
|
No that's crazy high. Let me work on something - thanks for the heads up On Jul 18, 2012, at 4:51 PM, cptjhmillerreply@reply.github.com wrote:
|
|
Same problem, here very high cpu (90% ish) on scanning library. Let me know if you want me to try adding debug etc to test. |
|
I took out the "checking empty filepaths" code, since i thought that might be the culprit but its not..... Maybe i can put together a little author script you guys can run that will basically mimic the library scan but with more logging? |
|
*python |
|
Yep, no problem, happy to run that, or if you'd like me to apply a patch to my install, whatever's easiest for you. |
|
Same here, would be happy to help out in what ever way is needed to track down this error/bug |
|
Cool, lemme write something up. A script is easier since you won't have to keep restarting headphones with changes On Jul 29, 2012, at 7:18 PM, cptjhmillerreply@reply.github.com wrote:
|
|
Sorry guys, posted the debug info in the wrong thread. I wrote up a script that tries to mimic the library scan. It's in the debug branch. To get it, cd into your headphones directory, and run: Depending on the results of that, I'll either tweak the debug script or have a fix ready |
|
I am in Turkey atm but will try it when i return home on Friday. I would try it but dont wont to risk crashing my server. |
|
A possible fix is in the develop branch |
|
No joy I'm afraid. First I tried the devlop branch. I replicated as before: went to Manage screen, and pressed save on the existing library path, top shows one core maxed out almost immediately. Next tried debug.py, and it behaves itself pretty well, cpu not going above 20 usage total. Sorry to be bearer of bad tidings :( |
|
Dumb question but did you restart after switching branches?
|
|
Accessing a sqlite db over a network share is a bad idea on so many levels... but yeah WAL doesn't work without shared memory. Now if you can change the logic so that a full scan doesn't update every single row in the db... the "upserts" in librarysync never have the opportunity to not run a query because the controlValueDict isn't populated from the db data that would be overwritten. So even when nothing changes, an update is always executed that's a good chunk of io that can be eliminated. Another way to reduce load would be to store a signature of the file along with the track in the db -- location + size + mtime is probably enough -- and don't even bother reading the id3 if the file has been scanned before. |
|
Nice one. Will look into it. Cleaning up the code to be more efficient Anyways - do you have any interest in helping me out on this more actively? Thanks!
|
|
I wouldn't mind helping out -- but I've never touched python. Code is code, Tracking mtime for each file should be safe. I can't imagine why a tagging On Tue, Jul 23, 2013 at 12:04 AM, rembo10 notifications@github.com wrote:
|
|
I'm really happy to see that this issue has been resurrected! I forked headphones in hoping that I could set it up with MySQL instead of SQLite to speed up the update process so my CPU doesn't burn for hours at a time, but after 12 hours of reviewing and mucking with the code I'm convinced of a few things: 1.) MySQL is a nice-to-have, but complicates installation for having to include mysql-python - which can get pretty hairy depending on the user's system. Even if you get past that hurdle, there's a major refactoring of the code to migrate SQLite statements to MySQL statements. ("?" becomes "%s" in queries, mass changes for dictionary support and reverse, etc) It's definitely not trivial. [don't mean to thread-jack, I know there's another thread about potential MySQL support] 2.) I'm in the same boat as normzbig. I'd love to help out, but python is pretty new to me. However, there are definitely ways to reduce database I/O and CPU load within the current code framework. We should get a list of potential opportunities going. Checking files against database contents before scanning is a great idea. Other ideas could include the ability to "pause" albums, so albums released years ago aren't always being updated with every update of "active artists". Or somehow pick the one best "release" of an album before committing to the database, since there are sometimes dozens of album variants on musicbrainz (all being committed). It probably sounds like pointless busy-work, but if we could establish a flowchart of all filesystem/musicbrainz/headphones.db I/O, I bet we could knock 80% of the read/write cycles out of the equation. |
|
Awesome. I just don't have enough experience to be able to figure out |
|
Can someone maybe email me a large database they'd been having trouble with? headphones app at gmail thanks |
|
SQLite really should be fine for this purpose. We just need to reduce the rembo10, my db is 75 mb. Not very large, but it's been growing rapidly On Tue, Jul 23, 2013 at 12:40 AM, Justin Evans notifications@github.comwrote:
|
|
Oh yeah. I'll get a fix for that - there needs to be more specialized stuff for various artists. As for the sqlite stuff, do you think it'll be problematic if we just switch to wal behind the scenes? right now journaling is off but in the case that the data dir is on a network share, maybe we can give the option of changing the journaling system in the config? |
|
I'm trying to understand the use case where someone runs their headphones On Tue, Jul 23, 2013 at 10:37 AM, rembo10 notifications@github.com wrote:
|
|
Haha i'm just trying to cover my bases
|
|
Another thing that would go a long way to preventing the system from The only downside I can think of would be that if the server went down, On Tue, Jul 23, 2013 at 11:27 AM, rembo10 notifications@github.com wrote:
|
|
I've managed to successfully run headphones with MySQL instead of SQLite - making the absolute minimum number of code changes across the board. I did this because using phpmyadmin really helps with visualizing the database contents as the program runs to understand what's going on under the hood. I'm importing my library right now, and preliminary finding is that it takes just as long to run through the initial sync as it does with SQLite. The CPU doesn't seem to burn as hot throughout the process, but I could be mis-remembering. The picture included is my CPU load somewhere in the middle of this process. I should point out that I don't consider this out of the ordinary, as I expect that importing and matching 20,000 tracks, and downloading all the associated metadata from musicbrainz and last.fm will take some time on first run. Once this is complete, I'll try the "update active artists" command and see what I see. In a perfect world, a regular update of the library should only take a handful of minutes to complete. One thing that I did notice is that the "have" table gets emptied every time the library is scanned. As mentioned by normzbig, it's probably a good idea to save the directory tree somewhere after each scan with some sort of global "datetime last scanned" variable - then append/replace the "have" table with all files modified since the last scanned datetime. But even that is peanuts (time+CPU-wise) compared to the fact that all active artists/albums get updated wholesale with each "active artist" update or "library scan". Scanning the library into the "have" table is definitely not the bottleneck; artist/album updating is far more intensive. The latter should be avoided wherever possible. To me, there are two relatively easy, near-term "bangs for the buck": |
|
I think youre right on all points. It's so messy. I'm gonna finish up
|
|
My headphones only really hammers my CPU when it goes to update my 119 active artists. It pretty much maxes out my cpu for hours (4+). I suppose i should go and disable some artists if i can to lessen the load. What property determines this update interval? Is it the 'library scan' interval? Should i be seeing this take so long and hurt so much? Note, i'm running my own musicbrainz mirror on the same machine (its resource consumption is almost minimal). |
|
I'm thinking that a good way to resolve this issue is to reduce artist If they're releasing new remixes every month, maybe headphones can look for I'm sure we could come up with a simple formula that, based on the data Thoughts? On Thu, Jul 25, 2013 at 5:15 PM, Kyle Brown notifications@github.comwrote:
|
|
Last I heard was that musicbrainz is going to add in a "last modified"
|
|
Can you guys try the latest version? It has the write-ahead logging mode enabled for the db Thanks |
|
This is just a quick fix solution - working on the heavy db stuff in the meantime |
|
Indexing helps a bit too. I added some query logging and then counted up 3785 headphones.db: SELECT TrackID, ReleaseID, AlbumID from alltracks Then added indexes: CREATE INDEX alltracks_albumtitle on alltracks( albumtitle collate nocase ); CREATE INDEX tracks_albumtitle on tracks( albumtitle collate nocase ); CREATE INDEX have_trackid on have( trackid ); On Wed, Jul 31, 2013 at 6:54 AM, rembo10 notifications@github.com wrote:
|
|
And this one: CREATE INDEX artists_artistid on artists( artistid ); Which is used by the repetitive query to update havetracks. On Mon, Aug 5, 2013 at 7:36 AM, Norm Zbig normzbig1@gmail.com wrote:
|
|
I'll be watching the development on this issue closely, because right now I have shut down headphones since refreshing the library is taking 100% of my CPU for days. In the mean time: would a temporarily solution be to set all my artists to 'paused'? I don't mind if I don't get updates on new released albums. thanks. |
|
Hi rembo10, I just initiated a pull request to address this problem. I believe I’ve come up with an algorithmic solution to CPU-burn (and length of time) when updating active artists. Was: Is: Notes:
Let’s get this bad boy merged! I’ll take a look at the library scanner next. |
|
Thanks @theguardian i've got my own fork and pulled in your changes. I'm in the process of doing an initial refresh now (which you said would take a bit longer). I'll then do another and see how it goes. |
|
What's the status on this issue? Headphones is starting to interfere with other processes (in a sense of using CPU which it shouldn't be using). These processes do not run when the load on the server is bigger then specified. |
|
There's a current pull request to dramatically improve both the Update Active Artists & Library Sync CPU + Time issues. Rembo says this is the next change he wants to roll-in. |
|
I have been using your repo @theguardian -- it's been nice having my cpu back to do other things :) |
|
Thanks @theguardian (and @rembo10). Saw the ETA of two weeks from @rembo10, so will be waiting patienly for it. |
|
I think we can close this issue.. I hope! |
and others
I have two problems with headphones :
What can I do to help you to solve this.
The text was updated successfully, but these errors were encountered: