Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Greedy, crashy, messy overuse of CPU, memory, and disk #492

Closed
Darklocq opened this issue Dec 2, 2019 · 14 comments
Closed

Greedy, crashy, messy overuse of CPU, memory, and disk #492

Darklocq opened this issue Dec 2, 2019 · 14 comments
Assignees
Milestone

Comments

@Darklocq
Copy link

@Darklocq Darklocq commented Dec 2, 2019

Keka is seizing virtually all available CPU resources, seems to be doing everything in RAM, and then doesn't clean up after itself when it fails.

I have a 12-core Mac Pro with 40 GB of RAM. Using Keka to compress several large files (e.g. big ISOs) at once has just crashed out my system several times, with "You are out of application memory" errors. Had to force-reboot twice.

Observing Keka's general behavior (I have a CPU usage monitor in my menu bar), I see that Keka will use between 25 and 100% of all available CPU when it is running, despite a bunch of other apps being open at the same time and needing resources. This tends to make the system slow down noticeably, sometimes to an unusable crawl. But Keka doesn't seem to get its job done much if any faster when it's using 100% CPU as when using a more reasonable amount.

When I drag a large file to Keka, and then do it again with another large file, and a third, so there are three Keka compression jobs going on, CPU usage is between 75 and 100% for ten minutes or more, and I'm apt to get "out of application memory" errors, and have to force quit other apps (quickly, so the system doesn't just lock up again). Keka does not seem to recognize when it is running more than one instance and set up any kind of queue or other resource management. It doesn't even seem aware of what resources it has available to it from the system, much less whether anything else needs them. Keka just wants ALL of them, right now.

From what I can tell, Keka is loading the entire file(s) it is going to compress into RAM at once, instead of reading it sequentially off the disk, and is also keeping the entire archive it's writing loaded into RAM while also sequentially writing it out (to the final location, not to a temp file). It's hard to be certain. But I was compressing a 7.4 GB file, a 2.1 GB one and a 3.1 GB one at once, for about 13 GB total, and had way more available RAM than that (not even counting swap space) and still got application memory errors, and a system slow-down that made me feel like I was running in a VM inside a VM. :-) The CPU load was in the 90 to 100% range throughout almost all of this, and did not drop much until Keka was down to processing a single remaining archive and was about 2/3 done with it.

In the process of this not-going-as-planned usage, I needed nominally about 13 GB to write out these files and had more than that (over 19 on that disk), but the first two crapped out with insufficient disk space errors in Keka (I emptied the trash to make a few more GB of room, and the third saved successfully). However, Keka did not remove the incomplete archives it had been writing out for the first two, failed, compressions. It just left them there, with names like "Symphony Orchestra Disc 2.dmg.7z" as if they were complete and successful compression jobs. An app like this needs to use a name like "Symphony Orchestra Disc 2.dmg.7z.kekatemp" while in compression progress, then rename it when complete or remove it if incomplete.

I'm not sure what you want to do with this report. This may be three or more unrelated problems at the code level, but experientially they seem related to me. Keka hogs all available resources, uses them very inefficiently, then fails in a messy way when it has (or thinks it has) run out of resources.

I'm hoping this can be resolved, because Keka's ability to be told things like "always compress, and always use 7z at max compression" is very handy for archiving infrequently-used stuff to save disk space, and also good for packaging stuff for distribution (though I also find that it's pointlessly prompting me for a save location when using an external disk, and doesn't do that when using an internal one, or maybe it has to do with what the filesystem is, like HFS+ versus ExFAT; I guess that's a fourth issue, though not a severe one).

System details: macOS 10.13.6, Mac Pro (Mid 2010), multiple hard drives including SSDs, 40 GB 1333 MHz DDR3, 2 x 2.93 GHz 6-Core Intel Xeon. Keka version: 1.1.20 (3358)

@Darklocq
Copy link
Author

@Darklocq Darklocq commented Dec 2, 2019

PS: I tried to remove the dmg and iso labels from this report, but don't seem to have a means of doing so. The issues reported herein have nothing to do with the format of what is being compressed; the issues also happened when I was compressing other stuff, like a "GarageBand Apple Loops" directory full of a large number of AIF files, along with some other Keka jobs compressing other stuff like another directory with WAV files in it.

@aonez aonez self-assigned this Dec 2, 2019
@aonez aonez added this to the Look at milestone Dec 2, 2019
@Darklocq
Copy link
Author

@Darklocq Darklocq commented Dec 2, 2019

PPS: By way of contrast, I often run Skyrim in Windows 10 in a VM which has been given half the CPU cores and 1/4 of my system RAM (a VM that I mostly just leave running for easy access to Windows unless I need my cores and RAM back for something big), and I still have more resources available to MacOS than I do when running Keka to compress a few files. Anyway, the above Keka tests were done with that VM shut off. Chrome was running, along with BBEdit, Terminal, my VPN, and a couple of small apps that were idle.

@Darklocq
Copy link
Author

@Darklocq Darklocq commented Dec 2, 2019

Update: I just did a test with everything pretty much idle; a single core, sometimes two, being used to keep the OS going, and not maxing out those cores; nothing really happening besides routine background processes, plus Chrome being open. I dragged a folder with ~2.5 GB of Kontakt samples and instruments onto Keka, and it shot up to 100% CPU usage of all 12 cores within 28 seconds, then dropped back to using 80–95% for three minutes, then back down to about 15–30% for about a minute, then up to 25–40% for the remaining time. Total time about 6.25 minutes, which might be reasonable given the CPU speed and that this was on a spinning hard drive not an SSD. But I'm not sure it's reasonable given how much CPU was used, nor could I see any difference in the progress bar speed between 15% CPU usage and 100%. These times and percentages are probably approximate, given the latency in how most CPU load meters work. The one I'm using is the one provided by iStat Menus 6.40.

I know from experience that if I do 2 or 3 compression jobs like this at once, it'll be 90–100% pretty much the entire time. I'm not going to test another large 4+ set of simultaneous jobs, since I keep crashing out and having to do a hard reboot.

@aonez
Copy link
Owner

@aonez aonez commented Dec 2, 2019

Hi @Darklocq,

First thanks for all the detailed feedback 😊

I think the new Performance settings (already available in the beta version) might be all you need. There you can set queue limits and also limit the per-operation thread usage, so you can always leave some CPU free for other tasks.

"You are out of application memory" errors. Had to force-reboot twice

Never had this issue in a Mac mini 2009 with 8GB of RAM, even when compressing files far bigger than 13GB. Not sure what's happening here in your side. Trying to replicate right now with 32GB of RAM and a 30GB big file, but never got more than 6,4GB of RAM used per operation and each additional operation used less RAM since there was less available. No luck reproducing that "out of memory" dialog doing 6 parallel operations with that big 30GB DMG file. Again, using the queue feature in the upcoming version should solve this issue for you.

doesn't seem to get its job done much if any faster when it's using 100% CPU

For compression (not extraction) the more CPU you use the faster it goes, but doing more than one (specially big) operation at a time might make it feel a slow process anyway. But as you already experienced compression will try to use all available CPU and other operations might become slower.

Keka did not remove the incomplete archives it had been writing

Checking this one for sure, this is not the expected behavior in failed compressions.

I also find that it's pointlessly prompting me for a save location when using an external disk, and doesn't do that when using an internal one

As long as that disk has write support (not NTFS without third party extensions, for example) you can use them without the "save" panel. Just be sure to "Enable external volumes access" in Keka -> Preferences -> File Access. You can also give access to only one particular volume or folder inside that volume. This is a sandbox limitation more than a Keka issue.

Keka does not seem to recognize when it is running more than one instance

Indeed, although multiple instances are enabled on purpose, there's no communication between them. Why do you use the different instances? If it is for having multiple windows with different settings, note that the beta has support for that (File -> New Window).

CPU load meters work. The one I'm using is the one provided by iStat Menus 6.40

I use the bundled Activity Monitor. As you already said CPU usage is directly connected with disk speed, compression in the RAM is what consumes all available/assigned CPU threads while disk write will use far less and even less for a spinning disk.

@gingerbeardman
Copy link
Contributor

@gingerbeardman gingerbeardman commented Dec 2, 2019

"You are out of application memory" errors.

Are you letting the system manage Virtual Memory? The only times I've had that are when I experimented with some low level system options surrounding memory management.

@Darklocq
Copy link
Author

@Darklocq Darklocq commented Dec 7, 2019

@Darklocq
Copy link
Author

@Darklocq Darklocq commented Dec 15, 2019

Update: The out-of-memory stuff stopped, when using Keka's new-ish options to limit its own resource utilization. It'd probably be wise for Keka to have some of these internal limitations pre-set to reasonable levels.

The non-beta version has thus been more stable for me. However, I've still been avoiding running more than one Keka operation at the same time, due to issues reported above. The "start one 10-minute compression job, then start another one half-way through the first" stuff still seems pretty iffy.

@aonez
Copy link
Owner

@aonez aonez commented Dec 16, 2019

Thanks for the update on the beta @Darklocq. I'm yet developing features, in fact I'm still calling it a development build rather than beta, since it is still being developed and not fully tested and bug-proofed. The default limitations in that build are 2 operations at a time, with only one compression and two extractions. The more consuming operations are compressions, that's why the harder limit there.

@StratosVX
Copy link

@StratosVX StratosVX commented Feb 7, 2020

I can also attest to Keka eating up all of my system memory. I have files (usually iso's) that I try to individually compress. If I use Keka, I've found it tries to do all of the files at once and I will come back to my computer to find that it crashed because it ran out of system memory. I ended up writing a python script to do this for me so that the files aren't all loaded up at the same time. I look forward to the queue feature in the upcoming release. However, in the current version, I have the preferences set to name compressed files the same as the file name and yet it asks me how I want to name each file. Hopefully that will be fixed when the queue feature is released.

@aonez
Copy link
Owner

@aonez aonez commented Feb 10, 2020

@StratosVX you've checked the 1.2.0 new queue system? I'm sure it will work for you.

However, in the current version, I have the preferences set to name compressed files the same as the file name and yet it asks me how I want to name each file

Check out this wiki page and let me know if that does not solve your issue.

@StratosVX
Copy link

@StratosVX StratosVX commented Feb 10, 2020

I haven't tried it yet. I just downloaded the beta linked above but between work and school I haven't had a chance to test it on my projects yet. Until a couple of days ago, I had only been running the official version.

@StratosVX
Copy link

@StratosVX StratosVX commented Feb 25, 2020

I finally got a chance to try the queue system and it is great. Thank you for adding that functionality. I no longer have to worry about Keka eating all of my memory and crashing my system.

@Darklocq
Copy link
Author

@Darklocq Darklocq commented Feb 29, 2020

I can confirm that everything reported here is resolved and that the queue system works, as of Dev version 1013 of Keka [it says Version 1.2.0-dev (3748) in its Get Info window]. I'll mark this closed unless you want it left open for something I didn't notice. RAM usage, CPU usage, disk usage are all reasonable unless you tell Keka to use too many processes and threads for your system (operator error!). Like any tool of this sort, if you try to compress 5 archives at once, that's going to require resources and really pound your drive. On my 2010 12-core Mac in macOS 10.13, I'm finding that setting everything to 3 (3 jobs, 3 compressions or 3 extractions among those 3 jobs, 3 threads per job, etc.) had perfectly acceptable results, even with Windows running in VMware, and Chrome open with a boatload of tabs, and so on. The queue system also works for when I want to squish a dozen things instead of 3. Issue reported in other thread, of the last beta version going unresponsive when leaving the application "focus", is also fixed. I am happily wallowing in 7zipping up a whole bunch of stuff right now. It may be my imagination, but the actual compression jobs seem faster than they used to, even when done in tandem. Woot!

@aonez
Copy link
Owner

@aonez aonez commented Feb 29, 2020

Nice to hear that @Darklocq! I'll keep on improving performance :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants