Greedy, crashy, messy overuse of CPU, memory, and disk #492
Comments
|
PS: I tried to remove the dmg and iso labels from this report, but don't seem to have a means of doing so. The issues reported herein have nothing to do with the format of what is being compressed; the issues also happened when I was compressing other stuff, like a "GarageBand Apple Loops" directory full of a large number of AIF files, along with some other Keka jobs compressing other stuff like another directory with WAV files in it. |
|
PPS: By way of contrast, I often run Skyrim in Windows 10 in a VM which has been given half the CPU cores and 1/4 of my system RAM (a VM that I mostly just leave running for easy access to Windows unless I need my cores and RAM back for something big), and I still have more resources available to MacOS than I do when running Keka to compress a few files. Anyway, the above Keka tests were done with that VM shut off. Chrome was running, along with BBEdit, Terminal, my VPN, and a couple of small apps that were idle. |
|
Update: I just did a test with everything pretty much idle; a single core, sometimes two, being used to keep the OS going, and not maxing out those cores; nothing really happening besides routine background processes, plus Chrome being open. I dragged a folder with ~2.5 GB of Kontakt samples and instruments onto Keka, and it shot up to 100% CPU usage of all 12 cores within 28 seconds, then dropped back to using 80–95% for three minutes, then back down to about 15–30% for about a minute, then up to 25–40% for the remaining time. Total time about 6.25 minutes, which might be reasonable given the CPU speed and that this was on a spinning hard drive not an SSD. But I'm not sure it's reasonable given how much CPU was used, nor could I see any difference in the progress bar speed between 15% CPU usage and 100%. These times and percentages are probably approximate, given the latency in how most CPU load meters work. The one I'm using is the one provided by iStat Menus 6.40. I know from experience that if I do 2 or 3 compression jobs like this at once, it'll be 90–100% pretty much the entire time. I'm not going to test another large 4+ set of simultaneous jobs, since I keep crashing out and having to do a hard reboot. |
|
Hi @Darklocq, First thanks for all the detailed feedback I think the new Performance settings (already available in the beta version) might be all you need. There you can set queue limits and also limit the per-operation thread usage, so you can always leave some CPU free for other tasks.
Never had this issue in a Mac mini 2009 with 8GB of RAM, even when compressing files far bigger than 13GB. Not sure what's happening here in your side. Trying to replicate right now with 32GB of RAM and a 30GB big file, but never got more than 6,4GB of RAM used per operation and each additional operation used less RAM since there was less available. No luck reproducing that "out of memory" dialog doing 6 parallel operations with that big 30GB DMG file. Again, using the queue feature in the upcoming version should solve this issue for you.
For compression (not extraction) the more CPU you use the faster it goes, but doing more than one (specially big) operation at a time might make it feel a slow process anyway. But as you already experienced compression will try to use all available CPU and other operations might become slower.
Checking this one for sure, this is not the expected behavior in failed compressions.
As long as that disk has write support (not NTFS without third party extensions, for example) you can use them without the "save" panel. Just be sure to "Enable external volumes access" in Keka -> Preferences -> File Access. You can also give access to only one particular volume or folder inside that volume. This is a sandbox limitation more than a Keka issue.
Indeed, although multiple instances are enabled on purpose, there's no communication between them. Why do you use the different instances? If it is for having multiple windows with different settings, note that the beta has support for that (File -> New Window).
I use the bundled Activity Monitor. As you already said CPU usage is directly connected with disk speed, compression in the RAM is what consumes all available/assigned CPU threads while disk write will use far less and even less for a spinning disk. |
Are you letting the system manage Virtual Memory? The only times I've had that are when I experimented with some low level system options surrounding memory management. |
|
On Mon, Dec 2, 2019 at 2:02 AM aONe ***@***.***> wrote:
I think the new Performance settings (already available in the beta
<https://beta.keka.io> version) might be all you need. There you can set
queue limits and also limit the per-operation thread usage, so you can
always leave some CPU free for other tasks.
Okay, I'll give that a try.
"You are out of application memory" errors. Had to force-reboot twice
Never had this issue in a Mac mini 2009 with 8GB of RAM, even when
compressing files far bigger than 13GB. Not sure what's happening here in
your side. Trying to replicate right now with 32GB of RAM and a 30GB big
file, but never got more than 6,4GB of RAM used per operation and each
additional operation used less RAM since there was less available. No luck
reproducing that "out of memory" dialog doing 6 parallel operations with
that big 30GB DMG file. Again, using the queue feature in the upcoming
version should solve this issue for you.
Hopefully. I don't have any theories for why the behavior would be
different on this machine, but maybe it won't matter. :-)
Are you letting the system manage Virtual Memory? The only times I've had
that are when I experimented with some low level system options surrounding
memory management.
Yes. I don't have any weird virtual memory apps.
Keka did not remove the incomplete archives it had been writing
Checking this one for sure, this is not the expected behavior in failed
compressions.
As I said, writing to a temp file then moving it to the intended name when
complete will fix this problem. Well, the app would still need to look for
leftover temp files on next start, in case it crashed (or power was lost,
or whatever) during the compression write-out.
I also find that it's pointlessly prompting me for a save location when
using an external disk, and doesn't do that when using an internal one
As long as that disk has write support (not NTFS without third party
extensions, for example) you can use them without the "save" panel. Just be
sure to "Enable external volumes access" in Keka -> Preferences -> File
Access. You can also give access to only one particular volume or folder
inside that volume. This is a sandbox limitation more than a Keka issue.
It's an ExFAT drive. Keka's the only app I've run into so far that won't
write to it without extra interaction, but I may not have tested it all
that much. I've actually disabled most of macOS's paranoia, since this
machine is well-firewalled, but I'm no expert on filesystem stuff. Anyway,
I had already enabled Keka's access to all volumes.
UPDATE: Nope, I missed one, and it was that one. Either that, or Keka
loses the access when a drive is unmounted and remounted later. Just
tested that, and it did not, so yeah – this one was just operator error on
my part. Sorry!
Keka does not seem to recognize when it is running more than one instance
Indeed, although multiple instances are enabled on purpose, there's no
communication between them. Why do you use the different instances?
If I'm compressing something for 10 minutes, and run into something else I
want to compress, then I'm not inclined to wait another 5 or 10 or whatever
for the first monster-size job to complete; I just start another and move
on.
If it is for having multiple windows with different settings, note that the
beta has support for that (File -> New Window).
Didn't have to do with different settings; it was just an additional
drag-'n'-drop operation. I use Keka entirely as a drag-'n'-drop 7z maker.
|
|
Update: The out-of-memory stuff stopped, when using Keka's new-ish options to limit its own resource utilization. It'd probably be wise for Keka to have some of these internal limitations pre-set to reasonable levels. The non-beta version has thus been more stable for me. However, I've still been avoiding running more than one Keka operation at the same time, due to issues reported above. The "start one 10-minute compression job, then start another one half-way through the first" stuff still seems pretty iffy. |
|
Thanks for the update on the beta @Darklocq. I'm yet developing features, in fact I'm still calling it a development build rather than beta, since it is still being developed and not fully tested and bug-proofed. The default limitations in that build are 2 operations at a time, with only one compression and two extractions. The more consuming operations are compressions, that's why the harder limit there. |
|
I can also attest to Keka eating up all of my system memory. I have files (usually iso's) that I try to individually compress. If I use Keka, I've found it tries to do all of the files at once and I will come back to my computer to find that it crashed because it ran out of system memory. I ended up writing a python script to do this for me so that the files aren't all loaded up at the same time. I look forward to the queue feature in the upcoming release. However, in the current version, I have the preferences set to name compressed files the same as the file name and yet it asks me how I want to name each file. Hopefully that will be fixed when the queue feature is released. |
|
@StratosVX you've checked the 1.2.0 new queue system? I'm sure it will work for you.
Check out this wiki page and let me know if that does not solve your issue. |
|
I haven't tried it yet. I just downloaded the beta linked above but between work and school I haven't had a chance to test it on my projects yet. Until a couple of days ago, I had only been running the official version. |
|
I finally got a chance to try the queue system and it is great. Thank you for adding that functionality. I no longer have to worry about Keka eating all of my memory and crashing my system. |
|
I can confirm that everything reported here is resolved and that the queue system works, as of Dev version 1013 of Keka [it says Version 1.2.0-dev (3748) in its Get Info window]. I'll mark this closed unless you want it left open for something I didn't notice. RAM usage, CPU usage, disk usage are all reasonable unless you tell Keka to use too many processes and threads for your system (operator error!). Like any tool of this sort, if you try to compress 5 archives at once, that's going to require resources and really pound your drive. On my 2010 12-core Mac in macOS 10.13, I'm finding that setting everything to 3 (3 jobs, 3 compressions or 3 extractions among those 3 jobs, 3 threads per job, etc.) had perfectly acceptable results, even with Windows running in VMware, and Chrome open with a boatload of tabs, and so on. The queue system also works for when I want to squish a dozen things instead of 3. Issue reported in other thread, of the last beta version going unresponsive when leaving the application "focus", is also fixed. I am happily wallowing in 7zipping up a whole bunch of stuff right now. It may be my imagination, but the actual compression jobs seem faster than they used to, even when done in tandem. Woot! |
|
Nice to hear that @Darklocq! I'll keep on improving performance :) |
Keka is seizing virtually all available CPU resources, seems to be doing everything in RAM, and then doesn't clean up after itself when it fails.
I have a 12-core Mac Pro with 40 GB of RAM. Using Keka to compress several large files (e.g. big ISOs) at once has just crashed out my system several times, with "You are out of application memory" errors. Had to force-reboot twice.
Observing Keka's general behavior (I have a CPU usage monitor in my menu bar), I see that Keka will use between 25 and 100% of all available CPU when it is running, despite a bunch of other apps being open at the same time and needing resources. This tends to make the system slow down noticeably, sometimes to an unusable crawl. But Keka doesn't seem to get its job done much if any faster when it's using 100% CPU as when using a more reasonable amount.
When I drag a large file to Keka, and then do it again with another large file, and a third, so there are three Keka compression jobs going on, CPU usage is between 75 and 100% for ten minutes or more, and I'm apt to get "out of application memory" errors, and have to force quit other apps (quickly, so the system doesn't just lock up again). Keka does not seem to recognize when it is running more than one instance and set up any kind of queue or other resource management. It doesn't even seem aware of what resources it has available to it from the system, much less whether anything else needs them. Keka just wants ALL of them, right now.
From what I can tell, Keka is loading the entire file(s) it is going to compress into RAM at once, instead of reading it sequentially off the disk, and is also keeping the entire archive it's writing loaded into RAM while also sequentially writing it out (to the final location, not to a temp file). It's hard to be certain. But I was compressing a 7.4 GB file, a 2.1 GB one and a 3.1 GB one at once, for about 13 GB total, and had way more available RAM than that (not even counting swap space) and still got application memory errors, and a system slow-down that made me feel like I was running in a VM inside a VM. :-) The CPU load was in the 90 to 100% range throughout almost all of this, and did not drop much until Keka was down to processing a single remaining archive and was about 2/3 done with it.
In the process of this not-going-as-planned usage, I needed nominally about 13 GB to write out these files and had more than that (over 19 on that disk), but the first two crapped out with insufficient disk space errors in Keka (I emptied the trash to make a few more GB of room, and the third saved successfully). However, Keka did not remove the incomplete archives it had been writing out for the first two, failed, compressions. It just left them there, with names like "Symphony Orchestra Disc 2.dmg.7z" as if they were complete and successful compression jobs. An app like this needs to use a name like "Symphony Orchestra Disc 2.dmg.7z.kekatemp" while in compression progress, then rename it when complete or remove it if incomplete.
I'm not sure what you want to do with this report. This may be three or more unrelated problems at the code level, but experientially they seem related to me. Keka hogs all available resources, uses them very inefficiently, then fails in a messy way when it has (or thinks it has) run out of resources.
I'm hoping this can be resolved, because Keka's ability to be told things like "always compress, and always use 7z at max compression" is very handy for archiving infrequently-used stuff to save disk space, and also good for packaging stuff for distribution (though I also find that it's pointlessly prompting me for a save location when using an external disk, and doesn't do that when using an internal one, or maybe it has to do with what the filesystem is, like HFS+ versus ExFAT; I guess that's a fourth issue, though not a severe one).
System details: macOS 10.13.6, Mac Pro (Mid 2010), multiple hard drives including SSDs, 40 GB 1333 MHz DDR3, 2 x 2.93 GHz 6-Core Intel Xeon. Keka version: 1.1.20 (3358)
The text was updated successfully, but these errors were encountered: