Greedy, crashy, messy overuse of CPU, memory, and disk

[Darklocq]

Keka is seizing virtually all available CPU resources, seems to be doing everything in RAM, and then doesn't clean up after itself when it fails.

I have a 12-core Mac Pro with 40 GB of RAM. Using Keka to compress several large files (e.g. big ISOs) at once has just crashed out my system several times, with "You are out of application memory" errors. Had to force-reboot twice.

Observing Keka's general behavior (I have a CPU usage monitor in my menu bar), I see that Keka will use between 25 and 100% of all available CPU when it is running, despite a bunch of other apps being open at the same time and needing resources. This tends to make the system slow down noticeably, sometimes to an unusable crawl. But Keka doesn't seem to get its job done much if any faster when it's using 100% CPU as when using a more reasonable amount.

When I drag a large file to Keka, and then do it again with another large file, and a third, so there are three Keka compression jobs going on, CPU usage is between 75 and 100% for ten minutes or more, and I'm apt to get "out of application memory" errors, and have to force quit other apps (quickly, so the system doesn't just lock up again). Keka does not seem to recognize when it is running more than one instance and set up any kind of queue or other resource management. It doesn't even seem aware of what resources it has available to it from the system, much less whether anything else needs them. Keka just wants ALL of them, right now.

From what I can tell, Keka is loading the entire file(s) it is going to compress into RAM at once, instead of reading it sequentially off the disk, and is also keeping the entire archive it's writing loaded into RAM while also sequentially writing it out (to the final location, not to a temp file). It's hard to be certain. But I was compressing a 7.4 GB file, a 2.1 GB one and a 3.1 GB one at once, for about 13 GB total, and had way more available RAM than that (not even counting swap space) and still got application memory errors, and a system slow-down that made me feel like I was running in a VM inside a VM. :-) The CPU load was in the 90 to 100% range throughout almost all of this, and did not drop much until Keka was down to processing a single remaining archive and was about 2/3 done with it.

In the process of this not-going-as-planned usage, I needed nominally about 13 GB to write out these files and had more than that (over 19 on that disk), but the first two crapped out with insufficient disk space errors in Keka (I emptied the trash to make a few more GB of room, and the third saved successfully). However, Keka did not remove the incomplete archives it had been writing out for the first two, failed, compressions. It just left them there, with names like "Symphony Orchestra Disc 2.dmg.7z" as if they were complete and successful compression jobs. An app like this needs to use a name like "Symphony Orchestra Disc 2.dmg.7z.kekatemp" while in compression progress, then rename it when complete or remove it if incomplete.

I'm not sure what you want to do with this report. This may be three or more unrelated problems at the code level, but experientially they seem related to me. Keka hogs all available resources, uses them very inefficiently, then fails in a messy way when it has (or thinks it has) run out of resources.

I'm hoping this can be resolved, because Keka's ability to be told things like "always compress, and always use 7z at max compression" is very handy for archiving infrequently-used stuff to save disk space, and also good for packaging stuff for distribution (though I also find that it's pointlessly prompting me for a save location when using an external disk, and doesn't do that when using an internal one, or maybe it has to do with what the filesystem is, like HFS+ versus ExFAT; I guess that's a fourth issue, though not a severe one).

System details: macOS 10.13.6, Mac Pro (Mid 2010), multiple hard drives including SSDs, 40 GB 1333 MHz DDR3, 2 x 2.93 GHz 6-Core Intel Xeon. Keka version: 1.1.20 (3358)

[Darklocq]

PS: I tried to remove the dmg and iso labels from this report, but don't seem to have a means of doing so. The issues reported herein have nothing to do with the format of what is being compressed; the issues also happened when I was compressing other stuff, like a "GarageBand Apple Loops" directory full of a large number of AIF files, along with some other Keka jobs compressing other stuff like another directory with WAV files in it.

[Darklocq]

PPS: By way of contrast, I often run Skyrim in Windows 10 in a VM which has been given half the CPU cores and 1/4 of my system RAM (a VM that I mostly just leave running for easy access to Windows unless I need my cores and RAM back for something big), and I still have more resources available to MacOS than I do when running Keka to compress a few files. Anyway, the above Keka tests were done with that VM shut off. Chrome was running, along with BBEdit, Terminal, my VPN, and a couple of small apps that were idle.

[Darklocq]

Update: I just did a test with everything pretty much idle; a single core, sometimes two, being used to keep the OS going, and not maxing out those cores; nothing really happening besides routine background processes, plus Chrome being open. I dragged a folder with ~2.5 GB of Kontakt samples and instruments onto Keka, and it shot up to 100% CPU usage of all 12 cores within 28 seconds, then dropped back to using 80–95% for three minutes, then back down to about 15–30% for about a minute, then up to 25–40% for the remaining time. Total time about 6.25 minutes, which might be reasonable given the CPU speed and that this was on a spinning hard drive not an SSD. But I'm not sure it's reasonable given how much CPU was used, nor could I see any difference in the progress bar speed between 15% CPU usage and 100%. These times and percentages are probably approximate, given the latency in how most CPU load meters work. The one I'm using is the one provided by iStat Menus 6.40.

I know from experience that if I do 2 or 3 compression jobs like this at once, it'll be 90–100% pretty much the entire time. I'm not going to test another large 4+ set of simultaneous jobs, since I keep crashing out and having to do a hard reboot.

[aonez]

Hi @Darklocq,

First thanks for all the detailed feedback 😊

I think the new Performance settings (already available in the beta version) might be all you need. There you can set queue limits and also limit the per-operation thread usage, so you can always leave some CPU free for other tasks.

"You are out of application memory" errors. Had to force-reboot twice

Never had this issue in a Mac mini 2009 with 8GB of RAM, even when compressing files far bigger than 13GB. Not sure what's happening here in your side. Trying to replicate right now with 32GB of RAM and a 30GB big file, but never got more than 6,4GB of RAM used per operation and each additional operation used less RAM since there was less available. No luck reproducing that "out of memory" dialog doing 6 parallel operations with that big 30GB DMG file. Again, using the queue feature in the upcoming version should solve this issue for you.

doesn't seem to get its job done much if any faster when it's using 100% CPU

For compression (not extraction) the more CPU you use the faster it goes, but doing more than one (specially big) operation at a time might make it feel a slow process anyway. But as you already experienced compression will try to use all available CPU and other operations might become slower.

Keka did not remove the incomplete archives it had been writing

Checking this one for sure, this is not the expected behavior in failed compressions.

I also find that it's pointlessly prompting me for a save location when using an external disk, and doesn't do that when using an internal one

As long as that disk has write support (not NTFS without third party extensions, for example) you can use them without the "save" panel. Just be sure to "Enable external volumes access" in Keka -> Preferences -> File Access. You can also give access to only one particular volume or folder inside that volume. This is a sandbox limitation more than a Keka issue.

Keka does not seem to recognize when it is running more than one instance

Indeed, although multiple instances are enabled on purpose, there's no communication between them. Why do you use the different instances? If it is for having multiple windows with different settings, note that the beta has support for that (File -> New Window).

CPU load meters work. The one I'm using is the one provided by iStat Menus 6.40

I use the bundled Activity Monitor. As you already said CPU usage is directly connected with disk speed, compression in the RAM is what consumes all available/assigned CPU threads while disk write will use far less and even less for a spinning disk.

[gingerbeardman]

"You are out of application memory" errors.

Are you letting the system manage Virtual Memory? The only times I've had that are when I experimented with some low level system options surrounding memory management.

[Darklocq]

On Mon, Dec 2, 2019 at 2:02 AM aONe ***@***.***> wrote: I think the new Performance settings (already available in the beta <https://beta.keka.io> version) might be all you need. There you can set queue limits and also limit the per-operation thread usage, so you can always leave some CPU free for other tasks.

Okay, I'll give that a try.

"You are out of application memory" errors. Had to force-reboot twice Never had this issue in a Mac mini 2009 with 8GB of RAM, even when compressing files far bigger than 13GB. Not sure what's happening here in your side. Trying to replicate right now with 32GB of RAM and a 30GB big file, but never got more than 6,4GB of RAM used per operation and each additional operation used less RAM since there was less available. No luck reproducing that "out of memory" dialog doing 6 parallel operations with that big 30GB DMG file. Again, using the queue feature in the upcoming version should solve this issue for you.

Hopefully. I don't have any theories for why the behavior would be different on this machine, but maybe it won't matter. :-)

Are you letting the system manage Virtual Memory? The only times I've had

that are when I experimented with some low level system options surrounding memory management. Yes. I don't have any weird virtual memory apps. Keka did not remove the incomplete archives it had been writing

Checking this one for sure, this is not the expected behavior in failed compressions.

As I said, writing to a temp file then moving it to the intended name when complete will fix this problem. Well, the app would still need to look for leftover temp files on next start, in case it crashed (or power was lost, or whatever) during the compression write-out. I also find that it's pointlessly prompting me for a save location when

using an external disk, and doesn't do that when using an internal one As long as that disk has write support (not NTFS without third party extensions, for example) you can use them without the "save" panel. Just be sure to "Enable external volumes access" in Keka -> Preferences -> File Access. You can also give access to only one particular volume or folder inside that volume. This is a sandbox limitation more than a Keka issue.

It's an ExFAT drive. Keka's the only app I've run into so far that won't write to it without extra interaction, but I may not have tested it all that much. I've actually disabled most of macOS's paranoia, since this machine is well-firewalled, but I'm no expert on filesystem stuff. Anyway, I had already enabled Keka's access to all volumes. UPDATE: Nope, I missed one, and it was that one. Either that, or Keka loses the access when a drive is unmounted and remounted later. Just tested that, and it did not, so yeah – this one was just operator error on my part. Sorry! Keka does not seem to recognize when it is running more than one instance

Indeed, although multiple instances are enabled on purpose, there's no communication between them. Why do you use the different instances?

If I'm compressing something for 10 minutes, and run into something else I want to compress, then I'm not inclined to wait another 5 or 10 or whatever for the first monster-size job to complete; I just start another and move on. If it is for having multiple windows with different settings, note that the

beta has support for that (File -> New Window).

Didn't have to do with different settings; it was just an additional drag-'n'-drop operation. I use Keka entirely as a drag-'n'-drop 7z maker.

[Darklocq]

Update: The out-of-memory stuff stopped, when using Keka's new-ish options to limit its own resource utilization. It'd probably be wise for Keka to have some of these internal limitations pre-set to reasonable levels.

The non-beta version has thus been more stable for me. However, I've still been avoiding running more than one Keka operation at the same time, due to issues reported above. The "start one 10-minute compression job, then start another one half-way through the first" stuff still seems pretty iffy.