[appimaged] Needs too much CPU, slows down Ubuntu 18.04 boot

[probonopd]

appimaged needs too much CPU, slows down Ubuntu 18.04 boot significantly until it shows the GNOME desktop. This is on a machine with 114 AppImages. The same desktop loading slowdown cannot be observed e.g., in Xubuntu 16.04.

I suspect it has to do with the fact that appimaged copies files into directories watched by the system all the time, constantly triggering some update functions from the system (e.g., for mime types, menus, etc.).

Hence we should not copy in the files after each AppImage processed by appimaged in order to save CPU power, but only after the system has quiesced (e.g., no further AppImages have been processed in the last X seconds). I.e., extract the files to a temporary location and move into $HOME everything at once.

@azubieta could you please have a look at this?

A good way to reproduce the issue is on a Ubuntu 18.04 Beta 2 Live ISO with SystemImageKit and the appimaged ExtensionImage, but I suspect the same may be true for other systems as well.

[simoniz0r]

I suspect it has to do with the fact that appimaged copies files into directories watched by the system all the time, constantly triggering some update functions from the system (e.g., for mime types, menus, etc.).

I don't think that's the case at all. If that were true, you'd get the same hang if you installed a while bunch of new applications that created files in the same areas.

This is just kinda the side effect of trying to watch a whole bunch of directories for AppImages instead of having a standard directory where they are stored.

[probonopd]

I don't think that's the case at all. If that were true, you'd get the same hang if you installed a while bunch of new applications that created files in the same areas.

No other application installation method puts hundreds of desktop and mime type files there within seconds.

This is just kinda the side effect of trying to watch a whole bunch of directories for AppImages instead of having a standard directory where they are stored.

So you think watching the directories is causing this high CPU usage? Would be inotify then. So why would this be an issue in Ubuntu 18.04 Beta 2 but not in many other distributions? How could we find out how much inotify actually uses?

[azubieta]

I took a fresh Ubuntu 18.04 Beta 2 install downloaded GVim appimage and make 200 copies of it. Then installed appimaged and restarted the system.
Indeed it was a bit more slow start but as long i could see it was gnome-shell the one consuming a lot of CPU not appimaged.

[probonopd]

Thanks for your test @azubieta - this seems to confirm my hypothesis that by copying in files repeatedly, we are repeatedly triggering some update functions from the system/GNOME Shell, doesn't it?

[TheAssassin]

@probonopd not really. Have you run ps regularly (e.g., every second), dumped the data to files, and then analyzed whether any of these "update" commands were

[azubieta]

I just removed appimaged from autostart and still getting gnome-shell consuming a lot of CPU. I'm starting to suspect that this might not be a bug in our code.

[probonopd]

@TheAssassin all I did so far was observe that Ubuntu 18.04 Beta 2 Live ISO takes forever to show the desktop when using appimaged with many AppImages.

[azubieta]

I have it installed in a virtual machine and it doesn't take forever to show the desktop. It just eats a lot (~25%) of CPU.

[simoniz0r]

No other application installation method puts hundreds of desktop and mime type files there within seconds.

That sounds like your problem, honestly. Running any xdg-* stuff takes quite a bit of CPU especially when you've got a lot of them running at once or running one right after another.

I use another app that has a bug where xdg-mime is called about a dozen times for 10-15 seconds, and it makes my CPU ramp up like crazy when that's happening. You really shouldn't be trying to set a whole bunch of application's info on boot or all at once ever; that's not going to be a good time for the end user.

So, I really doubt that it's anything that's GNOME specific. It's much more likely that you're just abusing xdg-* and causing the start time and CPU use to go nuts because of that.

Y'all should install xubuntu-desktop, lubuntu-desktop, or something lightweight to confirm that it is actually a problem with appimaged and not something GNOME is doing.

I personally have been running 18.04 with LXQt since the daily builds have started being released, and I haven't noticed any extra CPU use on startup (I do not run appimaged, though), so this is either appimaged or GNOME, but I kinda doubt it's GNOME.

[probonopd]

@simoniz0r we didn't say it's a GNOME specific issue. All desktops may be affected more or less. Instead of putting in hundreds of files one after another, appimaged should move them in at once (e.g., after it hasn't processed additional AppImages in the last 3 seconds or so).

[simoniz0r]

Sure, but it'd be a good idea to test it in more than one place (and also pretty easy for you to do; sudo apt install xubuntu-desktop) rather than just assume.

[TheAssassin]

Can't be "high priority" since nobody's taken action since April. Removed the tag therefore.

[probonopd]

Isn't the "high priority" tag signaling that someone should take action? Let's use it like that. So I think this issue genuinely deserves the high-priority tag. While I don't have the time to work on it myself anytime soon, I'd certainly appreciate if someone could.

[TheAssassin]

I think that's the "help wanted" tag, isn't it? But well, if you disagree, feel free to add it again. This issue doesn't seem very critical to me, as it does affect only a very small share of the users.

[TheAssassin]

Migrated to AppImageCommunity/appimaged#19.