Exporting large files shoots up resource usage and causes a crash

[dagrapealex]

Description of the problem

I was doing my usual save to organize stuff when I suddenly had this problem today. Every session, MyPaint seemed to freeze after a couple of minutes. After while, the tab closes (assuming I didn't start another tab of MyPaint). I tried uninstalling & MyPaint-v2.0.1-alt.AppImage, Linux, & even trying on another version (installing MyPaint-v2.0.1.AppImage & running it). I can always start another session after it freezes but I want to save time here.

Basic system details

MyPaint version: [then MyPaint-v2.0.1-alt.AppImage, currently MyPaint-v2.0.1.AppImage]
Operating system: [PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"]
Desktop environment: [Version 87.0.4280.109 (Official Build) (64-bit)](Chromebook)

Steps to reproduce

1. Start MyPaint
2. [Do your thing until it freezes. I don't know if this is a factor but I was trying to export a lot of layers. At it's max, I tried to export 214 files (unloadable) but I eventually got to the loadable 16 layers per session (33 was loadable but i got to 16, worried it was going to crash).(unloadable=loading bar eventually comes to a halt, ether that, or I just have to wait a couple of days)]
3. [In order to know it's froze, you do step 2 until you get to a point where the interface isn't responding to your clicks & button presses. If it doesn't respond after a minute or 2, it's likely frozen. If you continue waiting after this point, the tab eventually closes. You'll see the 'Gdk-Message: 18:14:58.541: Lost connection to Wayland compositor.' message created in the blink of an eye like the one in the "#### Backtraces or error messages" segment.]

Backtraces or error messages

(A)

No explicit user theme set

GLib-GIO-Message: 17:46:30.355: Using the 'memory' GSettings backend.  Your settings will not be saved or
 shared with other applications.

INFO: mypaint: Installation layout: conventional POSIX-like structure with prefix u'/tmp/.mount_MyPain3KJ
l0k/usr'

INFO: mypaint: Using brushdir path relative to installation-prefix

GLib-GIO-Message: 17:46:33.487: Using the 'memory' GSettings backend.  Your settings will not be saved or
 shared with other applications.

WARNING: gui.userconfig: Failed to load settings file: /home/thegrapealex/.config/mypaint/settings.json

WARNING: gui.userconfig: Failed to load settings: using defaults
INFO: gui.main: No locale setting found, using system locale

Gdk-Message: 18:14:58.541: Lost connection to Wayland compositor.

[jplloyd]

Thanks for the report, I have edited your message to fix some lines that were interpreted as markdown.

I will take a closer look at this on Wednesday, but until then there are some questions that might be relevant:

• Do the freezes only happen when exporting multiple layers, or also when drawing?
• How much RAM does your machine have?
• Does the process eat up a ton of memory when freezing? You can check this by monitoring with top $(pgrep -i mypaint) if you only have one instance of mypaint running.
• Do the freezes happen with the latest alpha appimage? (https://github.com/mypaint/mypaint-appimage/releases/tag/continuous)

[dagrapealex]

Do the freezes only happen when exporting multiple layers, or also when drawing?

It happens after a couple of minutes when I do my thing & that includes when I'm interacting with layers & drawing. For the exporting multiple layers part, it loads until it comes to a halt & does nothing, I haven't waited long enough to see if it usually crashes.

How much RAM does your machine have?

(I used the 'free' command on linux)

Mem:        1437504        5420     1419516           0       12568     1432084
Swap:             0           0           0

Does the process eat up a ton of memory when freezing? You can check this by monitoring with top $(pgrep -i mypaint) if you only have one instance of mypaint running.

(I was in the middle of something, after exporting 13 layers onto the file with 4, & doing what I wanted, it crashed at 3:46 PM (Pacific Time) at long last out of all the other times during the day. I ran your command & this spat out:

top - 15:53:53 up  5:31,  0 users,  load average: 0.02, 0.22, 0.30
Tasks:  32 total,   1 running,  31 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.7 st
MiB Mem :   1403.8 total,   1393.7 free,      5.6 used,      4.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1398.3 avail Mem 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                       
   36 root      20   0   54120   6692   5560 S   0.0   0.5  37:30.19 systemd-journal                               
   44 root      20   0   19720    876      0 S   0.0   0.1   0:02.26 systemd-udevd                                 
   69 root      20   0    9488   1284      4 S   0.0   0.1   0:00.14 dhclient                                      
   70 root      20   0   19520   1376    400 S   0.0   0.1   0:09.47 systemd-logind                                
   72 message+  20   0    9080    680      0 S   0.0   0.0   0:03.52 dbus-daemon                                   
   94 root      20   0    5384    116      0 S   0.0   0.0   0:00.08 agetty                                        
   95 root      20   0   15852    844      0 S   0.0   0.1   0:00.51 sshd                                          
   96 thegrap+  20   0   21576   4868   3064 S   0.0   0.3   0:02.05 systemd                                       
   98 thegrap+  20   0   23080   2116      0 S   0.0   0.1   0:00.00 (sd-pam)                                      
  123 thegrap+  20   0   41024   3476      0 S   0.0   0.2   0:06.06 ld-linux-x86-64                               
  124 thegrap+  20   0   41024   3476      0 S   0.0   0.2   0:06.94 ld-linux-x86-64                               
  125 thegrap+  20   0   15632    324      0 S   0.0   0.0   0:00.55 ld-linux-x86-64                               
  133 thegrap+  20   0   15632    320      0 S   0.0   0.0   0:00.44 ld-linux-x86-64                               
  142 thegrap+  20   0  354996  12688      0 S   0.0   0.9   0:11.03 ld-linux-x86-64                               
  143 thegrap+  20   0  354984  12648      0 S   0.0   0.9   0:03.47 ld-linux-x86-64                               
  198 thegrap+  20   0  837716   1780    344 S   0.0   0.1   0:54.67 ld-linux-x86-64                               
  486 root      20   0  239968   1364      0 S   0.0   0.1   0:01.62 polkitd                                       
  890 thegrap+  20   0   24348   1044    480 S   0.0   0.1   0:00.77 ld-linux-x86-64                               
  893 thegrap+  20   0    7916   3452   1996 S   0.0   0.2   0:00.32 bash                                          
  956 thegrap+  20   0    8840    344      0 S   0.0   0.0   0:00.10 dbus-daemon                                   
  957 thegrap+  20   0  315904    724      8 S   0.0   0.1   0:00.19 at-spi-bus-laun                               
  962 thegrap+  20   0    8840    824    428 S   0.0   0.1   0:00.04 dbus-daemon                                   
  964 thegrap+  20   0  174140   1984   1188 S   0.0   0.1   0:00.14 at-spi2-registr                               
 1222 root      20   0   16936   1112      0 S   0.0   0.1   0:00.43 sshd                                          
 1228 thegrap+  20   0   17076   1848    580 S   0.0   0.1   0:01.40 sshd                                          
 1229 thegrap+  20   0   17224   1348      8 S   0.0   0.1   0:00.71 sshd                                          
 1249 thegrap+  20   0   24348    552      0 S   0.0   0.0   0:00.47 ld-linux-x86-64                               
 1251 thegrap+  20   0    7916   1444      0 S   0.0   0.1   0:00.23 bash                                          
 1273 thegrap+  20   0   24348   8552   7996 S   0.0   0.6   0:00.31 ld-linux-x86-64                               

(This is just the chart frozen in time, it's constantly changing at the time I'm typing this)

Do the freezes happen with the latest alpha appimage? (https://github.com/mypaint/mypaint-appimage/releases/tag/continuous)

I got 'MyPaint-v2.0.1.AppImage' from (https://github.com/mypaint/mypaint/releases/tag/v2.0.1) from clicking on Latest stable release. on(https://github.com/mypaint/mypaint), I'm not sure if the alpha appimages=my appimages

[dagrapealex]

Update (if this is even related): I tried to reload the crashed auto-save of the file that crashed but Mypaint crashed after awhile of waiting.

[dagrapealex]

(I meant the file that crashed at 3:46 PM (Pacific Time))

[odysseywestra]

Okay did you test it with the latest builds that @jplloyd mention? We say that cause that lets us know if this is issue is persistant upstream and helps us identify the commit we need to patch into the stable builds and release a minor point release. Both the alpha and stable images have the same access to your filesystem. Just the alpha is newer than the stable.

[jplloyd]

It happens after a couple of minutes when I do my thing & that includes when I'm interacting with layers & drawing. For the exporting multiple layers part, it loads until it comes to a halt & does nothing, I haven't waited long enough to see if it usually crashes.

Ok, thank you.

How much RAM does your machine have?

(I used the 'free' command on linux)

Mem: 1437504 5420 1419516 0 12568 1432084

1.5GB does usually not go very far when working on projects with large number of layers (depending on the size of those layers), but that's no excuse for not handling low-memory situations better (if that is the root cause).

Does the process eat up a ton of memory when freezing? You can check this by monitoring with top $(pgrep -i mypaint) if you only have one instance of mypaint running.
(I was in the middle of something, after exporting 13 layers onto the file with 4, & doing what I wanted, it crashed at 3:46 PM (Pacific Time) at long last out of all the other times during the day. I ran your command & this spat out:

Sorry, I messed up by omitting a flag (and not testing what I wrote), the command should have been: top -p $(pgrep -i -f 'python.*mypaint') - and again, it assumes that only one mypaint process is running (you can start it at any point to monitor the status of the process).

Do the freezes happen with the latest alpha appimage? (https://github.com/mypaint/mypaint-appimage/releases/tag/continuous)

I got 'MyPaint-v2.0.1.AppImage' from (https://github.com/mypaint/mypaint/releases/tag/v2.0.1) from clicking on Latest stable release. on(https://github.com/mypaint/mypaint), I'm not sure if the alpha appimages=my appimages

The continous alpha appimages are rebuilt when a change is pushed (with some exceptions) so they reflect the latest stage of (published) development. You may notice that the alpha appimages have 2.1.0 in the name, while your stable version is 2.0.1.
There was a change a few months back that I thought could have fixed your crash, but now I'm leaning more towards the limited memory being behind it.

[jplloyd]

I could replicate the freezes on a Debian Buster installation on virtualbox (still can't say for sure that the cause of your freezes is the same):

I set the memory available to the virtual machine to 1.5GB, and created an .ora file with 32 layers at ~3k-ish resolution. I zoomed in and out of the image a bunch, to generate and cache mipmaps for different levels of zoom. Then I drew across a bunch of the layers, trying to cover a lot of tiles, to fill the undo stack. With this I could get up to about 1.1G RES memory used by the process.
At this point I could still save and reopen the image.

Starting firefox and opening a few tabs with normal but inefficient websites (reddit, newspapers, etc) easily pushed the total memory usage to the limit. When trying to save the .ora file under these conditions, either firefox would crash silently (3/4 times, in my tests) or mypaint would after getting stuck for a while. When firefox crashes first, mypaint proceeded without crashing.

[dagrapealex]

I tested the crash on (MyPaint-v2.1.0-alpha-336-g453f4e8-2020-12-31_09.15.AppImage). I exported 24 layers. It froze at around 12:23 PM Pacific Time after awhile.

[dagrapealex]

I was trying to load my crashed save having 25 layers on (MyPaint-v2.0.1.AppImage). It loaded, I saved, then done a lot more stuff until it stopped responding. The tab closed with this message in terminal appearing:
DIR: /tmp/.mount_MyPainTUYbNI
No explicit user theme set
GLib-GIO-Message: 16:07:29.792: Using the 'memory' GSettings backend. Your settings will not be saved or shared wi
th other applications.
INFO: mypaint: Installation layout: conventional POSIX-like structure with prefix u'/tmp/.mount_MyPainTUYbNI/usr'
INFO: mypaint: Using brushdir path relative to installation-prefix
GLib-GIO-Message: 16:07:31.969: Using the 'memory' GSettings backend. Your settings will not be saved or shared wi
th other applications.
INFO: gui.main: No locale setting found, using system locale
INFO: lib.i18n: POSIX: LANG='en_US.UTF-8'
INFO: lib.i18n: POSIX: LANGUAGE=None
INFO: gui.compatibility: Setting mode to 2.x (standard)
INFO: gui.compatibility: Setting default layer type to Pigment
INFO: gui.device: New device 'Wayland Pointer' (GDK_SOURCE_MOUSE, axes:2, class=GdkWaylandDevice, vendor=None, prod
uct=None)
INFO: gui.device: New device 'Wayland Touch' (GDK_SOURCE_TOUCHSCREEN, axes:2, class=GdkWaylandDevice, vendor=None,
product=None)
INFO: gui.document: Initialized background from u'/tmp/.mount_MyPainTUYbNI/usr/share/mypaint/backgrounds/mrmamurk/m
amurk_e_1.png'
WARNING: gui.keyboard: Ignoring keybinding for '/BrushModifierActions/BlendModeMenu'
xkbcommon: ERROR: Key "" added to modifier map for multiple modifiers; Using Mod3, ignoring Lock
INFO: gui.brushmanager: Switching default pigment setting to On
INFO: lib.document: load_ora: u'/home/thegrapealex/.local/share/mypaint/scratchpads/autosave.ora'
INFO: lib.document: 0.431s load_ora total
INFO: gui.filehandling: Loaded scratchpad from u'/home/thegrapealex/.local/share/mypaint/scratchpads/autosave.ora'
/tmp/.mount_MyPainTUYbNI/AppRun: line 57: 496 Killed python $DIR/usr/bin/mypaint "$@"

As for the 'top -p $(pgrep -i -f 'python.*mypaint') part, I'm not sure about it, I typed the command but there's no notable changes:
DIR: /tmp/.mount_MyPainM2YuAo
No explicit user theme set
GLib-GIO-Message: 18:07:51.209: Using the 'memory' GSettings backend. Your settings will not be saved or shared wi
th other applications.
INFO: mypaint: Installation layout: conventional POSIX-like structure with prefix u'/tmp/.mount_MyPainM2YuAo/usr'
INFO: mypaint: Using brushdir path relative to installation-prefix
GLib-GIO-Message: 18:07:53.817: Using the 'memory' GSettings backend. Your settings will not be saved or shared wi
th other applications.
INFO: gui.main: No locale setting found, using system locale
INFO: lib.i18n: POSIX: LANG='en_US.UTF-8'
INFO: lib.i18n: POSIX: LANGUAGE=None
INFO: gui.compatibility: Setting mode to 2.x (standard)
INFO: gui.compatibility: Setting default layer type to Pigment
INFO: gui.device: New device 'Wayland Pointer' (GDK_SOURCE_MOUSE, axes:2, class=GdkWaylandDevice, vendor=None, prod
uct=None)
INFO: gui.device: New device 'Wayland Touch' (GDK_SOURCE_TOUCHSCREEN, axes:2, class=GdkWaylandDevice, vendor=None,
product=None)
INFO: gui.document: Initialized background from u'/tmp/.mount_MyPainM2YuAo/usr/share/mypaint/backgrounds/mrmamurk/m
amurk_e_1.png'
WARNING: gui.keyboard: Ignoring keybinding for '/BrushModifierActions/BlendModeMenu'
xkbcommon: ERROR: Key "" added to modifier map for multiple modifiers; Using Mod3, ignoring Lock
INFO: gui.brushmanager: Switching default pigment setting to On
INFO: lib.document: load_ora: u'/home/thegrapealex/.local/share/mypaint/scratchpads/autosave.ora'
INFO: lib.document: 0.277s load_ora total
INFO: gui.filehandling: Loaded scratchpad from u'/home/thegrapealex/.local/share/mypaint/scratchpads/autosave.ora'
INFO: gui.compatibility: Setting default layer type to Pigment
INFO: gui.autorecover: Recovering AutosaveInfo(path=u'/home/thegrapealex/.cache/mypaint/doc.3Gx2oM/autosave', last_
Gdk-Message: 18:09:59.951: Lost connection to Wayland compositor.

Also, for the command itself, I know I should know how code works in the future but I honestly have no idea what to do
with this command, i tried planning it mid-way when it's running mypaint but the results are above, I also tried some stuff below:
thegrapealex@penguin:$ top -p $(pgrep -i -f 'pythonmypaint')
top: -p requires argument
thegrapealex@penguin:$ top ./MyPaint-v2.0.1.AppImage -p $(pgrep -i -f 'python.*mypaint')
top: unknown option '.'
Usage:
top -hv | -bcEHiOSs1 -d secs -n max -u|U user -p pid(s) -o field -w [cols]
thegrapealex@penguin:~$

I optimised the command into this: 'top $(pgrep -i -f 'pythonmypaint')' & I got this:

top - 18:36:53 up 2:33, 0 users, load average: 0.00, 0.02, 0.12
Tasks: 28 total, 1 running, 27 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.0 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st
MiB Mem : 1403.8 total, 1396.9 free, 5.7 used, 1.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1398.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
967 thegrap+ 20 0 11024 3476 3016 R 0.3 0.2 0:00.14 top
1 root 20 0 169284 4664 2560 S 0.0 0.3 1:32.89 systemd
33 root 20 0 52232 6768 5692 S 0.0 0.5 0:04.32 systemd-journal
45 root 20 0 19720 876 0 S 0.0 0.1 0:00.43 systemd-udevd
55 root 20 0 19392 1872 920 S 0.0 0.1 0:00.67 systemd-logind
56 message+ 20 0 9096 684 0 S 0.0 0.0 0:00.95 dbus-daemon
71 root 20 0 9488 1284 4 S 0.0 0.1 0:00.01 dhclient
93 root 20 0 5384 120 0 S 0.0 0.0 0:00.01 agetty
95 root 20 0 15852 832 0 S 0.0 0.1 0:00.02 sshd
96 thegrap+ 20 0 21512 2080 276 S 0.0 0.1 0:00.55 systemd
98 thegrap+ 20 0 23084 2116 0 S 0.0 0.1 0:00.00 (sd-pam)
124 thegrap+ 20 0 15632 324 0 S 0.0 0.0 0:00.11 ld-linux-x86-64
125 thegrap+ 20 0 40892 3380 4 S 0.0 0.2 0:00.48 ld-linux-x86-64
126 thegrap+ 20 0 40892 3380 4 S 0.0 0.2 0:00.40 ld-linux-x86-64
127 thegrap+ 20 0 15632 312 0 S 0.0 0.0 0:00.10 ld-linux-x86-64
142 thegrap+ 20 0 356012 12664 0 S 0.0 0.9 0:00.62 ld-linux-x86-64
143 thegrap+ 20 0 356016 12660 0 S 0.0 0.9 0:00.68 ld-linux-x86-64
198 thegrap+ 20 0 763988 1372 0 S 0.0 0.1 0:11.09 ld-linux-x86-64
211 root 20 0 16936 1112 0 S 0.0 0.1 0:00.08 sshd
217 thegrap+ 20 0 16936 1120 0 S 0.0 0.1 0:00.03 sshd
218 thegrap+ 20 0 16936 1124 0 S 0.0 0.1 0:00.02 sshd
220 thegrap+ 20 0 24348 1440 876 S 0.0 0.1 0:00.99 ld-linux-x86-64
222 thegrap+ 20 0 7916 4088 2624 S 0.0 0.3 0:00.56 bash
498 thegrap+ 20 0 8840 340 0 S 0.0 0.0 0:00.03 dbus-daemon
499 thegrap+ 20 0 315904 720 8 S 0.0 0.1 0:00.11 at-spi-bus-laun
504 thegrap+ 20 0 8840 1060 664 S 0.0 0.1 0:00.06 dbus-daemon
506 thegrap+ 20 0 174140 792 0 S 0.0 0.1 0:00.09 at-spi2-registr
522 root 20 0 239972 1340 0 S 0.0 0.1 0:00.33 polkitd

(I'm pretty sure I'm doing something wrong) What now?

[dagrapealex]

Ummm... I think this is not supposed to be crossed out, how did it get crossed out?
thegrapealex@penguin:$ top -p $(pgrep -i -f 'pythonmypaint')
top: -p requires argument
thegrapealex@penguin:

[jplloyd]

Sorry, I tend to assume knowledge about commands and such - bad habit.
What I'm interested in is how much memory the mypaint process is using when it freezes - since i suspect you are hitting the memory limit on your machine.

top is a resource monitor - showing information about running processes' resource usage, updated at regular intervals.
top has a flag -p that limits output to one or more processes by specifying process id(s).
pgrep takes a regular expression pattern and returns the identifiers of processes that match the pattern.

Running man top and man pgrep should give you more information.

Since you removed the -p flag in your command, all processes are shown (and since the mypaint process was not running, the pgrep command won't return anything). There are other ways to filter which processes are shown, but this seemed the most straightforward.

Note that the process will not appear in the list if it is not running - so run the command after you start mypaint, but before you do the freeze-inducing things. In the terminal window running top, press e to toggle the memory representation of the active process and E to toggle the representation of the total memory summary (the default ( kibibytes) is smaller than we need, and not so easy to read).

Ummm... I think this is not supposed to be crossed out, how did it get crossed out?

The lines got crossed out because of the tilde characters in your prompt string (gh markdown manual) - place terminal output between two sets of three backticks to avoid formatting issues:

```
Terminal output goes here
```

In short, we are interested in the memory usage of mypaint and the memory available to the system when the freezes occur. The time of the crashes is not important.

[dagrapealex]

DRIFT-VIDEO-2071823-2140528-1610068201517-1.mp4

(I actually recorded the crash! Nice!)

[jplloyd]

Ok, so it's using 99% cpu and quickly allocating quite a bit of memory. What were you doing in the program at the time?

[dagrapealex]

I decided to try & export around 24 layers or just making & polishing animation frames. (assuming my memory is right)