Skip to content

Memory leak observed in v3.1.3 #553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sgreadly opened this issue Aug 19, 2020 · 8 comments
Closed

Memory leak observed in v3.1.3 #553

sgreadly opened this issue Aug 19, 2020 · 8 comments
Labels

Comments

@sgreadly
Copy link

sgreadly commented Aug 19, 2020

Hi all,

I noticed on my Raspberry Pi Zero W every ~week feh was crashing. Turns out feh has signs of a memory leak? OOM kicked in. notice the rss for feh is way higher than any other application running.

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

$ feh --version
feh version 3.1.3
Compile-time switches: curl exif verscmp xinerama

The command I run is:

$ export DISPLAY=:0; feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/

dmesg:

[1094530.907681] feh invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
[1094530.907714] CPU: 0 PID: 929 Comm: feh Tainted: G         C        5.4.51+ #1327
[1094530.907724] Hardware name: BCM2835
[1094530.907735] Backtrace:
[1094530.907774] [] (dump_backtrace) from [] (show_stack+0x20/0x24)
[1094530.907795]  r6:c8337100 r5:c8337100 r4:c5917e08 r3:03684efa
[1094530.907830] [] (show_stack) from [] (dump_stack+0x20/0x28)
[1094530.907858] [] (dump_stack) from [] (dump_header+0x60/0x22c)
[1094530.907877] [] (dump_header) from [] (oom_kill_process+0xac/0x33c)
[1094530.907898]  r10:000000e8 r9:ced23880 r8:c0a280b0 r7:c0a280b0 r6:c8337100 r5:c5917e08
[1094530.907907]  r4:c08e9ed0
[1094530.907924] [] (oom_kill_process) from [] (out_of_memory+0xe8/0x3f8)
[1094530.907945]  r10:000000e8 r9:ced23880 r8:c0a280b0 r7:c0a280b0 r6:c0a25028 r5:c5917e08
[1094530.907954]  r4:c0a28320
[1094530.907978] [] (out_of_memory) from [] (__alloc_pages_nodemask+0xa60/0xf08)
[1094530.907993]  r10:00000000 r9:00000000 r8:00000043 r7:c5917df8 r6:00004ca7 r5:00000000
[1094530.908003]  r4:00000000
[1094530.908039] [] (__alloc_pages_nodemask) from [] (__pte_alloc+0x34/0x1d0)
[1094530.908060]  r10:00000800 r9:c83548fc r8:00000805 r7:c83548c0 r6:c83eeaf0 r5:c83548c0
[1094530.908069]  r4:c0a25028
[1094530.908090] [] (__pte_alloc) from [] (handle_mm_fault+0x948/0x9ec)
[1094530.908107]  r10:00000800 r9:c83548fc r8:00000805 r7:c83548c0 r6:d0850840 r5:c0a477f0
[1094530.908116]  r4:c0a25028
[1094530.908147] [] (handle_mm_fault) from [] (do_page_fault+0x32c/0x3d4)
[1094530.908167]  r9:c83548fc r8:00000805 r7:c83548c0 r6:abdfe208 r5:c8337100 r4:c5917fb0
[1094530.908186] [] (do_page_fault) from [] (do_translation_fault+0xac/0xb4)
[1094530.908205]  r10:fbba8000 r9:00005100 r8:00c5387d r7:c5917fb0 r6:abdfe208 r5:00000805
[1094530.908215]  r4:00000805
[1094530.908244] [] (do_translation_fault) from [] (do_DataAbort+0x44/0xc4)
[1094530.908260]  r6:abdfe208 r5:00000805 r4:c0a2a0f4 r3:00000005
[1094530.908283] [] (do_DataAbort) from [] (__dabt_usr+0x44/0x60)
[1094530.908294] Exception stack(0xc5917fb0 to 0xc5917ff8)
[1094530.908312] 7fa0:                                     ff8f7580 af5df23c abdfe208 000003b2
[1094530.908330] 7fc0: 0cce10b8 af5db008 04458004 0000143f ab182008 00005100 fbba8000 ffffca00
[1094530.908345] 7fe0: af5d6a08 be978640 00000d7f b6c34ee8 00000010 ffffffff
[1094530.908359]  r7:00c5387d r6:ffffffff r5:00000010 r4:b6c34ee8
[1094530.908367] Mem-Info:
[1094530.908393] active_anon:27232 inactive_anon:39679 isolated_anon:0
                  active_file:35 inactive_file:32 isolated_file:0
                  unevictable:4 dirty:0 writeback:0 unstable:0
                  slab_reclaimable:2007 slab_unreclaimable:2558
                  mapped:861 shmem:2382 pagetables:966 bounce:0
                  free:19556 free_pcp:0 free_cma:15635
[1094530.908416] Node 0 active_anon:108928kB inactive_anon:158716kB active_file:140kB inactive_file:128kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:3444kB dirty:0kB writeback:0kB shmem:9528kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[1094530.908445] Normal free:78224kB min:16384kB low:20480kB high:24576kB active_anon:108928kB inactive_anon:158716kB active_file:140kB inactive_file:128kB unevictable:16kB writepending:0kB present:393216kB managed:378100kB mlocked:16kB kernel_stack:1232kB pagetables:3864kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:62540kB
[1094530.908453] lowmem_reserve[]: 0 0
[1094530.908471] Normal: 614*4kB (UMEC) 417*8kB (UMEC) 275*16kB (UEC) 180*32kB (UMEC) 127*64kB (UMEC) 63*128kB (UMC) 32*256kB (UMC) 14*512kB (C) 10*1024kB (C) 4*2048kB (C) 3*4096kB (C) = 78224kB
[1094530.908569] 3012 total pagecache pages
[1094530.908581] 563 pages in swap cache
[1094530.908594] Swap cache stats: add 27539, delete 26976, find 620082/620795
[1094530.908601] Free swap  = 0kB
[1094530.908610] Total swap = 102396kB
[1094530.908617] 98304 pages RAM
[1094530.908625] 0 pages HighMem/MovableOnly
[1094530.908632] 3779 pages reserved
[1094530.908640] 16384 pages cma reserved
[1094530.908647] Tasks state (memory values in pages):
[1094530.908655] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[1094530.908687] [     74]     0    74     8567      777    77824      135             0 systemd-journal
[1094530.908703] [    107]     0   107     4590       12    16384      197         -1000 systemd-udevd
[1094530.908723] [    214]   100   214     5599       24    24576      144             0 systemd-timesyn
[1094530.908741] [    250]     0   250      919       11    12288       42             0 alsactl
[1094530.908757] [    251]     0   251    15538      164    34816      321             0 udisksd
[1094530.908772] [    255]     0   255     6184      101    18432      107             0 rsyslogd
[1094530.908787] [    262] 65534   262     1076        5    12288       39             0 thd
[1094530.908802] [    264]     0   264     3256       48    20480      123             0 systemd-logind
[1094530.908816] [    270]   108   270     1535      135    16384       39             0 avahi-daemon
[1094530.908831] [    272]     0   272     1990       18    16384       35             0 cron
[1094530.908846] [    282]   108   282     1439        4    14336       58             0 avahi-daemon
[1094530.908861] [    283]   104   283     1690      120    14336       54          -900 dbus-daemon
[1094530.908876] [    285]     0   285     6910       26    14336        9             0 rngd
[1094530.908891] [    298]     0   298     2681       12    18432       89             0 wpa_supplicant
[1094530.908905] [    322]     0   322      704       50    10240       41             0 dhcpcd
[1094530.908920] [    347]     0   347     9876       92    28672       99             0 polkitd
[1094530.908935] [    356]     0   356     7990       62    38912      294             0 nmbd
[1094530.908950] [    365]     0   365     9675        0    40960     1299             0 unattended-upgr
[1094530.908965] [    372]     0   372      530        0    10240       29             0 hciattach
[1094530.908980] [    376]     0   376     2677       31    18432      113         -1000 sshd
[1094530.908994] [    380]     0   380     2776       59    18432      103             0 wpa_supplicant
[1094530.909009] [    389]     0   389     2448        0    16384      101             0 bluetoothd
[1094530.909024] [    404]     0   404     9774       84    28672      116             0 lightdm
[1094530.909039] [    410]     0   410     6673        0    22528      112             0 bluealsa
[1094530.909054] [    656]     0   656    33851     2364   133120     4248             0 Xorg
[1094530.909068] [    657]     0   657     1401        0    12288      107             0 login
[1094530.909083] [    681]     0   681    12086      204    55296      405             0 smbd
[1094530.909098] [    737]   111   737     3521       11    22528      140             0 exim4
[1094530.909113] [    740]     0   740    11352       50    51200      423             0 smbd-notifyd
[1094530.909128] [    741]     0   741    11353       62    51200      412             0 cleanupd
[1094530.909143] [    742]     0   742    12083       64    51200      423             0 lpqd
[1094530.909158] [    803]     0   803     8126       33    28672      181             0 lightdm
[1094530.909172] [    809]  1000   809     3678        0    22528      253             0 systemd
[1094530.909187] [    815]  1000   815     4203       36    24576      381             0 (sd-pam)
[1094530.909202] [    831]  1000   831    13826       64    45056      277             0 lxsession
[1094530.909217] [    832]  1000   832     2117        0    16384      228             0 bash
[1094530.909231] [    842]  1000   842     1631        0    16384      106             0 dbus-daemon
[1094530.909246] [    875]  1000   875     1126       10    12288       62             0 ssh-agent
[1094530.909261] [    886]  1000   886    10595        0    26624      166             0 gvfsd
[1094530.909276] [    891]  1000   891    13452        0    26624      190             0 gvfsd-fuse
[1094530.909291] [    907]  1000   907    15491       67    45056      434             0 openbox
[1094530.909305] [    908]  1000   908    11780       73    38912      204             0 lxpolkit
[1094530.909320] [    909]  1000   909    37068     1504    86016     1038             0 lxpanel
[1094530.909335] [    911]  1000   911    18003        0    55296     1146             0 pcmanfm
[1094530.909350] [    919]  1000   919     1913        0    14336       37             0 bash
[1094530.909365] [    922]  1000   922     1126        0    12288       72             0 ssh-agent
[1094530.909379] [    929]  1000   929    86450    58327   342016     9229             0 feh
[1094530.909394] [    952]  1000   952    19605       41    38912      305             0 gvfs-udisks2-vo
[1094530.909409] [    955]  1000   955     7172        0    22528      151             0 menu-cached
[1094530.909424] [    962]  1000   962     9770        0    28672      129             0 gvfs-mtp-volume
[1094530.909439] [    966]  1000   966    13490       13    38912      189             0 gvfs-afc-volume
[1094530.909454] [    971]  1000   971     9766        0    26624      124             0 gvfs-goa-volume
[1094530.909468] [    975]  1000   975    10164        0    24576      152             0 gvfs-gphoto2-vo
[1094530.909483] [    988]  1000   988    13264       33    34816      191             0 gvfsd-trash
[1094530.909500] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=feh,pid=929,uid=1000
[1094530.909542] Out of memory: Killed process 929 (feh) total-vm:345800kB, anon-rss:233248kB, file-rss:60kB, shmem-rss:0kB, UID:1000 pgtables:334kB oom_score_adj:0
[1094531.086376] oom_reaper: reaped process 929 (feh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Ta.

@derf
Copy link
Owner

derf commented Aug 28, 2020

Hi!

Unfortunately, I can't reproduce that on amd64 and I don't have a raspberry pi with X11 at hand at the moment.

Could you install the valgrind package and run your command with it?

export DISPLAY=:0; valgrind /usr/bin/feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/ > /tmp/valgrind.log 2>&1

You can terminate feh after a few hours or days or wait until it crashes. Either way, I'm looking for valgrind's memory leak output, which looks like this:

LEAK SUMMARY:
   definitely lost: 0 bytes in 0 blocks
   indirectly lost: 0 bytes in 0 blocks
     possibly lost: 0 bytes in 0 blocks
   still reachable: 73,392 bytes in 675 blocks
        suppressed: 0 bytes in 0 blocks
Rerun with --leak-check=full to see details of leaked memory

If any lost value is reported as non-zero, there''s a leak. If "still reachable" is in the range of several hundred megabytes, there's probably also something odd.

@sgreadly
Copy link
Author

sgreadly commented Aug 30, 2020

Thanks Daniel!

I'd be happy to help. Quick advice (still picking up on valgrind), If I run your one liner, it seems to be running in the background, but the pictures don't show up - i.e. feh isn't working.

Same if I run it via my .sh script (auto after boot), or manually, or with sudo.

pi 928 921 86 15:58 ? 00:02:25 /usr/bin/valgrind.bin /usr/bin/feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/

The only output I get is via the terminal when run manually. valgrind.log is empty (not being written to)

pi@pizero:~ $ export DISPLAY=:0; sudo valgrind /usr/bin/feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/ 2>&1 > /var/tmp/valgrind.log
==1425== Memcheck, a memory error detector
==1425== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==1425== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==1425== Command: /usr/bin/feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/
==1425==
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/ld-2.28.so:
--1425-- Ignoring non-Dwarf2/3/4 block in .debug_info
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/ld-2.28.so:
--1425-- Last block truncated in .debug_info; ignoring
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401A5D0: index (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401A5D4: index (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4008040: _dl_dst_count (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4008288: expand_dynamic_string_token (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401AA80: strlen (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401AA84: strlen (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4017F68: malloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4017F74: malloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B5E8: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B608: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B618: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B634: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B63C: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B664: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B664: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B68C: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6A0: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6A4: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6B0: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x40180A4: calloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4017FA8: malloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401A160: mmap (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Syscall param mmap2(start) contains uninitialised byte(s)
==1425==    at 0x401A174: mmap (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Syscall param mmap2(length) contains uninitialised byte(s)
==1425==    at 0x401A174: mmap (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Syscall param mmap2(offset) contains uninitialised byte(s)
==1425==    at 0x401A174: mmap (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4017F44: malloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x400BDD0: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B5F4: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B630: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BD7C: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BD98: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401AA14: strdup (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B660: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B660: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B688: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4008E20: _dl_map_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Syscall param openat(filename) contains uninitialised byte(s)
==1425==    at 0x4019F4C: __open64_nocancel (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x40180E4: free (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x400BB84: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BB9C: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BBBC: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BBC0: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BBE0: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BC50: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BC64: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BCA8: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BCC0: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401AA30: strlen (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401AA48: strlen (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B628: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400BD9C: _dl_new_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4005D98: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005DC4: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005DD0: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005E50: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005E9C: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005EA0: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400602C: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x40060C0: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x40060DC: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4006114: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4006A0C: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x40061D4: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4010FF4: _dl_name_match_p (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4011008: _dl_name_match_p (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401A620: strcmp (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4010FFC: _dl_name_match_p (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4013660: _dl_get_origin (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B690: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B694: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4013674: _dl_get_origin (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x400832C: expand_dynamic_string_token (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x40082E4: expand_dynamic_string_token (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4008114: _dl_dst_substitute (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401A67C: strcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4008198: _dl_dst_substitute (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B668: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B668: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x400926C: _dl_map_object (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B7A0: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B6AC: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B900: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B6B0: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B6B4: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B648: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/libm-2.28.so:
--1425-- Ignoring non-Dwarf2/3/4 block in .debug_info
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/libm-2.28.so:
--1425-- Last block truncated in .debug_info; ignoring
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6A8: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6AC: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x401B6B4: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x4017FF0: malloc (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005FCC: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4005FE0: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x4006020: _dl_map_object_from_fd (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/libc-2.28.so:
--1425-- Ignoring non-Dwarf2/3/4 block in .debug_info
--1425-- WARNING: Serious error when reading debug info
--1425-- When reading debug info from /lib/arm-linux-gnueabihf/libc-2.28.so:
--1425-- Last block truncated in .debug_info; ignoring
==1425== Conditional jump or move depends on uninitialised value(s)
==1425==    at 0x400E3FC: _dl_map_object_deps (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x400E410: _dl_map_object_deps (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B850: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==
==1425== Use of uninitialised value of size 4
==1425==    at 0x401B684: memcpy (in /lib/arm-linux-gnueabihf/ld-2.28.so)
==1425==

@derf
Copy link
Owner

derf commented Aug 30, 2020

The empty output is due to a typo in my original reply. I quickly edited it, but the notification mail had already been sent out. The correct command is

export DISPLAY=:0; valgrind /usr/bin/feh -qrYzZFD 120.0 --auto-rotate --zoom fill /home/pi/Pictures/frame/ > /tmp/valgrind.log 2>&1

How long did you wait after starting the command? I forgot to mention that valgrind slows feh down significantly, as it instruments each and every memory allocation in order to track memory leaks. Especially on large directories or an old raspberry pi, it may take up to a few minutes for the first picture to show.

Could you check again whether it works when giving it some time? You don't need to use sudo.

@sgreadly
Copy link
Author

I left it for just a few minutes, but will let it run overnight and report in the morning. Checking now I see memcheck-arm-li (valgrind) hammering the poor CPU so I'll let it be :)

@derf
Copy link
Owner

derf commented Aug 30, 2020

Okay, scratch that. valgrind on Raspbian seems to be horribly broken when debugging a program which uses libdl, such as feh. "valgrind /bin/true" and "valgrind /bin/date" work fine even though they spew the same errors, whereas both "valgrind /bin/ls" and "valgrind /usr/bin/feh ..." fail (as in, get stuck at 100%cpu without apparent progress). I'll get back to you once I know how to debug feh on your system (or once I am able to do so myself)

@sgreadly
Copy link
Author

Appreciate your help on this :/ Yea ~10hrs later, still images don't pop up, and nothing else outputted to the logfile other than the above initial dump.

Let me know if you need my help any time.

@hawgcaller
Copy link

hawgcaller commented Dec 4, 2020

feh v3.5 has been crashing on multiple Raspberry Pi 3B+'s about 4 days after being invoked. dmesg reports the same OOM call as sgreadly so it appears that I'm experiencing the same memory leak.

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ feh --version
feh version 3.5
Compile-time switches: exif verscmp 

The command I run is:

/usr/local/bin/feh \
--quiet \
--recursive \
--hide-pointer \
--randomize \
--fullscreen \
--slideshow-delay 10 \
--fontpath /usr/share/fonts/truetype/dejavu/ \
--font DejaVuSans/20 \
--zoom max \
--auto-rotate \
--info '/home/pi/exifdate.sh %F' \
/home/pi/slideshow &

and exifdate.sh, which extracts the date from EXIF data and formats it:

#!/bin/bash
exiftool -DateTimeOriginal -d "%b %d,%Y" "$1" | # extract date from EXIF and format
         cut -d : -f 2-2 |                      # cut text before and after the second : to remove non date/time fields
         cut -d ' ' -f -3                       # cut text after third space to remove remained of time field

dmesg:

[258103.323357] ieee80211 phy0: brcmf_run_escan: error (-110)
[258103.323383] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[258149.279622] feh invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[258149.279638] CPU: 3 PID: 704 Comm: feh Tainted: G         C        5.4.72-v7+ #1356
[258149.279641] Hardware name: BCM2835
[258149.279644] Backtrace: 
[258149.279660] [<8010e1c0>] (dump_backtrace) from [<8010e538>] (show_stack+0x20/0x24)
[258149.279668]  r7:ffffffff r6:00000000 r5:60000113 r4:80e996f0
[258149.279678] [<8010e518>] (show_stack) from [<8090e830>] (dump_stack+0xd4/0x118)
[258149.279686] [<8090e75c>] (dump_stack) from [<8028df9c>] (dump_header+0x64/0x1f8)
[258149.279693]  r10:00100cca r9:a6054000 r8:00000000 r7:00000558 r6:80bbbb00 r5:a9ed1e80
[258149.279696]  r4:a6055e50 r3:00000000
[258149.279702] [<8028df38>] (dump_header) from [<8028d334>] (oom_kill_process+0x17c/0x188)
[258149.279708]  r7:00000558 r6:80bbbb00 r5:a6055e50 r4:a9ed1e80
[258149.279713] [<8028d1b8>] (oom_kill_process) from [<8028dd9c>] (out_of_memory+0x244/0x344)
[258149.279718]  r7:80e04fa8 r6:80e08200 r5:a6055e50 r4:a9ed1e80
[258149.279727] [<8028db58>] (out_of_memory) from [<802da420>] (__alloc_pages_nodemask+0xbe8/0x121c)
[258149.279731]  r6:80e9b3c8 r5:00001f67 r4:00000000
[258149.279740] [<802d9838>] (__alloc_pages_nodemask) from [<802c3480>] (handle_mm_fault+0x558/0xc98)
[258149.279745]  r10:00000055 r9:6c851000 r8:00000001 r7:00000040 r6:00000055 r5:950ae540
[258149.279748]  r4:00000000
[258149.279757] [<802c2f28>] (handle_mm_fault) from [<8092e0ac>] (do_page_fault+0x148/0x38c)
[258149.279762]  r10:00000055 r9:b4858c40 r8:b4858c00 r7:a9ed1e80 r6:00000817 r5:6c851000
[258149.279765]  r4:a6055fb0
[258149.279772] [<8092df64>] (do_page_fault) from [<80116138>] (do_DataAbort+0x4c/0xd0)
[258149.279777]  r10:fa472800 r9:00005dc0 r8:a6055fb0 r7:8092df64 r6:6c851000 r5:00000817
[258149.279780]  r4:80e0a884
[258149.279786] [<801160ec>] (do_DataAbort) from [<80101d24>] (__dabt_usr+0x44/0x60)
[258149.279789] Exception stack(0xa6055fb0 to 0xa6055ff8)
[258149.279794] 5fa0:                                     ff040605 7081bfe8 6c851000 00001497
[258149.279800] 5fc0: 2aed8448 7081b488 05b8d804 0000176f 677dd008 00005dc0 fa472800 ffffc180
[258149.279804] 5fe0: 6d368d80 7ee10588 000006a1 76ceaee8 00000010 ffffffff
[258149.279810]  r8:10c5387d r7:10c5383d r6:ffffffff r5:00000010 r4:76ceaee8
[258149.279813] Mem-Info:
[258149.279827] active_anon:103935 inactive_anon:107683 isolated_anon:0
                 active_file:1989 inactive_file:2005 isolated_file:32
                 unevictable:4 dirty:0 writeback:0 unstable:0
                 slab_reclaimable:5004 slab_unreclaimable:4037
                 mapped:3772 shmem:2416 pagetables:1156 bounce:0
                 free:7843 free_pcp:169 free_cma:1
[258149.279837] Node 0 active_anon:415740kB inactive_anon:430732kB active_file:7956kB inactive_file:8020kB unevictable:16kB isolated(anon):0kB isolated(file):128kB mapped:15088kB dirty:0kB writeback:0kB shmem:9664kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[258149.279851] Normal free:31372kB min:53248kB low:57344kB high:61440kB active_anon:415740kB inactive_anon:430732kB active_file:8340kB inactive_file:8276kB unevictable:16kB writepending:0kB present:970752kB managed:947028kB mlocked:16kB kernel_stack:1568kB pagetables:4624kB bounce:0kB free_pcp:676kB local_pcp:0kB free_cma:4kB
[258149.279854] lowmem_reserve[]: 0 0
[258149.279862] Normal: 1671*4kB (UMEC) 722*8kB (UME) 439*16kB (UME) 230*32kB (UME) 75*64kB (UME) 4*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 32156kB
[258149.279897] 6768 total pagecache pages
[258149.279904] 347 pages in swap cache
[258149.279909] Swap cache stats: add 27200, delete 26853, find 22930/23140
[258149.279913] Free swap  = 0kB
[258149.279917] Total swap = 102396kB
[258149.279921] 242688 pages RAM
[258149.279925] 0 pages HighMem/MovableOnly
[258149.279930] 5931 pages reserved
[258149.279934] 16384 pages cma reserved
[258149.279938] Tasks state (memory values in pages):
[258149.279943] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[258149.279963] [    112]     0   112     8304     1827    88064      138             0 systemd-journal
[258149.279970] [    152]     0   152     4625      186    20480      152         -1000 systemd-udevd
[258149.279981] [    292]     0   292     2787     1084    18432       50             0 mount.ntfs
[258149.279989] [    310]   100   310     5603      108    26624      127             0 systemd-timesyn
[258149.279997] [    351]     0   351     1987      146    18432       31             0 cron
[258149.280004] [    353]     0   353     6378      243    26624      102             0 rsyslogd
[258149.280012] [    355]     0   355     3260      248    22528      127             0 systemd-logind
[258149.280019] [    359]   104   359     1688      163    14336       60          -900 dbus-daemon
[258149.280027] [    366]     0   366     2685       98    20480       89             0 wpa_supplicant
[258149.280034] [    370]     0   370     6914       80    18432       10             0 rngd
[258149.280041] [    374]     0   374      923      126    10240       41             0 alsactl
[258149.280049] [    386]   108   386     1476       73    14336       30             0 avahi-daemon
[258149.280056] [    391]     0   391    16339      502    40960      309             0 udisksd
[258149.280064] [    401] 65534   401     1080       79    14336       37             0 thd
[258149.280071] [    410]   108   410     1443       13    12288       50             0 avahi-daemon
[258149.280079] [    455]     0   455    10116       78    30720      110             0 polkitd
[258149.280086] [    462]     0   462     2781      474    20480      103             0 wpa_supplicant
[258149.280093] [    492]     0   492      535        0     8192       32             0 hciattach
[258149.280101] [    495]     0   495     2483       92    18432       55             0 bluetoothd
[258149.280108] [    497]     0   497     6677       68    22528      119             0 bluealsa
[258149.280116] [    564]     0   564      726      133    10240       47             0 dhcpcd
[258149.280123] [    571]     0   571    10027      138    28672      119             0 lightdm
[258149.280131] [    578]     0   578    53045     3344   157696     3747             0 Xorg
[258149.280138] [    579]     0   579     1405       78    12288      110             0 login
[258149.280145] [    591]  1000   591     3677       74    22528      183             0 systemd
[258149.280153] [    592]  1000   592     4207       68    26624      350             0 (sd-pam)
[258149.280160] [    602]  1000   602     2123       67    16384      227             0 bash
[258149.280168] [    626]     0   626     8131      134    28672      184             0 lightdm
[258149.280175] [    630]  1000   630    14083      514    45056      272             0 lxsession
[258149.280182] [    638]  1000   638     1636       79    16384      110             0 dbus-daemon
[258149.280189] [    662]  1000   662     1130       16    12288       62             0 ssh-agent
[258149.280197] [    673]  1000   673    10853      168    30720      166             0 gvfsd
[258149.280205] [    678]  1000   678    14193      138    36864      192             0 gvfsd-fuse
[258149.280212] [    687]  1000   687    15508      658    45056      302             0 openbox
[258149.280219] [    689]  1000   689    11784      357    40960      233             0 lxpolkit
[258149.280227] [    692]  1000   692    38846     3038    92160      911             0 lxpanel
[258149.280234] [    694]  1000   694    23110      693    65536     1069             0 pcmanfm
[258149.280241] [    704]  1000   704   220868   201114   892928    13320             0 feh
[258149.280249] [    707]  1000   707     1130        0    14336       72             0 ssh-agent
[258149.280257] [    735]  1000   735    24475       86    55296      345             0 gvfs-udisks2-vo
[258149.280264] [    741]  1000   741    14007      166    40960      203             0 gvfs-afc-volume
[258149.280272] [    746]  1000   746    10026        0    24576      124             0 gvfs-goa-volume
[258149.280279] [    750]  1000   750    10032        0    32768      135             0 gvfs-mtp-volume
[258149.280286] [    757]  1000   757     7176      114    26624      152             0 menu-cached
[258149.280294] [    761]  1000   761    10424        5    24576      164             0 gvfs-gphoto2-vo
[258149.280301] [    766]  1000   766    13780      174    38912      200             0 gvfsd-trash
[258149.280309] [    773]  1000   773     7745        0    28672      113             0 gvfsd-metadata
[258149.280321] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=feh,pid=704,uid=1000
[258149.280358] Out of memory: Killed process 704 (feh) total-vm:883472kB, anon-rss:802828kB, file-rss:1624kB, shmem-rss:0kB, UID:1000 pgtables:872kB oom_score_adj:0
[258149.484538] oom_reaper: reaped process 704 (feh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[258163.323459] ieee80211 phy0: brcmf_run_escan: error (-110)
[258163.323484] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)

I've restarted feh and am monitoring the process using top. After 34.5 hours, feh is using 45.5% of memory. When first executed, feh was using ~10% of memory and that value has slowly grown since.

I believe this may be related to an issue I'd previously reported here (https://github.com/derf/feh/issues/564). The "Failed to run info command" also occurred about four days after starting feh, which is why I think the two may be related. As an attempt to resolve the command run failure I changed from calling exif directly from the --info option and instead use exiftool called from a script and now I am getting the memory leak crash.

I'm happy to help debug this further if needed.

p.s. My image source folder contains over 31000 JPG files.

@derf
Copy link
Owner

derf commented Dec 5, 2020

Alright... Looks like I've found the issue. It's not a memory leak per se, but has pretty much the same effect and is indeed a bug.

When feh loads an image (and has been compiled with exif=1), it stores both the image itself and the accompanying EXIF data in its internal filelist data structure. When switching to the next image, it deletes the image from memory, but keeps the EXIF data. With long-running slideshows spanning thousands of files, this adds up over time.

I did some tests with my personal photo collection and found that feh stores about 50kB of EXIF data per image. For slideshow-delay=10 and 34.5 hours runtime, that gives around 600MB of memory used just for EXIF data, which isn't too far from the numbers you reported.

I'll publish a bugfix release which deletes both image and EXIF data from memory when feh switches to a new image. Thank you for the detailed reports.

@derf derf closed this as completed in d473ac1 Dec 5, 2020
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Dec 19, 2020
Sun, 06 Dec 2020 08:34:15 +0100  Daniel Friesel <derf+feh@finalrewind.org>

* Release v3.6.1
    * Fix excessive memory consumption when showing long-running slideshows
      with thousands to tens of thousands of images and feh has been compiled
      with exif=1 (see derf/feh#553)
    * Fix memory leak when showing long-running slideshows with relatively few
      images and feh has been compiled with exif=1 (ibid.)
    * Fix memory leak when reloading an image and feh has been compiled with
      exif=1
    * Fix memory leak in --draw-exif
    * Fix memory leak when reloading HTTP files with --no-conversion-cache

Mon, 30 Nov 2020 19:44:47 +0100  Daniel Friesel <derf+feh@finalrewind.org>
* Release v3.6
    * Add flip and rotate options to the menu
    * Improve unloadable image detection time (e.g. for large video files) by
      checking a file's header before passing it to Imlib2. For rarely used
      image formats, there is a very small chance that an image which could be
      loaded by feh 3.5 is reported as unloadable by feh 3.6 due to this
      change. Set FEH_SKIP_MAGIC=1 to bypass the header check in this case. See
      <https://phab.enlightenment.org/T8739> and
      <derf/feh#505> for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants