Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolve issues identified by Valgrind #910

Merged
merged 12 commits into from
May 20, 2020
Merged

Resolve issues identified by Valgrind #910

merged 12 commits into from
May 20, 2020

Conversation

abraunegg
Copy link
Owner

  • bug fixes

@abraunegg
bug fixes
5e13dba 
* bug fixes
@abraunegg abraunegg changed the title bug fixes WIP: Hunting down potential memory leaks May 6, 2020
@abraunegg
Update files
3a1193a 
* Update files based on tracing issues with Valgrind
@abraunegg
Merge branch 'master' into memory-leak-hunting
71b62e1
@abraunegg abraunegg mentioned this pull request May 11, 2020
Merged
@abraunegg
Update sync.d
1c07bbc 
* Update logging statement
@abraunegg
Merge branch 'memory-leak-hunting' of https://github.com/abraunegg/on…
0f25e3e 
…edrive into memory-leak-hunting
@abraunegg
Merge branch 'master' into memory-leak-hunting
805c10f
@abraunegg
Update log.d
7f9ab2e 
* Fix memory logger
@abraunegg
Update leak fixes
304d2b8 
* Update leak fixes
@abraunegg
cleanup PR
ba653ae 
cleanup PR
@abraunegg
PR updates
5fa84c7 
* PR Updates
@abraunegg
Update main.d
1b4d69e 
* Make sure the DB .wal cleanup is being done via exit scopes
@abraunegg
Copy link
Owner Author

@norbusan
This is good for review now. I have been running this with many files & over long periods and memory use is rather constant:
memory usage

Service Details:

● onedrive.service - OneDrive Free Client
   Loaded: loaded (/usr/lib/systemd/system/onedrive.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-05-16 12:58:07 AEST; 18h ago
     Docs: https://github.com/abraunegg/onedrive
 Main PID: 15807 (onedrive)
    Tasks: 1 (limit: 23987)
   Memory: 2.6G
   CGroup: /system.slice/onedrive.service
           └─15807 /usr/local/bin/onedrive --monitor --verbose --enable-logging

Data Details:

  • Files: 89262
  • Folder: 12959
  • Size: 64GB

Observations / Notes:
The memory is fairly consistent in use. The increases seen are when data was added locally and uploaded to OneDrive

@abraunegg abraunegg requested a review from norbusan May 16, 2020 21:46
@abraunegg abraunegg changed the title WIP: Hunting down potential memory leaks Resolve issues identified by Valgrind May 16, 2020
@abraunegg
minor tweaks
11b8ed9 
* minor tweaks
@abraunegg
Copy link
Owner Author

Memory output post data delete:

15807:   /usr/local/bin/onedrive --monitor --verbose --enable-logging
         Address Perm   Offset Device     Inode   Size   Rss   Pss Referenced Anonymous LazyFree ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
        00400000 r-xp 00000000  fd:00 134999052   3636  2348  2346       2348         0        0              0              0               0    0       0      0 onedrive
        0098c000 r--p 0038c000  fd:00 134999052     16    16    14         16         4        0              0              0               0    0       0      0 onedrive
        00990000 rw-p 00390000  fd:00 134999052    568   568   568        568        28        0              0              0               0    0       0      0 onedrive
        00a1e000 rw-p 00000000  00:00         0     20    20    20         20        20        0              0              0               0    0       0      0 
        02043000 rw-p 00000000  00:00         0   6692  6504  6504       6380      6504        0              0              0               0    0       0      0 [heap]
    7f1e22300000 rw-p 00000000  00:00         0  29696 25600 25600      25600     25600        0              0              0               0    0       0      0 
    7f1e24000000 rw-p 00000000  00:00         0    132    24    24         24        24        0              0              0               0    0       0      0 
    7f1e24021000 ---p 00000000  00:00         0  65404     0     0          0         0        0              0              0               0    0       0      0 
    7f1e286bc000 rw-s 001f8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286c4000 rw-s 001f0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286cc000 rw-s 001e8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286d4000 rw-s 001e0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286dc000 rw-s 001d8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286e4000 rw-s 001d0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286ec000 rw-s 001c8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286f4000 rw-s 001c0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e286fc000 rw-s 001b8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28704000 rw-s 001b0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2870c000 rw-s 001a8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28714000 rw-s 001a0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2871c000 rw-s 00198000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28724000 rw-s 00190000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2872c000 rw-s 00188000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28734000 rw-s 00180000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2873c000 rw-s 00178000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28744000 rw-s 00170000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2874c000 rw-s 00168000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28754000 rw-s 00160000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2875c000 rw-s 00158000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28764000 rw-s 00150000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2876c000 rw-s 00148000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28774000 rw-s 00140000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2877c000 rw-s 00138000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28784000 rw-s 00130000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2878c000 rw-s 00128000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28794000 rw-s 00120000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2879c000 rw-s 00118000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287a4000 rw-s 00110000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287ac000 rw-s 00108000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287b4000 rw-s 00100000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287bc000 rw-s 000f8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287c4000 rw-s 000f0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287cc000 rw-s 000e8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287d4000 rw-s 000e0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287dc000 rw-s 000d8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287e4000 rw-s 000d0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287ec000 rw-s 000c8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287f4000 rw-s 000c0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e287fc000 rw-s 000b8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28804000 rw-s 000b0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2880c000 rw-s 000a8000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e28814000 rw-p 00000000  00:00         0  17408 17408 17408      17408     17408        0              0              0               0    0       0      0 
    7f1e29914000 r-xp 00000000  fd:00 201580632     20    20     2         20         0        0              0              0               0    0       0      0 libnss_dns-2.28.so
    7f1e29919000 ---p 00005000  fd:00 201580632   2048     0     0          0         0        0              0              0               0    0       0      0 libnss_dns-2.28.so
    7f1e29b19000 r--p 00005000  fd:00 201580632      4     4     4          4         4        0              0              0               0    0       0      0 libnss_dns-2.28.so
    7f1e29b1a000 rw-p 00006000  fd:00 201580632      4     4     4          4         4        0              0              0               0    0       0      0 libnss_dns-2.28.so
    7f1e29b1b000 ---p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e29b1c000 rw-p 00000000  00:00         0   8192    20    20         20        20        0              0              0               0    0       0      0 
    7f1e2a31c000 r-xp 00000000  fd:00 201580634     44    44     1         44         0        0              0              0               0    0       0      0 libnss_files-2.28.so
    7f1e2a327000 ---p 0000b000  fd:00 201580634   2044     0     0          0         0        0              0              0               0    0       0      0 libnss_files-2.28.so
    7f1e2a526000 r--p 0000a000  fd:00 201580634      4     4     4          0         4        0              0              0               0    0       0      0 libnss_files-2.28.so
    7f1e2a527000 rw-p 0000b000  fd:00 201580634      4     4     4          4         4        0              0              0               0    0       0      0 libnss_files-2.28.so
    7f1e2a528000 rw-p 00000000  00:00         0     24     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2a52e000 r--s 00000000  fd:00 202933202   8212    68     8         68         0        0              0              0               0    0       0      0 passwd
    7f1e2ad33000 r-xp 00000000  fd:00 202862463     32    32     0         32         0        0              0              0               0    0       0      0 libnss_sss.so.2
    7f1e2ad3b000 ---p 00008000  fd:00 202862463   2044     0     0          0         0        0              0              0               0    0       0      0 libnss_sss.so.2
    7f1e2af3a000 r--p 00007000  fd:00 202862463      4     4     4          0         4        0              0              0               0    0       0      0 libnss_sss.so.2
    7f1e2af3b000 rw-p 00008000  fd:00 202862463      4     4     4          0         4        0              0              0               0    0       0      0 libnss_sss.so.2
    7f1e2af3c000 rw-p 00000000  00:00         0   4096  2992  2992       2964      2992        0              0              0               0    0       0      0 
    7f1e2b33c000 r-xp 00000000  fd:00 201580509    524    64     0         64         0        0              0              0               0    0       0      0 libpcre2-8.so.0.7.1
    7f1e2b3bf000 ---p 00083000  fd:00 201580509   2044     0     0          0         0        0              0              0               0    0       0      0 libpcre2-8.so.0.7.1
    7f1e2b5be000 r--p 00082000  fd:00 201580509      4     4     4          0         4        0              0              0               0    0       0      0 libpcre2-8.so.0.7.1
    7f1e2b5bf000 rw-p 00083000  fd:00 201580509      4     4     4          0         4        0              0              0               0    0       0      0 libpcre2-8.so.0.7.1
    7f1e2b5c0000 r-xp 00000000  fd:00 201580876    128    64    20         64         0        0              0              0               0    0       0      0 libcrypt.so.1.1.0
    7f1e2b5e0000 ---p 00020000  fd:00 201580876   2044     0     0          0         0        0              0              0               0    0       0      0 libcrypt.so.1.1.0
    7f1e2b7df000 r--p 0001f000  fd:00 201580876      4     4     4          0         4        0              0              0               0    0       0      0 libcrypt.so.1.1.0
    7f1e2b7e0000 rw-p 00000000  00:00         0     36     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2b7e9000 r-xp 00000000  fd:00 203590757    156   128     1        128         0        0              0              0               0    0       0      0 libselinux.so.1
    7f1e2b810000 ---p 00027000  fd:00 203590757   2048     0     0          0         0        0              0              0               0    0       0      0 libselinux.so.1
    7f1e2ba10000 r--p 00027000  fd:00 203590757      4     4     4          4         4        0              0              0               0    0       0      0 libselinux.so.1
    7f1e2ba11000 rw-p 00028000  fd:00 203590757      4     4     4          0         4        0              0              0               0    0       0      0 libselinux.so.1
    7f1e2ba12000 rw-p 00000000  00:00         0      8     4     4          0         4        0              0              0               0    0       0      0 
    7f1e2ba14000 r-xp 00000000  fd:00 201778657    124     0     0          0         0        0              0              0               0    0       0      0 libbrotlicommon.so.1.0.6
    7f1e2ba33000 ---p 0001f000  fd:00 201778657   2044     0     0          0         0        0              0              0               0    0       0      0 libbrotlicommon.so.1.0.6
    7f1e2bc32000 r--p 0001e000  fd:00 201778657      4     4     4          0         4        0              0              0               0    0       0      0 libbrotlicommon.so.1.0.6
    7f1e2bc33000 rw-p 0001f000  fd:00 201778657      4     4     4          0         4        0              0              0               0    0       0      0 libbrotlicommon.so.1.0.6
    7f1e2bc34000 r-xp 00000000  fd:00 201880737    112     0     0          0         0        0              0              0               0    0       0      0 libsasl2.so.3.0.0
    7f1e2bc50000 ---p 0001c000  fd:00 201880737   2048     0     0          0         0        0              0              0               0    0       0      0 libsasl2.so.3.0.0
    7f1e2be50000 r--p 0001c000  fd:00 201880737      4     4     4          0         4        0              0              0               0    0       0      0 libsasl2.so.3.0.0
    7f1e2be51000 rw-p 0001d000  fd:00 201880737      4     4     4          0         4        0              0              0               0    0       0      0 libsasl2.so.3.0.0
    7f1e2be52000 r-xp 00000000  fd:00 201580638     80    64    10         64         0        0              0              0               0    0       0      0 libresolv-2.28.so
    7f1e2be66000 ---p 00014000  fd:00 201580638   2044     0     0          0         0        0              0              0               0    0       0      0 libresolv-2.28.so
    7f1e2c065000 r--p 00013000  fd:00 201580638      4     4     4          4         4        0              0              0               0    0       0      0 libresolv-2.28.so
    7f1e2c066000 rw-p 00014000  fd:00 201580638      4     4     4          4         4        0              0              0               0    0       0      0 libresolv-2.28.so
    7f1e2c067000 rw-p 00000000  00:00         0      8     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2c069000 r-xp 00000000  fd:00 201581339     12    12     3         12         0        0              0              0               0    0       0      0 libkeyutils.so.1.6
    7f1e2c06c000 ---p 00003000  fd:00 201581339   2044     0     0          0         0        0              0              0               0    0       0      0 libkeyutils.so.1.6
    7f1e2c26b000 r--p 00002000  fd:00 201581339      4     4     4          0         4        0              0              0               0    0       0      0 libkeyutils.so.1.6
    7f1e2c26c000 rw-p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2c26d000 r-xp 00000000  fd:00 201881000     60    60    19         60         0        0              0              0               0    0       0      0 libkrb5support.so.0.1
    7f1e2c27c000 ---p 0000f000  fd:00 201881000   2048     0     0          0         0        0              0              0               0    0       0      0 libkrb5support.so.0.1
    7f1e2c47c000 r--p 0000f000  fd:00 201881000      4     4     4          0         4        0              0              0               0    0       0      0 libkrb5support.so.0.1
    7f1e2c47d000 rw-p 00010000  fd:00 201881000      4     4     4          0         4        0              0              0               0    0       0      0 libkrb5support.so.0.1
    7f1e2c47e000 r-xp 00000000  fd:00 201581568   1524   116    62        116         0        0              0              0               0    0       0      0 libunistring.so.2.1.0
    7f1e2c5fb000 ---p 0017d000  fd:00 201581568   2044     0     0          0         0        0              0              0               0    0       0      0 libunistring.so.2.1.0
    7f1e2c7fa000 r--p 0017c000  fd:00 201581568     16    16    16          0        16        0              0              0               0    0       0      0 libunistring.so.2.1.0
    7f1e2c7fe000 rw-p 00180000  fd:00 201581568      4     4     4          0         4        0              0              0               0    0       0      0 libunistring.so.2.1.0
    7f1e2c7ff000 r-xp 00000000  fd:00 201580830     88     0     0          0         0        0              0              0               0    0       0      0 libz.so.1.2.11
    7f1e2c815000 ---p 00016000  fd:00 201580830   2044     0     0          0         0        0              0              0               0    0       0      0 libz.so.1.2.11
    7f1e2ca14000 r--p 00015000  fd:00 201580830      4     4     4          0         4        0              0              0               0    0       0      0 libz.so.1.2.11
    7f1e2ca15000 rw-p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2ca16000 r-xp 00000000  fd:00 201778659     44     0     0          0         0        0              0              0               0    0       0      0 libbrotlidec.so.1.0.6
    7f1e2ca21000 ---p 0000b000  fd:00 201778659   2044     0     0          0         0        0              0              0               0    0       0      0 libbrotlidec.so.1.0.6
    7f1e2cc20000 r--p 0000a000  fd:00 201778659      4     4     4          0         4        0              0              0               0    0       0      0 libbrotlidec.so.1.0.6
    7f1e2cc21000 rw-p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2cc22000 r-xp 00000000  fd:00 201880748     56     0     0          0         0        0              0              0               0    0       0      0 liblber-2.4.so.2.10.9
    7f1e2cc30000 ---p 0000e000  fd:00 201880748   2048     0     0          0         0        0              0              0               0    0       0      0 liblber-2.4.so.2.10.9
    7f1e2ce30000 r--p 0000e000  fd:00 201880748      4     4     4          0         4        0              0              0               0    0       0      0 liblber-2.4.so.2.10.9
    7f1e2ce31000 rw-p 0000f000  fd:00 201880748      4     4     4          0         4        0              0              0               0    0       0      0 liblber-2.4.so.2.10.9
    7f1e2ce32000 r-xp 00000000  fd:00 201880750    300     0     0          0         0        0              0              0               0    0       0      0 libldap-2.4.so.2.10.9
    7f1e2ce7d000 ---p 0004b000  fd:00 201880750   2044     0     0          0         0        0              0              0               0    0       0      0 libldap-2.4.so.2.10.9
    7f1e2d07c000 r--p 0004a000  fd:00 201880750      8     8     8          0         8        0              0              0               0    0       0      0 libldap-2.4.so.2.10.9
    7f1e2d07e000 rw-p 0004c000  fd:00 201880750      4     4     4          0         4        0              0              0               0    0       0      0 libldap-2.4.so.2.10.9
    7f1e2d07f000 r-xp 00000000  fd:00 201580840     12    12     3         12         0        0              0              0               0    0       0      0 libcom_err.so.2.1
    7f1e2d082000 ---p 00003000  fd:00 201580840   2044     0     0          0         0        0              0              0               0    0       0      0 libcom_err.so.2.1
    7f1e2d281000 r--p 00002000  fd:00 201580840      4     4     4          0         4        0              0              0               0    0       0      0 libcom_err.so.2.1
    7f1e2d282000 rw-p 00003000  fd:00 201580840      4     4     4          0         4        0              0              0               0    0       0      0 libcom_err.so.2.1
    7f1e2d283000 r-xp 00000000  fd:00 201880992    104    64    21         64         0        0              0              0               0    0       0      0 libk5crypto.so.3.1
    7f1e2d29d000 ---p 0001a000  fd:00 201880992   2044     0     0          0         0        0              0              0               0    0       0      0 libk5crypto.so.3.1
    7f1e2d49c000 r--p 00019000  fd:00 201880992      8     8     8          0         8        0              0              0               0    0       0      0 libk5crypto.so.3.1
    7f1e2d49e000 rw-p 0001b000  fd:00 201880992      4     4     4          0         4        0              0              0               0    0       0      0 libk5crypto.so.3.1
    7f1e2d49f000 r-xp 00000000  fd:00 201880998    892   132    45        132         0        0              0              0               0    0       0      0 libkrb5.so.3.3
    7f1e2d57e000 ---p 000df000  fd:00 201880998   2048     0     0          0         0        0              0              0               0    0       0      0 libkrb5.so.3.3
    7f1e2d77e000 r--p 000df000  fd:00 201880998     60    60    60          0        60        0              0              0               0    0       0      0 libkrb5.so.3.3
    7f1e2d78d000 rw-p 000ee000  fd:00 201880998      8     8     8          0         8        0              0              0               0    0       0      0 libkrb5.so.3.3
    7f1e2d78f000 r-xp 00000000  fd:00 201880988    308    64    21         64         0        0              0              0               0    0       0      0 libgssapi_krb5.so.2.2
    7f1e2d7dc000 ---p 0004d000  fd:00 201880988   2048     0     0          0         0        0              0              0               0    0       0      0 libgssapi_krb5.so.2.2
    7f1e2d9dc000 r--p 0004d000  fd:00 201880988      8     8     8          0         8        0              0              0               0    0       0      0 libgssapi_krb5.so.2.2
    7f1e2d9de000 rw-p 0004f000  fd:00 201880988      4     4     4          0         4        0              0              0               0    0       0      0 libgssapi_krb5.so.2.2
    7f1e2d9df000 r-xp 00000000  fd:00 202207362   2736  2432   691       2408         0        0              0              0               0    0       0      0 libcrypto.so.1.1.1c
    7f1e2dc8b000 ---p 002ac000  fd:00 202207362   2048     0     0          0         0        0              0              0               0    0       0      0 libcrypto.so.1.1.1c
    7f1e2de8b000 r--p 002ac000  fd:00 202207362    172   172   172        132       172        0              0              0               0    0       0      0 libcrypto.so.1.1.1c
    7f1e2deb6000 rw-p 002d7000  fd:00 202207362     16    16    16         12        16        0              0              0               0    0       0      0 libcrypto.so.1.1.1c
    7f1e2deba000 rw-p 00000000  00:00         0     16    16    16         12        16        0              0              0               0    0       0      0 
    7f1e2debe000 r-xp 00000000  fd:00 202207364    536   460   344        460         0        0              0              0               0    0       0      0 libssl.so.1.1.1c
    7f1e2df44000 ---p 00086000  fd:00 202207364   2048     0     0          0         0        0              0              0               0    0       0      0 libssl.so.1.1.1c
    7f1e2e144000 r--p 00086000  fd:00 202207364     36    36    36         28        36        0              0              0               0    0       0      0 libssl.so.1.1.1c
    7f1e2e14d000 rw-p 0008f000  fd:00 202207364     16    16    16         16        16        0              0              0               0    0       0      0 libssl.so.1.1.1c
    7f1e2e151000 rw-p 00000000  00:00         0      4     4     4          4         4        0              0              0               0    0       0      0 
    7f1e2e152000 r-xp 00000000  fd:00 201778446     64     0     0          0         0        0              0              0               0    0       0      0 libpsl.so.5.3.1
    7f1e2e162000 ---p 00010000  fd:00 201778446   2044     0     0          0         0        0              0              0               0    0       0      0 libpsl.so.5.3.1
    7f1e2e361000 r--p 0000f000  fd:00 201778446      4     4     4          0         4        0              0              0               0    0       0      0 libpsl.so.5.3.1
    7f1e2e362000 rw-p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2e363000 r-xp 00000000  fd:00 201880743    540     0     0          0         0        0              0              0               0    0       0      0 libssh.so.4.8.1
    7f1e2e3ea000 ---p 00087000  fd:00 201880743   2048     0     0          0         0        0              0              0               0    0       0      0 libssh.so.4.8.1
    7f1e2e5ea000 r--p 00087000  fd:00 201880743      8     8     8          0         8        0              0              0               0    0       0      0 libssh.so.4.8.1
    7f1e2e5ec000 rw-p 00089000  fd:00 201880743      8     8     8          0         8        0              0              0               0    0       0      0 libssh.so.4.8.1
    7f1e2e5ee000 r-xp 00000000  fd:00 201581573    112    64    40         64         0        0              0              0               0    0       0      0 libidn2.so.0.3.6
    7f1e2e60a000 ---p 0001c000  fd:00 201581573   2048     0     0          0         0        0              0              0               0    0       0      0 libidn2.so.0.3.6
    7f1e2e80a000 r--p 0001c000  fd:00 201581573      4     4     4          0         4        0              0              0               0    0       0      0 libidn2.so.0.3.6
    7f1e2e80b000 rw-p 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 
    7f1e2e80c000 r-xp 00000000  fd:00 201880406    144     0     0          0         0        0              0              0               0    0       0      0 libnghttp2.so.14.17.0
    7f1e2e830000 ---p 00024000  fd:00 201880406   2044     0     0          0         0        0              0              0               0    0       0      0 libnghttp2.so.14.17.0
    7f1e2ea2f000 r--p 00023000  fd:00 201880406      4     4     4          0         4        0              0              0               0    0       0      0 libnghttp2.so.14.17.0
    7f1e2ea30000 rw-p 00024000  fd:00 201880406      8     8     8          0         8        0              0              0               0    0       0      0 libnghttp2.so.14.17.0
    7f1e2ea32000 r-xp 00000000  fd:00 201580622   1764  1588    87       1528         0        0              0              0               0    0       0      0 libc-2.28.so
    7f1e2ebeb000 ---p 001b9000  fd:00 201580622   2048     0     0          0         0        0              0              0               0    0       0      0 libc-2.28.so
    7f1e2edeb000 r--p 001b9000  fd:00 201580622     16    16    16         16        16        0              0              0               0    0       0      0 libc-2.28.so
    7f1e2edef000 rw-p 001bd000  fd:00 201580622      8     8     8          8         8        0              0              0               0    0       0      0 libc-2.28.so
    7f1e2edf1000 rw-p 00000000  00:00         0     16    16    16         16        16        0              0              0               0    0       0      0 
    7f1e2edf5000 r-xp 00000000  fd:00 203590736     92    92     2         92         0        0              0              0               0    0       0      0 libgcc_s-8-20190507.so.1
    7f1e2ee0c000 ---p 00017000  fd:00 203590736   2044     0     0          0         0        0              0              0               0    0       0      0 libgcc_s-8-20190507.so.1
    7f1e2f00b000 r--p 00016000  fd:00 203590736      4     4     4          4         4        0              0              0               0    0       0      0 libgcc_s-8-20190507.so.1
    7f1e2f00c000 rw-p 00017000  fd:00 203590736      4     4     4          4         4        0              0              0               0    0       0      0 libgcc_s-8-20190507.so.1
    7f1e2f00d000 r-xp 00000000  fd:00 201580640     28    28     0         28         0        0              0              0               0    0       0      0 librt-2.28.so
    7f1e2f014000 ---p 00007000  fd:00 201580640   2048     0     0          0         0        0              0              0               0    0       0      0 librt-2.28.so
    7f1e2f214000 r--p 00007000  fd:00 201580640      4     4     4          4         4        0              0              0               0    0       0      0 librt-2.28.so
    7f1e2f215000 rw-p 00008000  fd:00 201580640      4     4     4          0         4        0              0              0               0    0       0      0 librt-2.28.so
    7f1e2f216000 r-xp 00000000  fd:00 201580626   1540   360    60        360         0        0              0              0               0    0       0      0 libm-2.28.so
    7f1e2f397000 ---p 00181000  fd:00 201580626   2044     0     0          0         0        0              0              0               0    0       0      0 libm-2.28.so
    7f1e2f596000 r--p 00180000  fd:00 201580626      4     4     4          4         4        0              0              0               0    0       0      0 libm-2.28.so
    7f1e2f597000 rw-p 00181000  fd:00 201580626      4     4     4          0         4        0              0              0               0    0       0      0 libm-2.28.so
    7f1e2f598000 r-xp 00000000  fd:00 201580636    108    96     0         96         0        0              0              0               0    0       0      0 libpthread-2.28.so
    7f1e2f5b3000 ---p 0001b000  fd:00 201580636   2044     0     0          0         0        0              0              0               0    0       0      0 libpthread-2.28.so
    7f1e2f7b2000 r--p 0001a000  fd:00 201580636      4     4     4          4         4        0              0              0               0    0       0      0 libpthread-2.28.so
    7f1e2f7b3000 rw-p 0001b000  fd:00 201580636      4     4     4          4         4        0              0              0               0    0       0      0 libpthread-2.28.so
    7f1e2f7b4000 rw-p 00000000  00:00         0     16     4     4          4         4        0              0              0               0    0       0      0 
    7f1e2f7b8000 r-xp 00000000  fd:00 201580624     12    12     0         12         0        0              0              0               0    0       0      0 libdl-2.28.so
    7f1e2f7bb000 ---p 00003000  fd:00 201580624   2044     0     0          0         0        0              0              0               0    0       0      0 libdl-2.28.so
    7f1e2f9ba000 r--p 00002000  fd:00 201580624      4     4     4          4         4        0              0              0               0    0       0      0 libdl-2.28.so
    7f1e2f9bb000 rw-p 00003000  fd:00 201580624      4     4     4          0         4        0              0              0               0    0       0      0 libdl-2.28.so
    7f1e2f9bc000 r-xp 00000000  fd:00 201581202   1076   700   207        700         0        0              0              0               0    0       0      0 libsqlite3.so.0.8.6
    7f1e2fac9000 ---p 0010d000  fd:00 201581202   2044     0     0          0         0        0              0              0               0    0       0      0 libsqlite3.so.0.8.6
    7f1e2fcc8000 r--p 0010c000  fd:00 201581202     16    16    16          8        16        0              0              0               0    0       0      0 libsqlite3.so.0.8.6
    7f1e2fccc000 rw-p 00110000  fd:00 201581202     12    12    12         12        12        0              0              0               0    0       0      0 libsqlite3.so.0.8.6
    7f1e2fccf000 r-xp 00000000  fd:00 201880759    552   420   144        420         0        0              0              0               0    0       0      0 libcurl.so.4.5.0
    7f1e2fd59000 ---p 0008a000  fd:00 201880759   2048     0     0          0         0        0              0              0               0    0       0      0 libcurl.so.4.5.0
    7f1e2ff59000 r--p 0008a000  fd:00 201880759     12    12    12         12        12        0              0              0               0    0       0      0 libcurl.so.4.5.0
    7f1e2ff5c000 rw-p 0008d000  fd:00 201880759      4     4     4          4         4        0              0              0               0    0       0      0 libcurl.so.4.5.0
    7f1e2ff5d000 r-xp 00000000  fd:00 201580614    160   160     1        160         0        0              0              0               0    0       0      0 ld-2.28.so
    7f1e2ff86000 rw-s 000a0000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ff8e000 rw-s 00098000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ff96000 rw-s 00090000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ff9e000 rw-s 00088000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffa6000 rw-s 00080000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffae000 rw-s 00078000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffb6000 rw-s 00070000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffbe000 rw-s 00068000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffc6000 rw-s 00060000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffce000 rw-s 00058000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffd6000 rw-s 00050000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffde000 rw-s 00048000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffe6000 rw-s 00040000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2ffee000 rw-s 00038000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2fff6000 rw-s 00030000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e2fffe000 rw-s 00028000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e30006000 rw-s 00020000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e3000e000 rw-s 00018000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e30016000 rw-s 00010000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e3001e000 rw-p 00000000  00:00         0   1364  1100  1100       1096      1100        0              0              0               0    0       0      0 
    7f1e30175000 rw-s 00008000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e3017d000 rw-s 00000000  fd:00  72732761     32    32    32         32         0        0              0              0               0    0       0      0 items.sqlite3-shm
    7f1e30185000 r--p 00028000  fd:00 201580614      4     4     4          4         4        0              0              0               0    0       0      0 ld-2.28.so
    7f1e30186000 rw-p 00029000  fd:00 201580614      4     4     4          4         4        0              0              0               0    0       0      0 ld-2.28.so
    7f1e30187000 rw-p 00000000  00:00         0      4     4     4          4         4        0              0              0               0    0       0      0 
    7ffccbafd000 rw-p 00000000  00:00         0    132    84    84         24        84        0              0              0               0    0       0      0 [stack]
    7ffccbbf1000 r--p 00000000  00:00         0     12     0     0          0         0        0              0              0               0    0       0      0 [vvar]
    7ffccbbf4000 r-xp 00000000  00:00         0      8     4     0          4         0        0              0              0               0    0       0      0 [vdso]
ffffffffff600000 r-xp 00000000  00:00         0      4     0     0          0         0        0              0              0               0    0       0      0 [vsyscall]
                                                ====== ===== ===== ========== ========= ======== ============== ============== =============== ==== ======= ====== 
                                                232052 66760 61188      66140     54452        0              0              0               0    0       0      0 KB

@abraunegg abraunegg mentioned this pull request May 16, 2020
Closed
@norbusan
Copy link
Collaborator

Hi Alex,
that is a pain to review, lots of noise, renaming of variables. Above that, the commits are not logical units that can be reviewed one by one.

I read through the whole change, not that I found something strange. What puzzles me is the replacement of

auto foo = somecall(...)

to

sometype foo = trivialinitializer
foo = somecall()

Is the auto mechanism not working properly and eating up memory?

I will install that patch locally and let it run for some time.

@norbusan
Copy link
Collaborator

First comment after some iteration with reporting memory: it seems to be stable, most of the time I get

memory usedSize = 369440
memory freeSize = 12213472

and around syncs I aget

memory usedSize = 5633792
memory freeSize = 6949120

these are the only numbers by now. When doing a periodic sync it switched to the lower numbers, and after that back to the upper numbers. No obvious changes for several loops.

So seems to be fine...

@abraunegg
Copy link
Owner Author

@norbusan

that is a pain to review, lots of noise, renaming of variables. Above that, the commits are not logical units that can be reviewed one by one.

Yep .. nature of 'whack-a-mole' when testing each change locally, seeing what next Valgrind complained with.

Is the auto mechanism not working properly and eating up memory?

Not 100% sure. Valgrind would complain when a variable was set to 'auto' and this not 'initialised' separately, but when set to correct type, then set to a value or function response, the Valgrind error would not appear. Personally I think that, that is a Valgrind issue not knowing D or the internal constructs ... but those changes - for example - 11b8ed9

error reported without change:

==28693== Conditional jump or move depends on uninitialised value(s)
==28693==    at 0x71B6D5: _D2gc4impl12conservativeQw3Gcx__T4markVbi0Vbi1ZQoMFNbNlSQCcQCcQCaQClQBq__T9ScanRangeVbi0ZQpZv (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x7160FB: _D2gc4impl12conservativeQw3Gcx12markParallelMFNbbZv (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x71592B: _D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x713C50: _D2gc4impl12conservativeQw3Gcx10smallAllocMFNbmKmkxC8TypeInfoZPv (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x71964C: _D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12mallocNoSyncMFNbmkKmxC8TypeInfoZPvS_DQEgQEgQEeQEp10mallocTimelS_DQFiQFiQFgQFr10numMallocslTmTkTmTxQCzZQFcMFNbKmKkKmKxQDsZQDl (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x711271: _D2gc4impl12conservativeQw14ConservativeGC6mallocMFNbmkxC8TypeInfoZPv (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x67F51A: gc_malloc (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x6CC464: _D2rt3aaA11fakeEntryTIFKSQxQw4ImplxC8TypeInfoxQlZC15TypeInfo_Struct (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x6CBB87: _D2rt3aaA4Impl6__ctorMFNcMxC25TypeInfo_AssociativeArraymZSQCeQCeQCd (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x6801AE: _d_assocarrayliteralTX (in /home/alex/code/onedrive-leak-work/onedrive)
==28693==    by 0x669078: _D4sync10SyncEngine22uploadLastModifiedTimeMFAxaQdQfS3std8datetime7systime7SysTimeZv (src/sync.d:3929)
==28693==    by 0x667133: _D4sync10SyncEngine13uploadNewFileMFxAyaZv (src/sync.d:3424)
==28693==  Uninitialised value was created by a stack allocation
==28693==    at 0x645CEC: _Dmain (src/main.d:19)

That commit - fixes that ... for no 'logical' reason as it is technically the same ...

@abraunegg
Copy link
Owner Author

@norbusan
What do you want to do with this PR? I know that this fixes slight memory leak issues based on pre / post valgrind running / output

norbusan
norbusan approved these changes May 20, 2020
View reviewed changes
Copy link
Collaborator

@norbusan norbusan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I had it now running for 2 days. Memory still increases slightly when files are downloaded, but I guess that is to be expected. The changes - as far as I could check with all the noise - look ok.

@abraunegg abraunegg merged commit 206ab8f into master May 20, 2020
@abraunegg abraunegg deleted the memory-leak-hunting branch May 20, 2020 01:37
@abraunegg abraunegg added this to the v2.4.2 milestone May 25, 2020
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators May 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@norbusan norbusan norbusan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v2.4.2
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@abraunegg @norbusan