Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upMemory leak? #163
Memory leak? #163
Comments
|
Is the hash-file placed on the same btrfs volumn? Your code look like it is 2016-11-09 11:39 GMT+01:00 Niccolò Belli notifications@github.com:
|
|
Almost, on / I mounted the @ subvolume. I obviously wanted to do df of /home/niko/nosnap/rootfs, but I forgot to use the right path. Anyway even for / 134.98 GiB + 1.8 GiB (hash file) should be 136.78 GiB which is less than the reported free space after the (partial) deduplication (137.16 GiB). Somehow the used space increased instead of decreasing. |
|
Let us check the block usage. Execute this script please and post the output cd /tmp echo "00-49: " $(echo "$stats" | grep "usage 0.[0-4]" -c) The btrfs-debugfs script is from the btrfs progs source and report the 2016-11-09 18:20 GMT+01:00 Niccolò Belli notifications@github.com:
|
|
Here it is: |
|
This tells us that 6,93 GB (best case) - 19,8 GB (worst case) are allocated but not used. 6 * 0.3 + 43 * 0.2 + 94 * 0.1 = 19,8 This is normal btrfs behaivier, most notably, if you have files who are written often.
This command should take less then a half hour. you can check the progress with
After balance please check usage again
and if possible report the usage and the new results from my script. |
|
Ok here is the situation after running a full balance with Then I runned duperemove with Despite running the whole system without even a graphical interface to save memory it still got killed by OOM killer: dmesg: https://paste.pound-python.org/show/jKlCEg7ZlYGdZfiJXzRy/ After the (partial) duperemove the disk usage actually got worse: So I decided to run a full balance once again with Edit: damn, I forgot to delete that damn 1.4GiB hashfile once again. Anyway if you will add 1.4GiB to the estimated free space it will still be less than the previously available free space. |
|
What kernel version are you running? There was a memory leak in the kernel side implementation which got fixed in Linux v4.2. |
|
I'm running 4.8.7 |
Hi, I have 8GB of ram and 8GB of swap, I closed all other applications and I run duperemove this way:
In both cases it dies because it goes out of memory:
What's even more interesting is that disk usage goes higher after the (partial) deduplication:
Before: $ btrfs filesystem df /
Data, single: total=149.35GiB, used=134.98GiB
System, single: total=3.00MiB, used=48.00KiB
Metadata, single: total=3.01GiB, used=1.80GiB
GlobalReserve, single: total=268.59MiB, used=0.00B
After: $ btrfs filesystem df /
Data, single: total=149.35GiB, used=137.16GiB
System, single: total=3.00MiB, used=48.00KiB
Metadata, single: total=3.01GiB, used=1.92GiB
GlobalReserve, single: total=283.55MiB, used=0.00B
Considering the hashfile is only 1.8GB the disk usage increased a little bit instead of decreasing:
$ ls -l rootfs.hash -rw-r--r-- 1 root root 1882632192 9 nov 00.50 rootfs.hash
$ ls -lh rootfs.hash -rw-r--r-- 1 root root 1,8G 9 nov 00.50 rootfs.hash