Huge memory consumption

[petepi3]

Running on Ubuntu trough WSL, memory consumption steadily rises few megabytes every second until antirety of my system memory is full and twitch-dl gets stuck. Shutting the program down doesn't release any of that memory.

this happens everytime with every download consistently between system reboots.

Possible memory leak?

WSL 10.0.19041.1566 running ubuntu 20.04 LTS
16GB of system memory
I can provide any other specs / logs if necessary

[ihabunek]

That's strange. Twitch-dl should not use much memory since VODs are saved to disk and not held in memory.
I don't have windows installed so can't really experiment with it.

The only thing that comes to mind if WSL keeps temp files in memory for some reason...
Could you run this in python and tell me what you get back?

import tempfile
tempfile.gettempdir()

[petepi3]

$ python
Python 2.7.18 (default, Mar 8 2021, 13:02:45)
[GCC 9.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import tempfile
tempfile.gettempdir()>>> tempfile.gettempdir()
'/tmp'

[ihabunek]

If anyone is running WSL, I'd love some help debugging this issue.

[aromatic-dev]

Hi, I can reproduce this issue. @petepi3 you can get the memory freed up by shutting down WSL, by running wsl --shutdown in your Windows terminal.

[aromatic-dev]

@ihabunek Indeed, this isn't a memory leak in the traditional sense -- the memory used by the twitch-dl process during the download is relatively small and more or less constant (as observed in htop) and goes away once the process terminates.

Your suspicion was correct: cleaning up twitch-dl's temp files (e.g. rm -r /tmp/twitch-dl/* ) will reduce the memory used by the vmmem Windows process.

@petepi3 as another temporary workaround, you could execute the above file removal command under WSL after you've finished your download (assuming, of course, that the download finishes before the memory limit is reached).

[ihabunek]

Hm, WSL stores /tmp in memory? That's unexpected. You could try moving the temp dir which twitch-dl uses to somewhere on disk. For example in your home dir:

mkdir ~/tmp
TMP=~/tmp twitch-dl download <videoid>

[aromatic-dev]

Unfortunately, this has no effect on the memory consumption -- in my case, I explicitly configured WSL to use 4GB by creating the file %UserProfile%/.wslconfig with the following contents:

[wsl2]
memory=4GB

vmmem quickly reached this limit, but twitch-dl continued to download despite this. @petepi3 this might be the most straight-forward solution for you.

[shrubbroom]

Hm, WSL stores /tmp in memory? That's unexpected. You could try moving the temp dir which twitch-dl uses to somewhere on disk. For example in your home dir:

mkdir ~/tmp
TMP=~/tmp twitch-dl download <videoid>

I believe not only wsl have /tmp stored in RAM. archlinux and other distros which use tmpfs in /tmp stores the directory in RAM or swap. It should be much better if the default TMP is changed to /var/cache or ~/.cache rather than /tmp.