question re pip install

[aaannndddyyy]

Description of the bug

In your install instructions it says "Run pip install moodle-dl as administrator. "
But running 'sudo pip install moodle.dl', pip complains that it is not happy with being run with root privileges:
"WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
"
Can I safely ignore this?

Also I get this:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
isisdl 1.3.19 requires requests~=2.28.1, but you have requests 2.31.0 which is incompatible.
Successfully installed requests-2.31.0

Steps to reproduce the issue

Expected behavior

Possible Fix

Technical details

[C0D3D3V]

I think it is time we revise the installation instructions.
The warning does say everything you need to know. If you are installing packages using pip, it will conflict with your package manager. More and more distributions started adding python packages to their repositories. One of the bigest package repos has Arch linux. If you install moodle-dl from your linux distro repo everything is fine, because it will manage the dependencies using your package manager. If you install moodle-dl using pip, your packagemanager can not know about it, and it can also come to conflicts.

Since the installation method of moodle-dl itself is not the best, e.g. if you have installed moodle-dl using your package manager and python gets updated on Arch linux, moodle-dl will not work without reinstallation. Thats why I currently would recommend installing moodle-dl in a python virtual environment.

We can collect some best practice for python virtualenv here, and than revise the instructions.

You can also look at my instructions here, I added expert installation instructions using virtualenv: https://github.com/C0D3D3V/bbb-dl
But I did not make it the only (default) instructions, because it confused the users.

Better alternative would be, that we ship our own bundels for windows like yt-dlp does, so windows user do no longer need to run installations as administrators. And also investigate how we can fix the installation process using package managers.

Regarding your other questions:
The choosen version of requests is arbitrary, you can downgrade if your other packages are broken. pip can only install one version at the time. Thats why it is even better to use a virtual enviroment.
You do not need to run pip as sudo on linux. On linux you most of the time do not need to run pip as sudo. The only advantage will be that all users have access to the packages installed by pip (e.g. moodle-dl). But on windows the installation process often needs Administrator privileges, thats why I have written it there. Windows users often know less about installation processes than linux users... so I hoped that will help them.

[aaannndddyyy]

Thank you for your answer. I followed the instructions with the virtual environment. Am on linux but had initially followed the instructions word by word, in order not to screw up.
It installed fine now. Is running too, but I often get some errors.
FWIW, these are the most common ones:

19:48:16 [5233] yt-dlp Warning: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "lUdCB6cqpvo")
19:48:16 [5234] yt-dlp Warning: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "rwxE35URuF8")

19:48:13 [5232] Download of the external file was canceled because of HTTP error: 405 Method Not Allowed

19:53:09 [5659] Download of the external file was canceled because of HTTP error: 403 Forbidden

19:53:43 [5800] yt-dlp Error: ERROR: list index out of range

19:58:15 [6261] yt-dlp Warning: The program functionality for this site has been marked as broken, and will probably not work.

19:58:21 [6290] OperationalError('unable to open database file')
19:58:21 [6290] Error while trying to download file: unable to open database file

19:58:16 [6261] yt-dlp Error: ERROR: [instagram:user] fachini_physik_hu?igshid=YmMyMTA2M2Y=: Unable to extract data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

And yes, yt-dlp says it's not outdated:

yt-dlp -U
Latest version: stable@2024.04.09 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.04.09 from yt-dlp/yt-dlp)

19:33:05 [2948] Error while trying to download file: [Errno 36] File name too long: '/media/user/BT/xiao/mooopip/Introduction to Elementary Particle Physics WS22⧸23/General/Forum/23-01-29 Perimeter Scholars International/MWHRF7Tn86GW7tJsth4kgtM9W97gBTk4WdpRcN22w10D3q3phV1-WJV7CgVymW6fcM5J3-N9HbW8CMLSj1Qh82PW3T5YMX4jBKvSVkWycg5FRN8yW1YRTF61LX-D4W8307pW4LS7v5W53GHym3s65R7W54NjXp75d5JzW1mGKH74zrHK_W92ty0P4tVhjTVYd43c4kPGH5Vk3b0N756X7gW8xPNGb1c1HNRW58Ytts5NjQg5W3kf1CW9dMXgxVLKcGK4ZqStbW7jkKF_8t6PstW2gTsWF6d21J3W3jFFw_4gVwMtW3gzMCf7jMlYXW2WHK4S1N-BnvW2Xxshd1fPCgrW89hP_f84-sFNN7WcpDKF-CCQW6nr_G163H2__VsW9Sy5GDLxpW7vqY9m2R7nXdW7-PNJs7dJHCg36C41.html'
19:33:06 [2951] OSError(36, 'File name too long')

13:38:07 [10381] OperationalError('unable to open database file')
13:38:07 [10381] Error while trying to download file: unable to open database file
13:38:08 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (1) (#fsinformatik-gdp：hu-berlin-1).mp3.part'
13:38:08 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (2) (#fsinformatik-gdp：hu-berlin-2).mp3.part'
13:38:08 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (3) (#fsinformatik-gdp：hu-berlin-3).mp3.part'
13:38:08 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (4) (#fsinformatik-gdp：hu-berlin-4).mp3.part'
13:38:08 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (5) (#fsinformatik-gdp：hu-berlin-5).mp3.part'
13:38:09 [10407] yt-dlp Error: ERROR: unable to open for writing: [Errno 24] Too many open files: '/media/user/BT/xiao/mooopip/Grundlagen der Programmierung (WiSe 2022⧸23)/Allgemeines/Forum/22-10-31 Element-Raum/Element (6) (#fsinformatik-gdp：hu-berlin-6).mp3.part'
13:39:51 [9643] yt-dlp Error: ERROR: [requests] Unexpected error: RecursionError: maximum recursion depth exceeded while calling a Python object; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
13:39:51 [9643] yt-dlp Error: ERROR: [urllib] Unexpected error: RecursionError: maximum recursion depth exceeded while calling a Python object; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
13:39:51 [9643] yt-dlp Error: ERROR: Unable to handle request: Unsupported url scheme: "https" (websockets) + 2 unexpected error(s)

After this it just hangs for hours and I must kill it with ctrl+c

[C0D3D3V]

unable to open database file is concerning.

Especially the last block is very concerning.

Also the filename to long error is strange, I thought I fixed all filename bugs ^^.

All other yt-dlp errors are expected. And you will also get the errors if you run yt-dlp on that URL solo.

But please open new issues if you really need some. Because all these errors are external files and probably not that high prio for you.

[C0D3D3V]

Also if you really are interested in these files, than check what URLs are downloaded in each failed task, and open Issues on the yt-dlp repo.
Especially task 10407 and 9643 could be interesting for yt-dlp maintainers (if that are good URLs)

[aaannndddyyy]

It could be related to that it's putting the stuff onto an external hdd that's connected via usb3 , it could just be too slow and then lock up? internal simply doesn't have enough space, and ideally it should just work right? But I can kill it and restart it, if that helps and doesn't cause harm. Just wanna be sure I'm not making it all get into a mess. No idea about the too many files open and the max recursion.
Yeah, external files are probably less important, so that's not a pressing issue. Not high prio, but if you wish I can still open an issue there if you want to .
Would be nice if in the end, Moodle-DL could make like a small list of files (youtube, vimeo, cloud urls and from what course they were) that it has failed on so that one can go hunt for them manually. Or maybe it already does it...

[C0D3D3V]

Failed tasks are redownloaded on next run. And also printed at the end of download process.

You could try to store the database on a different location (on your main storage) and download files on external hdd.

[C0D3D3V]

you can cancel moodle-dl any time it tries its best to keep its state save. So it will continue where you have stopped

[C0D3D3V]

To store the database in another location use the misc_files_path option in config.json. Do not forget to move the database to the new location before next run.

Maybe we should think about not closing the database at all, while moodle-dl runs.

[aaannndddyyy]

Just did that. copied db to local, changed config.jason accordingly. and restarted. Thank you. :)