Slow on Raspberry Pi

[giampiero7]

Hi,
I'm running youtube-dl on a raspberry pi (Raspbian GNU/Linux 7 (wheezy))...
youtube-dl seems to be slow on each operation. Even just calling "youtube-dl --version" returns after about 7 seconds...
Anyone with the same problem?

[zackp30]

I'm getting the same problem as well.

[phihag]

Thank you very much for the report. I can reproduce this as well on my Rasperry Pi. time shows that the time is spent in userspace, most likely by the Python interpreter.

We should (and will) investigate:

• Would it help to load extractors (by far the most code) on demand?
• How much time is spent for loading the binary (i.e. add sys.exit(0)) at the first line. It may be beneficial to turn off zip compression.
• Can we improve the Python interpreter, or compile youtube-dl to binary code directly?

[giampiero7]

Hi,
Coming back on this after a while...
I updated to version 2014.06.26, but the issue is still there.
Tried with unzipping youtube-dl and running it with python -m youtube_dl... still slow...
So zip compression doesn't seem to be the issue here...

EDIT: still slow... but faster than when zipped... now youtube-dl --version completes in about 3 secs

Any progress on this?

[zackp30]

@philhag I reckon it would help a lot to load the extractors on demand, for the others, the speed change of youtube-dl would most likely be very small.

[samizdis]

This also occurs on Raspi with Arch Linux ARM. Version 2015.03.09 takes about 12s to run --version, and -g (what I'm mainly using it for) takes 22s.

[acpadhi]

Any updates on this yet ? It has been almost more than an year since the issue was raised, and no solution so far. I have checked Python doing other stuff pretty fast on the same Pi as it is expected.

So unlikely, there is any issue with python. May be there is something, that the new versions are trying to do which is causing this wait ~14 seconds issue.

[dstftw]

@acpadhi run strace -o yt-dl-strace.log -t -fff youtube-dl --verbose and post the yt-dl-strace.log file.

[acpadhi]

There were multiple ones created. Zipped them up into a single file. Please download the below link.
http://transworldsoft.net/pi/all_strace_logs.tar.gz

[acpadhi]

@dstftw any update on this ? Did the logs work for you or need any other info ?
I am planning to use this in one of my projects, and the delay it has now makes it just unusable. Tried with quvi, but quvi fails with some videos which have signature requirements. Some videos requiring the signature param work in quvi and some do not. those that fail in quvi do play in youtube-dl.

[SharkWipf]

I ran into this issue as well and decided to dive into the code to check it out.
I noticed that on every single execution of youtube-dl, regardless of what you're trying to do with it, it imports every single download module youtube-dl has, which in turn import their own dependencies.
By removing all imports I didn't need from this file and just leaving the 3 sites I need, I cut down the execution time from 8-9 seconds to 2 seconds.
Still a lot, even on the RasPi, but a whole lot lot less.

I haven't looked into the code much deeper, but it seems at least the bulk of the delay is caused by loading the ~900 downloading modules and their dependencies on every execution of youtube-dl.
Honestly I'm surprised it works as fast as it does on "proper" computers.
I'd suggest dynamically loading modules when necessary, rather than always loading everything you can load.
I haven't looked into where the remaining 2 seconds load time come from yet, but 2 seconds execution time works well enough for me for now.

[remitamine]

but it seems at least the bulk of the delay is caused by loading the ~900 downloading modules and their dependencies on every execution of youtube-dl.

i will post some ideas that may be can help to solve this problem:
first method(lazy loading of extractors):
by sepate the detection if the extractor(YoutubeDL.py will handle this indirectly) is needed and the extraction logic.

• generate meta-data about extractors and determine what extractor is needed from this metadata.
  for example saving the the extractor path and the regex and let YoutubeDl.py to load the metadata and decide when the extractor is needed(match the entred urls with the the _VALID_URL) what's not good about this is the duplication of information, the need for regeneration after every change in regex and it doesn't fit for every extrator(some extractors don't depend only on the _VALID_URL).
• separte the declaration of the extraction logic and the code to know if the extractor is needed or not(_VALID_URL and suitable) in different files.

or ...(some ideas by other developers).

the second method is adding an option to prevent youtube-dl from exiting so it will load only once and than let the user to interact with with opened process(i think this will be simpler to implement than the first two ideas and it doesn't involve a lot of changes).
i think it will be good for others also to share ideas may be we can find a good solution.

[yan12125]

I believe this problem is already fixed with #8497. Feel free to paste your timing results if it's still slow with lazy extractors.

Unzipping files may help, too.

[Dingo64]

The problem is still on ARM Debian 8. It takes about 10 seconds to see version number.
I installed using:

curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
chmod a+rx /usr/local/bin/youtube-dl

The version number is the newest: 2018.01.07
Next time I installed using pip and it shows version number in a 2-3 seconds. But pip doesn't offer up-to-date version and the newest I got from pip is 2017.12.31.

[yan12125]

@Dingo64: You may want a custom build with lazy extractors enabled: #8497 (comment)

[Dingo64]

Thanks, I'll try