Help me understand this performance issue with YoutubeDL.py?

[anohren]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

Background: I'm trying to understand the source of a performance issue in a Kodi addon that calls YoutubeDL.py (2020.05.08). It currently takes about 55 seconds to parse a Youtube playlist with 29 entries, on a Raspberry Pi 3B (running OSMC).

I profiled the addon with cProfile, and found out that 52 of 55 seconds were spent in YoutubeDL.py:771(extract_info). I then narrowed down the profiling to only that function call, with this result (sorted by time spent in itself and subfunctions):

Thu May 28 16:18:15 2020    profiling_stats

         2944951 function calls (2792145 primitive calls) in 56.276 seconds

   Ordered by: cumulative time
   List reduced from 690 to 30 due to restriction <30>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     30/1    0.024    0.001   56.328   56.328 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/YoutubeDL.py:771(extract_info)
     59/1    0.013    0.000   44.156   44.156 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/YoutubeDL.py:841(process_ie_result)
       30    0.080    0.003   39.877    1.329 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/common.py:524(extract)
       29    2.115    0.073   38.970    1.344 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/youtube.py:1683(_real_extract)
       31    0.003    0.000   23.901    0.771 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/youtube.py:267(_download_webpage_handle)
       31    0.003    0.000   23.898    0.771 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/common.py:650(_download_webpage_handle)
       31    0.003    0.000   23.518    0.759 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/common.py:595(_request_webpage)
       31    0.001    0.000   23.427    0.756 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/YoutubeDL.py:2234(urlopen)
       31    0.009    0.000   23.399    0.755 /usr/lib/python2.7/urllib2.py:411(open)
     2272    0.033    0.000   19.718    0.009 /usr/lib/python2.7/ssl.py:759(recv)
     2272    0.033    0.000   19.682    0.009 /usr/lib/python2.7/ssl.py:642(read)
     2272   19.646    0.009   19.646    0.009 {method 'read' of '_ssl._SSLSocket' objects}
     3009    0.106    0.000   19.599    0.007 /usr/lib/python2.7/socket.py:410(readline)
       31    0.027    0.001   18.656    0.602 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/utils.py:2624(http_response)
  5142/31    0.205    0.000   18.196    0.587 /usr/lib/python2.7/socket.py:340(read)
      331    0.007    0.000   18.166    0.055 /usr/lib/python2.7/httplib.py:564(read)
      300    0.119    0.000   18.159    0.061 /usr/lib/python2.7/httplib.py:609(_read_chunked)
    15740    0.400    0.000   13.619    0.001 /usr/lib/python2.7/re.py:230(_compile)
     1109    0.100    0.000   13.189    0.012 /usr/lib/python2.7/sre_compile.py:567(compile)
     2372    0.037    0.000   11.941    0.005 /usr/lib/python2.7/re.py:192(compile)
      999    0.046    0.000   11.719    0.012 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/extractor/common.py:402(suitable)
1240/1109    0.074    0.000    9.514    0.009 /usr/lib/python2.7/sre_parse.py:706(parse)
5651/1240    0.307    0.000    9.367    0.008 /usr/lib/python2.7/sre_parse.py:317(_parse_sub)
8150/1251    3.397    0.000    9.325    0.007 /usr/lib/python2.7/sre_parse.py:395(_parse)
       31    0.002    0.000    4.455    0.144 /usr/lib/python2.7/urllib2.py:439(_open)
       62    0.002    0.000    4.453    0.072 /usr/lib/python2.7/urllib2.py:399(_call_chain)
       31    0.003    0.000    4.450    0.144 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/utils.py:2724(https_open)
       31    0.010    0.000    4.446    0.143 /usr/lib/python2.7/urllib2.py:1152(do_open)
       29    0.098    0.003    4.435    0.153 /home/osmc/.kodi/addons/plugin.video.sendtokodi/youtube_dl/YoutubeDL.py:1412(process_video_result)
     1249    0.026    0.000    4.044    0.003 /usr/lib/python2.7/re.py:143(search)

I'm working under the assumption that 56 seconds for YoutubeDL.py:771(extract_info) is unreasonably long.

I'm wondering if there is an obvious cause for this performance, or if someone could point me in the right direction toward figuring it out?

A simplified example of what the call looks like:

ydl_opts = {
    'format': 'best'
}

url = "https://www.youtube.com/playlist?list=PLHs6JH9ueCokcP2hje8PVPPqjucNJFUvO" 
ydl = YoutubeDL(ydl_opts)
ydl.add_default_info_extractors()

with ydl:
    result = ydl.extract_info(url, download=False)

I'm hoping to make this an order of magnitude faster somehow.

[dstftw]

It's a reasonable time since this involves extracting all individual videos including downloading webpages and data needed for individual video extraction.

[anohren]

Thanks for the evaluation. With that info I could determine that my use case requires extract_flat.