Reducing performance impact of _socket.getaddrinfo #4070

ndren · 2022-06-14T16:41:56Z

Checklist

I'm requesting a feature unrelated to a specific site
I've looked through the README
I've verified that I'm running yt-dlp version 2022.05.18 (update instructions) or later (specify commit)
I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
I've read the guidelines for opening an issue

Description

I was downloading a video off YouTube with cProfile turned on (profile attached) and I have found that half the time for downloading was being spent in _socket.getaddrinfo, which was being called 25 times. I have tested running socket.getaddrinfo("youtube.com", 443) repeatedly and it does get rate limited after a while. Would there be a practical way to, for example, cache these outputs?

For context, I have attached the modified /bin/yt-dlp script and the full cProfile output. The key is the second column, tottime, which measures, in seconds, the amount of time spent in a specific function, without any subcalls to other functions affecting the results.

Verbose log

[debug] Command-line config: ['-vU', 'https://yewtu.be/watch?v=hdd1wuCQ3tY', '-f', '299+250']
[debug] User config "/home/$USER/.config/yt-dlp/config": ['-f', 'mp4[height>=720]']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version 2022.05.18 [b14d52355]
[debug] Python version 3.10.5 (CPython 64bit) - Linux-5.15.46-1-lts-x86_64-with-glibc2.35
[debug] Checking exe version: ffprobe -bsfs
[debug] Checking exe version: ffmpeg -bsfs
[debug] exe versions: ffmpeg 5.0.1 (setts), ffprobe 5.0.1
[debug] Optional libraries: certifi-2021.10.08, sqlite3-2.6.0
[debug] Proxy map: {}
Latest version: 2022.05.18, Current version: 2022.05.18
yt-dlp is up to date (2022.05.18)
[debug] [youtube] Extracting URL: https://yewtu.be/watch?v=hdd1wuCQ3tY
[youtube] hdd1wuCQ3tY: Downloading webpage
[youtube] hdd1wuCQ3tY: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] hdd1wuCQ3tY: Downloading 1 format(s): 299+250
WARNING: Requested formats are incompatible for merge and will be merged into mkv
[debug] Invoking http downloader on "https://rr4---sn-u5a3u5a3-h5oee.googlevideo.com/videoplayback?expire=1655245757&ei=XbeoYpnvE9b8gAeW84WABg&ip=92.19.187.126&id=o-AJ9f6a7lKMEvnI9uOl9wnKwybqNHKheSzS1X_otlCu1c&itag=299&source=youtube&requiressl=yes&mh=tw&mm=31%2C29&mn=sn-u5a3u5a3-h5oee%2Csn-aigl6nek&ms=au%2Crdu&mv=m&mvi=4&pl=21&initcwndbps=1671250&vprv=1&mime=video%2Fmp4&gir=yes&clen=217642702&dur=726.016&lmt=1655144589588864&mt=1655223898&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6319224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgPPjxWnpBM2NAYcrvWsO4RXqn9_45sapIVEdRcW42mAQCIQDr3h2STpmP9gnY2B7TskvGqPVzaa0lqXQBJlAOka0eeQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgL2SSKE4V_WFCqS3P9OnD1JEokc2pRvWjdKhG1c2FY7wCID12BFJkqq27_Y8BgE3co4zxbOdH3D2bEPDfene2OWlq"
[download] Destination: Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f299.mp4

[download]   0.0% of 207.56MiB at Unknown speed ETA Unknown
[download]   0.0% of 207.56MiB at  1.99MiB/s ETA 01:45     
[download]   0.0% of 207.56MiB at  3.44MiB/s ETA 01:00
[download]   0.0% of 207.56MiB at  5.54MiB/s ETA 00:37
[download]   0.0% of 207.56MiB at  4.34MiB/s ETA 00:47
[download]   0.0% of 207.56MiB at  5.17MiB/s ETA 00:40
[download]   0.1% of 207.56MiB at  6.59MiB/s ETA 00:31
[download]   0.1% of 207.56MiB at  8.83MiB/s ETA 00:23
[download]   0.2% of 207.56MiB at  9.74MiB/s ETA 00:21
[download]   0.5% of 207.56MiB at 13.58MiB/s ETA 00:15
[download]   1.0% of 207.56MiB at 14.56MiB/s ETA 00:14
[download]   1.9% of 207.56MiB at 15.29MiB/s ETA 00:13
[download]   3.9% of 207.56MiB at 15.92MiB/s ETA 00:12
[download]   4.8% of 207.56MiB at 15.16MiB/s ETA 00:13
[download]   4.8% of 207.56MiB at 596.46KiB/s ETA 05:44
[download]   4.8% of 207.56MiB at  1.27MiB/s ETA 02:36 
[download]   4.8% of 207.56MiB at  2.47MiB/s ETA 01:20
[download]   4.8% of 207.56MiB at  4.62MiB/s ETA 00:42
[download]   4.8% of 207.56MiB at  3.50MiB/s ETA 00:56
[download]   4.8% of 207.56MiB at  3.97MiB/s ETA 00:49
[download]   4.9% of 207.56MiB at  5.38MiB/s ETA 00:36
[download]   4.9% of 207.56MiB at  6.57MiB/s ETA 00:30
[download]   5.0% of 207.56MiB at  8.96MiB/s ETA 00:22
[download]   5.3% of 207.56MiB at 11.71MiB/s ETA 00:16
[download]   5.8% of 207.56MiB at 14.26MiB/s ETA 00:13
[download]   6.7% of 207.56MiB at 15.59MiB/s ETA 00:12
[download]   8.6% of 207.56MiB at 16.56MiB/s ETA 00:11
[download]   9.6% of 207.56MiB at 16.87MiB/s ETA 00:11
[download]   9.6% of 207.56MiB at 306.33KiB/s ETA 10:31
[download]   9.6% of 207.56MiB at 779.80KiB/s ETA 04:07
[download]   9.6% of 207.56MiB at  1.60MiB/s ETA 01:57 
[download]   9.6% of 207.56MiB at  2.97MiB/s ETA 01:03
[download]   9.6% of 207.56MiB at  3.80MiB/s ETA 00:49
[download]   9.6% of 207.56MiB at  4.64MiB/s ETA 00:40
[download]   9.7% of 207.56MiB at  1.68MiB/s ETA 01:51
[download]   9.7% of 207.56MiB at  2.96MiB/s ETA 01:03
[download]   9.8% of 207.56MiB at  5.10MiB/s ETA 00:36
[download]  10.1% of 207.56MiB at  7.94MiB/s ETA 00:23
[download]  10.6% of 207.56MiB at 10.85MiB/s ETA 00:17
[download]  11.5% of 207.56MiB at 13.35MiB/s ETA 00:13
[download]  13.5% of 207.56MiB at 15.19MiB/s ETA 00:11
[download]  14.4% of 207.56MiB at 15.65MiB/s ETA 00:11
[download]  14.4% of 207.56MiB at 897.18KiB/s ETA 03:26
[download]  14.4% of 207.56MiB at  1.77MiB/s ETA 01:41 
[download]  14.4% of 207.56MiB at  3.27MiB/s ETA 00:54
[download]  14.4% of 207.56MiB at  5.82MiB/s ETA 00:30
[download]  14.4% of 207.56MiB at  4.52MiB/s ETA 00:39
[download]  14.4% of 207.56MiB at  4.51MiB/s ETA 00:39
[download]  14.4% of 207.56MiB at  6.18MiB/s ETA 00:28
[download]  14.5% of 207.56MiB at  5.17MiB/s ETA 00:34
[download]  14.6% of 207.56MiB at  7.64MiB/s ETA 00:23
[download]  14.9% of 207.56MiB at 10.48MiB/s ETA 00:16
[download]  15.3% of 207.56MiB at 12.83MiB/s ETA 00:13
[download]  16.3% of 207.56MiB at 14.67MiB/s ETA 00:11
[download]  18.2% of 207.56MiB at 16.00MiB/s ETA 00:10
[download]  19.1% of 207.56MiB at 16.35MiB/s ETA 00:10
[download]  19.1% of 207.56MiB at 883.57KiB/s ETA 03:17
[download]  19.1% of 207.56MiB at  1.54MiB/s ETA 01:49 
[download]  19.1% of 207.56MiB at  2.66MiB/s ETA 01:03
[download]  19.1% of 207.56MiB at  4.89MiB/s ETA 00:34
[download]  19.1% of 207.56MiB at  3.36MiB/s ETA 00:50
[download]  19.2% of 207.56MiB at  4.28MiB/s ETA 00:39
[download]  19.2% of 207.56MiB at  5.81MiB/s ETA 00:28
[download]  19.2% of 207.56MiB at  8.26MiB/s ETA 00:20
[download]  19.4% of 207.56MiB at  8.15MiB/s ETA 00:20
[download]  19.6% of 207.56MiB at 11.30MiB/s ETA 00:14
[download]  20.1% of 207.56MiB at 13.85MiB/s ETA 00:11
[download]  21.0% of 207.56MiB at 15.24MiB/s ETA 00:10
[download]  23.0% of 207.56MiB at 16.36MiB/s ETA 00:09
[download]  23.9% of 207.56MiB at 16.70MiB/s ETA 00:09
[download]  23.9% of 207.56MiB at Unknown speed ETA 02:45
[download]  23.9% of 207.56MiB at  1.68MiB/s ETA 01:34   
[download]  23.9% of 207.56MiB at  2.97MiB/s ETA 00:53
[download]  23.9% of 207.56MiB at  5.11MiB/s ETA 00:31
[download]  23.9% of 207.56MiB at  4.74MiB/s ETA 00:33
[download]  23.9% of 207.56MiB at  4.44MiB/s ETA 00:35
[download]  24.0% of 207.56MiB at  6.21MiB/s ETA 00:25
[download]  24.0% of 207.56MiB at  7.84MiB/s ETA 00:20
[download]  24.2% of 207.56MiB at  8.48MiB/s ETA 00:18
[download]  24.4% of 207.56MiB at 11.29MiB/s ETA 00:13
[download]  24.9% of 207.56MiB at 13.54MiB/s ETA 00:11
[download]  25.8% of 207.56MiB at 15.30MiB/s ETA 00:10
[download]  27.8% of 207.56MiB at 16.09MiB/s ETA 00:09
[download]  28.6% of 207.56MiB at 16.62MiB/s ETA 00:08
[download]  28.6% of 207.56MiB at 713.20KiB/s ETA 03:37
[download]  28.6% of 207.56MiB at  1.41MiB/s ETA 01:45 
[download]  28.6% of 207.56MiB at  2.52MiB/s ETA 00:59
[download]  28.6% of 207.56MiB at  4.61MiB/s ETA 00:32
[download]  28.6% of 207.56MiB at  3.60MiB/s ETA 00:41
[download]  28.7% of 207.56MiB at  4.73MiB/s ETA 00:31
[download]  28.7% of 207.56MiB at  5.60MiB/s ETA 00:26
[download]  28.7% of 207.56MiB at  8.07MiB/s ETA 00:18
[download]  28.9% of 207.56MiB at 10.84MiB/s ETA 00:13
[download]  29.1% of 207.56MiB at 12.63MiB/s ETA 00:11
[download]  29.6% of 207.56MiB at 14.46MiB/s ETA 00:10
[download]  30.5% of 207.56MiB at 15.76MiB/s ETA 00:09
[download]  32.5% of 207.56MiB at 16.71MiB/s ETA 00:08
[download]  33.3% of 207.56MiB at 17.06MiB/s ETA 00:08
[download]  33.3% of 207.56MiB at 546.85KiB/s ETA 04:21
[download]  33.3% of 207.56MiB at  1.21MiB/s ETA 01:55 
[download]  33.3% of 207.56MiB at  2.13MiB/s ETA 01:05
[download]  33.3% of 207.56MiB at  3.98MiB/s ETA 00:34
[download]  33.3% of 207.56MiB at  3.38MiB/s ETA 00:41
[download]  33.3% of 207.56MiB at  4.40MiB/s ETA 00:31
[download]  33.4% of 207.56MiB at  6.02MiB/s ETA 00:23
[download]  33.4% of 207.56MiB at  7.83MiB/s ETA 00:17
[download]  33.5% of 207.56MiB at 10.58MiB/s ETA 00:13
[download]  33.8% of 207.56MiB at 11.81MiB/s ETA 00:11
[download]  34.3% of 207.56MiB at 14.04MiB/s ETA 00:09
[download]  35.2% of 207.56MiB at 15.50MiB/s ETA 00:08
[download]  37.2% of 207.56MiB at 16.55MiB/s ETA 00:07
[download]  37.9% of 207.56MiB at 16.76MiB/s ETA 00:07
[download]  37.9% of 207.56MiB at Unknown speed ETA Unknown
[download]  37.9% of 207.56MiB at  1.99MiB/s ETA 01:05     
[download]  37.9% of 207.56MiB at  3.18MiB/s ETA 00:40
[download]  37.9% of 207.56MiB at  5.16MiB/s ETA 00:25
[download]  37.9% of 207.56MiB at  4.13MiB/s ETA 00:31
[download]  38.0% of 207.56MiB at  4.94MiB/s ETA 00:26
[download]  38.0% of 207.56MiB at  5.78MiB/s ETA 00:22
[download]  38.1% of 207.56MiB at  7.54MiB/s ETA 00:17
[download]  38.2% of 207.56MiB at 10.18MiB/s ETA 00:12
[download]  38.4% of 207.56MiB at 12.81MiB/s ETA 00:09
[download]  38.9% of 207.56MiB at 14.80MiB/s ETA 00:08
[download]  39.9% of 207.56MiB at 15.89MiB/s ETA 00:07
[download]  41.8% of 207.56MiB at 16.68MiB/s ETA 00:07
[download]  42.5% of 207.56MiB at 16.93MiB/s ETA 00:07
[download]  42.5% of 207.56MiB at 909.83KiB/s ETA 02:17
[download]  42.5% of 207.56MiB at  1.60MiB/s ETA 01:15 
[download]  42.5% of 207.56MiB at  2.71MiB/s ETA 00:44
[download]  42.5% of 207.56MiB at  4.52MiB/s ETA 00:26
[download]  42.5% of 207.56MiB at  3.95MiB/s ETA 00:30
[download]  42.5% of 207.56MiB at  4.37MiB/s ETA 00:27
[download]  42.6% of 207.56MiB at  5.16MiB/s ETA 00:23
[download]  42.6% of 207.56MiB at  7.24MiB/s ETA 00:16
[download]  42.8% of 207.56MiB at  9.83MiB/s ETA 00:12
[download]  43.0% of 207.56MiB at 12.84MiB/s ETA 00:09
[download]  43.5% of 207.56MiB at 14.47MiB/s ETA 00:08
[download]  44.4% of 207.56MiB at 15.84MiB/s ETA 00:07
[download]  46.4% of 207.56MiB at 16.76MiB/s ETA 00:06
[download]  47.1% of 207.56MiB at 17.06MiB/s ETA 00:06
[download]  47.1% of 207.56MiB at 788.11KiB/s ETA 02:24
[download]  47.1% of 207.56MiB at  1.61MiB/s ETA 01:08 
[download]  47.1% of 207.56MiB at  3.11MiB/s ETA 00:35
[download]  47.1% of 207.56MiB at  5.68MiB/s ETA 00:19
[download]  47.1% of 207.56MiB at  2.42MiB/s ETA 00:45
[download]  47.1% of 207.56MiB at  3.89MiB/s ETA 00:28
[download]  47.2% of 207.56MiB at  5.19MiB/s ETA 00:21
[download]  47.2% of 207.56MiB at  7.02MiB/s ETA 00:15
[download]  47.3% of 207.56MiB at  9.96MiB/s ETA 00:10
[download]  47.6% of 207.56MiB at 12.16MiB/s ETA 00:08
[download]  48.1% of 207.56MiB at 14.80MiB/s ETA 00:07
[download]  49.0% of 207.56MiB at 15.57MiB/s ETA 00:06
[download]  50.9% of 207.56MiB at 15.93MiB/s ETA 00:06
[download]  51.7% of 207.56MiB at 16.22MiB/s ETA 00:06
[download]  51.7% of 207.56MiB at Unknown speed ETA 01:43
[download]  51.7% of 207.56MiB at  1.93MiB/s ETA 00:52   
[download]  51.7% of 207.56MiB at  3.46MiB/s ETA 00:29
[download]  51.7% of 207.56MiB at  6.01MiB/s ETA 00:16
[download]  51.7% of 207.56MiB at  1.63MiB/s ETA 01:01
[download]  51.7% of 207.56MiB at  3.19MiB/s ETA 00:31
[download]  51.7% of 207.56MiB at  3.94MiB/s ETA 00:25
[download]  51.8% of 207.56MiB at  5.93MiB/s ETA 00:16
[download]  51.9% of 207.56MiB at  8.58MiB/s ETA 00:11
[download]  52.2% of 207.56MiB at 11.00MiB/s ETA 00:09
[download]  52.6% of 207.56MiB at 13.22MiB/s ETA 00:07
[download]  53.6% of 207.56MiB at 15.36MiB/s ETA 00:06
[download]  55.5% of 207.56MiB at 15.92MiB/s ETA 00:05
[download]  56.5% of 207.56MiB at 16.53MiB/s ETA 00:05
[download]  56.5% of 207.56MiB at 370.39KiB/s ETA 04:15
[download]  56.5% of 207.56MiB at 789.84KiB/s ETA 01:57
[download]  56.5% of 207.56MiB at  1.56MiB/s ETA 00:57 
[download]  56.5% of 207.56MiB at  2.91MiB/s ETA 00:31
[download]  56.5% of 207.56MiB at  3.27MiB/s ETA 00:27
[download]  56.5% of 207.56MiB at  4.06MiB/s ETA 00:22
[download]  56.6% of 207.56MiB at  5.24MiB/s ETA 00:17
[download]  56.6% of 207.56MiB at  6.21MiB/s ETA 00:14
[download]  56.7% of 207.56MiB at  8.13MiB/s ETA 00:11
[download]  57.0% of 207.56MiB at 11.38MiB/s ETA 00:07
[download]  57.5% of 207.56MiB at 13.93MiB/s ETA 00:06
[download]  58.4% of 207.56MiB at 15.06MiB/s ETA 00:05
[download]  60.4% of 207.56MiB at 15.03MiB/s ETA 00:05
[download]  61.1% of 207.56MiB at 15.14MiB/s ETA 00:05
[download]  61.1% of 207.56MiB at 698.82KiB/s ETA 02:00
[download]  61.1% of 207.56MiB at  1.40MiB/s ETA 00:58 
[download]  61.1% of 207.56MiB at  2.69MiB/s ETA 00:30
[download]  61.1% of 207.56MiB at  4.88MiB/s ETA 00:16
[download]  61.1% of 207.56MiB at  1.69MiB/s ETA 00:47
[download]  61.1% of 207.56MiB at  2.93MiB/s ETA 00:27
[download]  61.1% of 207.56MiB at  3.82MiB/s ETA 00:21
[download]  61.2% of 207.56MiB at  5.25MiB/s ETA 00:15
[download]  61.3% of 207.56MiB at  8.22MiB/s ETA 00:09
[download]  61.6% of 207.56MiB at 10.33MiB/s ETA 00:07
[download]  62.0% of 207.56MiB at 12.35MiB/s ETA 00:06
[download]  63.0% of 207.56MiB at 14.21MiB/s ETA 00:05
[download]  64.9% of 207.56MiB at 15.27MiB/s ETA 00:04
[download]  65.7% of 207.56MiB at 15.83MiB/s ETA 00:04
[download]  65.7% of 207.56MiB at Unknown speed ETA Unknown
[download]  65.7% of 207.56MiB at  2.04MiB/s ETA 00:35     
[download]  65.7% of 207.56MiB at  3.63MiB/s ETA 00:19
[download]  65.7% of 207.56MiB at  6.35MiB/s ETA 00:11
[download]  65.8% of 207.56MiB at  1.84MiB/s ETA 00:38
[download]  65.8% of 207.56MiB at  2.50MiB/s ETA 00:28
[download]  65.8% of 207.56MiB at  3.64MiB/s ETA 00:19
[download]  65.9% of 207.56MiB at  5.33MiB/s ETA 00:13
[download]  66.0% of 207.56MiB at  8.25MiB/s ETA 00:08
[download]  66.2% of 207.56MiB at 10.54MiB/s ETA 00:06
[download]  66.7% of 207.56MiB at 12.11MiB/s ETA 00:05
[download]  67.7% of 207.56MiB at 14.46MiB/s ETA 00:04
[download]  69.6% of 207.56MiB at 14.24MiB/s ETA 00:04
[download]  70.6% of 207.56MiB at 14.85MiB/s ETA 00:04
[download]  70.6% of 207.56MiB at 965.98KiB/s ETA 01:05
[download]  70.6% of 207.56MiB at  1.85MiB/s ETA 00:33 
[download]  70.6% of 207.56MiB at  3.35MiB/s ETA 00:18
[download]  70.6% of 207.56MiB at  5.94MiB/s ETA 00:10
[download]  70.6% of 207.56MiB at  3.83MiB/s ETA 00:16
[download]  70.6% of 207.56MiB at  4.84MiB/s ETA 00:12
[download]  70.6% of 207.56MiB at  5.51MiB/s ETA 00:11
[download]  70.7% of 207.56MiB at  7.70MiB/s ETA 00:07
[download]  70.8% of 207.56MiB at 10.49MiB/s ETA 00:05
[download]  71.0% of 207.56MiB at 12.05MiB/s ETA 00:04
[download]  71.5% of 207.56MiB at 13.20MiB/s ETA 00:04
[download]  72.5% of 207.56MiB at 14.64MiB/s ETA 00:03
[download]  74.4% of 207.56MiB at 12.49MiB/s ETA 00:04
[download]  75.3% of 207.56MiB at 13.18MiB/s ETA 00:03
[download]  75.3% of 207.56MiB at 302.18KiB/s ETA 02:54
[download]  75.3% of 207.56MiB at 721.37KiB/s ETA 01:13
[download]  75.3% of 207.56MiB at  1.43MiB/s ETA 00:35 
[download]  75.3% of 207.56MiB at  2.81MiB/s ETA 00:18
[download]  75.3% of 207.56MiB at  3.32MiB/s ETA 00:15
[download]  75.3% of 207.56MiB at  4.02MiB/s ETA 00:12
[download]  75.4% of 207.56MiB at  5.58MiB/s ETA 00:09
[download]  75.4% of 207.56MiB at  7.86MiB/s ETA 00:06
[download]  75.5% of 207.56MiB at  9.22MiB/s ETA 00:05
[download]  75.8% of 207.56MiB at 11.62MiB/s ETA 00:04
[download]  76.3% of 207.56MiB at 13.45MiB/s ETA 00:03
[download]  77.2% of 207.56MiB at 14.96MiB/s ETA 00:03
[download]  79.2% of 207.56MiB at 16.34MiB/s ETA 00:02
[download]  80.1% of 207.56MiB at 16.50MiB/s ETA 00:02
[download]  80.1% of 207.56MiB at 853.54KiB/s ETA 00:50
[download]  80.1% of 207.56MiB at  1.58MiB/s ETA 00:26 
[download]  80.1% of 207.56MiB at  3.14MiB/s ETA 00:13
[download]  80.1% of 207.56MiB at  5.85MiB/s ETA 00:07
[download]  80.1% of 207.56MiB at  4.25MiB/s ETA 00:09
[download]  80.1% of 207.56MiB at  4.94MiB/s ETA 00:08
[download]  80.1% of 207.56MiB at  6.56MiB/s ETA 00:06
[download]  80.2% of 207.56MiB at  9.01MiB/s ETA 00:04
[download]  80.3% of 207.56MiB at  9.63MiB/s ETA 00:04
[download]  80.6% of 207.56MiB at 11.87MiB/s ETA 00:03
[download]  81.0% of 207.56MiB at 13.54MiB/s ETA 00:02
[download]  82.0% of 207.56MiB at 14.82MiB/s ETA 00:02
[download]  83.9% of 207.56MiB at 16.21MiB/s ETA 00:02
[download]  84.8% of 207.56MiB at 16.53MiB/s ETA 00:01
[download]  84.8% of 207.56MiB at 601.94KiB/s ETA 00:54
[download]  84.8% of 207.56MiB at  1.14MiB/s ETA 00:27 
[download]  84.8% of 207.56MiB at  2.17MiB/s ETA 00:14
[download]  84.8% of 207.56MiB at  4.01MiB/s ETA 00:07
[download]  84.8% of 207.56MiB at  3.95MiB/s ETA 00:08
[download]  84.8% of 207.56MiB at  4.94MiB/s ETA 00:06
[download]  84.8% of 207.56MiB at  5.99MiB/s ETA 00:05
[download]  84.9% of 207.56MiB at  6.58MiB/s ETA 00:04
[download]  85.0% of 207.56MiB at  8.59MiB/s ETA 00:03
[download]  85.3% of 207.56MiB at 10.64MiB/s ETA 00:02
[download]  85.7% of 207.56MiB at 14.07MiB/s ETA 00:02
[download]  86.7% of 207.56MiB at 15.60MiB/s ETA 00:01
[download]  88.6% of 207.56MiB at 16.56MiB/s ETA 00:01
[download]  89.4% of 207.56MiB at 16.86MiB/s ETA 00:01
[download]  89.4% of 207.56MiB at 578.37KiB/s ETA 00:39
[download]  89.4% of 207.56MiB at  1.25MiB/s ETA 00:17 
[download]  89.4% of 207.56MiB at  2.45MiB/s ETA 00:09
[download]  89.4% of 207.56MiB at  4.61MiB/s ETA 00:04
[download]  89.4% of 207.56MiB at  3.72MiB/s ETA 00:05
[download]  89.4% of 207.56MiB at  4.84MiB/s ETA 00:04
[download]  89.4% of 207.56MiB at  5.90MiB/s ETA 00:03
[download]  89.5% of 207.56MiB at  8.83MiB/s ETA 00:02
[download]  89.6% of 207.56MiB at 11.68MiB/s ETA 00:01
[download]  89.8% of 207.56MiB at 13.55MiB/s ETA 00:01
[download]  90.3% of 207.56MiB at 15.38MiB/s ETA 00:01
[download]  91.3% of 207.56MiB at 16.11MiB/s ETA 00:01
[download]  93.2% of 207.56MiB at 16.70MiB/s ETA 00:00
[download]  94.1% of 207.56MiB at 17.05MiB/s ETA 00:00
[download]  94.1% of 207.56MiB at Unknown speed ETA Unknown
[download]  94.1% of 207.56MiB at  1.88MiB/s ETA 00:06     
[download]  94.1% of 207.56MiB at  3.59MiB/s ETA 00:03
[download]  94.1% of 207.56MiB at  6.36MiB/s ETA 00:01
[download]  94.1% of 207.56MiB at  4.34MiB/s ETA 00:02
[download]  94.1% of 207.56MiB at  4.37MiB/s ETA 00:02
[download]  94.2% of 207.56MiB at  5.66MiB/s ETA 00:02
[download]  94.2% of 207.56MiB at  7.76MiB/s ETA 00:01
[download]  94.4% of 207.56MiB at 10.18MiB/s ETA 00:01
[download]  94.6% of 207.56MiB at  8.94MiB/s ETA 00:01
[download]  95.1% of 207.56MiB at 11.56MiB/s ETA 00:00
[download]  96.0% of 207.56MiB at 10.25MiB/s ETA 00:00
[download]  98.0% of 207.56MiB at 12.99MiB/s ETA 00:00
[download]  98.9% of 207.56MiB at 13.68MiB/s ETA 00:00
[download]  98.9% of 207.56MiB at Unknown speed ETA Unknown
[download]  98.9% of 207.56MiB at  2.69MiB/s ETA 00:00     
[download]  98.9% of 207.56MiB at  4.33MiB/s ETA 00:00
[download]  98.9% of 207.56MiB at  7.41MiB/s ETA 00:00
[download]  98.9% of 207.56MiB at  3.84MiB/s ETA 00:00
[download]  99.0% of 207.56MiB at  4.13MiB/s ETA 00:00
[download]  99.0% of 207.56MiB at  5.52MiB/s ETA 00:00
[download]  99.0% of 207.56MiB at  6.90MiB/s ETA 00:00
[download]  99.2% of 207.56MiB at 10.05MiB/s ETA 00:00
[download]  99.4% of 207.56MiB at 12.20MiB/s ETA 00:00
[download]  99.9% of 207.56MiB at 14.68MiB/s ETA 00:00
[download] 100% of 207.56MiB at 15.06MiB/s ETA 00:00  
[download] 100% of 207.56MiB in 00:30               
[debug] Invoking http downloader on "https://rr4---sn-u5a3u5a3-h5oee.googlevideo.com/videoplayback?expire=1655245757&ei=XbeoYpnvE9b8gAeW84WABg&ip=92.19.187.126&id=o-AJ9f6a7lKMEvnI9uOl9wnKwybqNHKheSzS1X_otlCu1c&itag=250&source=youtube&requiressl=yes&mh=tw&mm=31%2C29&mn=sn-u5a3u5a3-h5oee%2Csn-aigl6nek&ms=au%2Crdu&mv=m&mvi=4&pl=21&initcwndbps=1671250&vprv=1&mime=audio%2Fwebm&gir=yes&clen=6353610&dur=726.061&lmt=1655144821645923&mt=1655223898&fvip=1&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6318224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAOSPcSxVBsJN-o6Qco956wH0oVC42nVjFNuoqBXnar4kAiEA4d5XmzuuGiAEh9SuzBz62cYG0TBpalt6MzuZ8ku3IYQ%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgL2SSKE4V_WFCqS3P9OnD1JEokc2pRvWjdKhG1c2FY7wCID12BFJkqq27_Y8BgE3co4zxbOdH3D2bEPDfene2OWlq"
[download] Destination: Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f250.webm

[download]   0.0% of 6.06MiB at 870.91KiB/s ETA 00:07
[download]   0.0% of 6.06MiB at  1.82MiB/s ETA 00:03 
[download]   0.1% of 6.06MiB at  3.51MiB/s ETA 00:01
[download]   0.2% of 6.06MiB at  6.51MiB/s ETA 00:00
[download]   0.5% of 6.06MiB at  3.74MiB/s ETA 00:01
[download]   1.0% of 6.06MiB at  4.53MiB/s ETA 00:01
[download]   2.0% of 6.06MiB at  5.68MiB/s ETA 00:01
[download]   4.1% of 6.06MiB at  7.83MiB/s ETA 00:00
[download]   8.2% of 6.06MiB at 10.37MiB/s ETA 00:00
[download]  16.5% of 6.06MiB at 12.86MiB/s ETA 00:00
[download]  33.0% of 6.06MiB at 15.28MiB/s ETA 00:00
[download]  66.0% of 6.06MiB at 16.29MiB/s ETA 00:00
[download] 100% of 6.06MiB at  9.85MiB/s ETA 00:00  
[download] 100% of 6.06MiB in 00:05               
[Merger] Merging formats into "Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f299.mp4' -i 'file:Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f250.webm' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].temp.mkv'
Deleting original file Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f250.webm (pass -k to keep)
Deleting original file Fixing the DMCA is making all the right people mad [hdd1wuCQ3tY].f299.mp4 (pass -k to keep)

ndren · 2022-06-14T16:46:27Z

Modified yt-dlp:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
import sys
import cProfile
from yt_dlp import main
with cProfile.Profile() as prof:
    sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
    try:
        main()
    except:
        print("Exit!")
prof.print_stats()

I have attached the output as well. See in particular column 2, which has raw time taken excluding any subcalls to other functions.
cProfile.txt

pukkandan · 2022-06-14T17:06:08Z

I have tested running socket.getaddrinfo("youtube.com", 443) repeatedly and it does get rate limited after a while.

How many times did you do this? I can't reproduce even after 1M calls

Testing code:

import socket, timeit

N = 1000; LIM = 1_000_000
for i in range(LIM//N):
    print(f'{(i + 1) * N:>10d}', timeit.timeit(
        'socket.getaddrinfo("youtube.com", 443)',
        globals=globals(), number=N))

ndren · 2022-06-14T18:11:35Z

At the time, after about every five calls, the sixth call takes 5 seconds to run. Here are a few timings, in seconds:
[0.04784846305847168, 0.05150747299194336, 0.0475771427154541, 0.04690694808959961, 0.04703855514526367, 0.04845619201660156, 0.04768562316894531, 5.027868270874023, 0.049422502517700195, 0.05527448654174805, 0.04706525802612305, 0.05326032638549805, 0.05347108840942383, 0.05081009864807129, 0.05355954170227051, 5.004327774047852, 0.04681515693664551, 0.04791736602783203, 0.05260920524597168, 0.04738640785217285, 0.0494229793548584, 5.030548810958862, 0.05075502395629883]

ndren · 2022-06-14T18:20:05Z

I have found that it is probably my ISP that is throttling these requests, as this happens to all hosts... Should I close the issue?

pukkandan · 2022-06-14T18:25:57Z

Windows:

>>> import socket, timeit
>>> N = 1; LIM = 20
>>>
>>> for i in range(LIM//N):
...     print(f'{(i + 1) * N:>10d}', timeit.timeit(
...         'socket.getaddrinfo("youtube.com", 443)',
...         globals=globals(), number=N))
...
         1 0.030426299897953868
         2 0.0006706998683512211
         3 0.0003217000048607588
         4 0.00029360014013946056
         5 0.0002962998114526272
         6 0.00029530003666877747
         7 0.0002888999879360199
         8 0.0002865998540073633
         9 0.000282099936157465
        10 0.0003408000338822603
        11 0.00032909982837736607
        12 0.0002951999194920063
        13 0.0003035000991076231
        14 0.0002933000214397907
        15 0.00038430001586675644
        16 0.0003557000309228897
        17 0.0004267999902367592
        18 0.001039199996739626
        19 0.0005477000959217548
        20 0.0003512999974191189

Linux:

>>> import socket, timeit
>>> N = 1; LIM = 20
>>>
>>> for i in range(LIM//N):
...     print(f'{(i + 1) * N:>10d}', timeit.timeit(
...         'socket.getaddrinfo("youtube.com", 443)',
...         globals=globals(), number=N))
...
         1 0.019489400001475587
         2 0.0036687999963760376
         3 0.0029771999979857355
         4 0.003344099997775629
         5 0.002450100000714883
         6 0.016608899997663684
         7 0.0029057999927317724
         8 0.004744399993796833
         9 0.003114899998763576
        10 0.0027027999894926324
        11 0.004003299996838905
        12 0.014604500000132248
        13 0.004866500006755814
        14 0.003497900004731491
        15 0.0024579000019002706
        16 0.0023951000039232895
        17 0.0036401999968802556
        18 0.0034352999937254936
        19 0.0025858000008156523
        20 0.002880100000766106

On linux, I too get a slight bump in the timing for some of the requests. I suspect that both OSes are already caching it, but linux invalidates the cache it after every few requests. But still, I don't get the 100x slowdown like you do. My hypothesis is that either your network or OS has some issue that is slowing it down. Can you test it on another device/network?

pukkandan · 2022-06-14T18:29:06Z

I have found that it is probably my ISP that is throttling these requests, as this happens to all hosts... Should I close the issue?

In that case, there is really nothing we can do. In theory, we could cache it like you suggest, but you can't expect every program you use to implement DNS caching separately. You should instead try to figure out what exactly is causing the slowdown

ndren · 2022-06-14T18:45:42Z

Found it, it was my custom public DNS resolver. Thank you!

coletdjnz · 2022-06-14T19:55:47Z

Possibly related: ytdl-org/youtube-dl#13734

yt-dl/p opens a new connection for each request which probably explains why there are so many DNS requests. #3668 should improve this by introducing persistent connections.

Adds support for HTTPS proxies and persistent connections (keep-alive) Closes #1890 Resolves #4070 Resolves ytdl-org/youtube-dl#32549 Resolves ytdl-org/youtube-dl#14523 Resolves ytdl-org/youtube-dl#13734 Authored by: coletdjnz, Grub4K, bashonly

Adds support for HTTPS proxies and persistent connections (keep-alive) Closes yt-dlp#1890 Resolves yt-dlp#4070 Resolves ytdl-org/youtube-dl#32549 Resolves ytdl-org/youtube-dl#14523 Resolves ytdl-org/youtube-dl#13734 Authored by: coletdjnz, Grub4K, bashonly

ndren added enhancement New feature or request triage Untriaged issue labels Jun 14, 2022

ndren closed this as completed Jun 14, 2022

pukkandan added external issue Issue with an external tool and removed triage Untriaged issue labels Jun 14, 2022

coletdjnz mentioned this issue Jun 27, 2022

[rh:requests] Add support for requests #3668

Merged

9 tasks

coletdjnz linked a pull request Jun 27, 2022 that will close this issue

[rh:requests] Add support for requests #3668

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reducing performance impact of _socket.getaddrinfo #4070

Reducing performance impact of _socket.getaddrinfo #4070

ndren commented Jun 14, 2022 •

edited

ndren commented Jun 14, 2022

pukkandan commented Jun 14, 2022 •

edited

ndren commented Jun 14, 2022

ndren commented Jun 14, 2022

pukkandan commented Jun 14, 2022 •

edited

pukkandan commented Jun 14, 2022

ndren commented Jun 14, 2022

coletdjnz commented Jun 14, 2022

Reducing performance impact of _socket.getaddrinfo #4070

Reducing performance impact of _socket.getaddrinfo #4070

Comments

ndren commented Jun 14, 2022 • edited

Checklist

Description

Verbose log

ndren commented Jun 14, 2022

pukkandan commented Jun 14, 2022 • edited

ndren commented Jun 14, 2022

ndren commented Jun 14, 2022

pukkandan commented Jun 14, 2022 • edited

pukkandan commented Jun 14, 2022

ndren commented Jun 14, 2022

coletdjnz commented Jun 14, 2022

ndren commented Jun 14, 2022 •

edited

pukkandan commented Jun 14, 2022 •

edited

pukkandan commented Jun 14, 2022 •

edited