Cannot download long videos from Youtube because of DNS flood

[codedokode]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.03.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.03.14

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

Didn't check because this is irrelevant to the issue.

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

15:05 tmp: ./youtube-dl -v --hls-prefer-native --format 300 --ffmpeg-location /d/port/libav-11.3/usr/bin/  -o v6.avi --no-mtime 'https://www.youtube.com/watch?v=QL0nLqDdKdI'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--hls-prefer-native', u'--format', u'300', u'--ffmpeg-location', u'/d/port/libav-11.3/usr/bin/', u'-o', u'v6.avi', u'--no-mtime', u'https://www.youtube.com/watch?v=QL0nLqDdKdI']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.03.14
[debug] Python version 2.7.10 (CPython) - CYGWIN_NT-5.1-2.5.2-0.297-5-3-i686-32bit
[debug] exe versions: avconv 11.3, avprobe 11.3
[debug] Proxy map: {}
[youtube] QL0nLqDdKdI: Downloading webpage
[youtube] QL0nLqDdKdI: Downloading video info webpage
[youtube] QL0nLqDdKdI: Extracting video information
[youtube] QL0nLqDdKdI: Downloading m3u8 information
WARNING: [youtube] QL0nLqDdKdI: Skipping DASH manifest: ExtractorError(u'Cannot decrypt si
gnature without player_url; please report this issue on https://yt-dl.org/bug . Make sure
you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-
dl with the --verbose flag and include its complete output.',)
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/hls_playlis
t/id/QL0nLqDdKdI.1/itag/300/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/
cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D298/hls_chunk_h
ost/r4---sn-1uxaxjvh-axql.googlevideo.com/ei/V6mvWo...2ragBA/force_finished/1/playl
ist_type/CLEAN/gcr/ru/initcwndbps/10830/mm/32/mn/sn-1uxaxjvh-axql/ms/lv/mv/m/pl/24/dover/1
0/keepalive/yes/mt/1521461539/ip/x.y.z.w/ipbits/0/expire/1521483192/sparams/ip,ipbits
,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,
ei,force_finished,playlist_type,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/xyzwxyzwxyzw8146BB164EBFF61665.70AB5D835FFCDDA56EDF29F1/key/dg_yt0/playlist/index
.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 2880
[download] Destination: v6.avi
[download]   2.5% of ~1.34GiB at 10.66MiB/s ETA 01:09
ERROR: Interrupted by user

---

Description of your issue, suggested solution and other information

Some videos are stored as collection of small segments at Youtube. When downloading a large video from Youtube that contains many segments, Youtube-dl sends new DNS request for every segment even if the host name is the same. Example video URL: https://www.youtube.com/watch?v=QL0nLqDdKdI . At the moment of writing this video contains 2880 segments, all located on the same host. On fast Internet channel youtube-dl is downloading 4 segments per second, making 4 DNS requests per second with the same host name. I am using DNS server 8.8.8.8 thats stops responding after sending approximately 1000 requests with the same name in short interval of time. After trying several more times, youtube-dl exits with error:

[download]  26.2% of ~1.35GiB at  9.74MiB/s ETA 06:30ERROR: unable to download video data:
 <urlopen error [Errno 8] Name or service not known>

Please note that the error message is confusing, because it doesn't say anything about the real problem.

The DNS response TTL field value for r4---sn-1uxaxjvh-axql.googlevideo.com equals 12 minutes and 3 seconds.

I have searched issues for "dns cache", "long videos", "getaddrinfo failed".

In an issue #13734 @dstftw wrote that this is an issue with a DNS server. I cannot agree with that. Sending thousands of similar queries looks more like flood or DoS attack, no wonder that DNS server stops responding. I think this is an issue with youtube-dl. I looked for an options to make pauses between downloading segments, but couldn't find any. I have also searched for "dns cache" in README but didn't find anything.

Please note that the issue is with sending too much DNS request. Using or not using persistent HTTP connections have no relation to it. So the issue #14523 is irrelevant.

I found a similar issue with downloading long video #15641 . DNS fllod might the the problem of errors there. Also, an issue #15746 - looks like the same problem.

Using --hls-prefer-native or --hls-prefer-ffmpeg doesn't make any difference. ffmpeg does the same thing.

My suggestion is to implement a simple DNS cache that caches the IP address of a host or DNS response for the lifetime of a process. It is highly unlikely that IP address would change during download so there is no need to handle expire time. It is much easier than make user configure local DNS cache and somehow point youtube-dl to it.

To repoduce:

• make sure you have fast enough Internet connection
• set your DNS server as 8.8.8.8 and disable local DNS cache
• run command ./youtube-dl -v --hls-prefer-native --format 300 --ffmpeg-location <location> -o v1.avi --no-mtime 'https://www.youtube.com/watch?v=QL0nLqDdKdI'
• after several minutes server 8.8.8.8 will stop responding

[codedokode]

Also I have found an issue #4966 where there might be the same problem, with ffmpeg sending too many requests.

[codedokode]

I have tried to use --sleep-interval options but it doesn't work for fragments. The option --limit-rate set to some small value like 100K works, but the download becomes very slow.

[Zenexer]

Encountering a similar issue with TBS/Adult Swim, even with a cache on my router. Seems my router dislikes getting flooded.

[Zenexer]

Workaround: add temporary hosts file entries for queried domains.