Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YT lying about 22(best) 720 avc+mp4a file availability, cuts connection, dupes youtube-dl into broken download #16140

Closed
raszpl opened this issue Apr 10, 2018 · 2 comments

Comments

@raszpl
Copy link

@raszpl raszpl commented Apr 10, 2018

  • I've verified and I assure that I'm running youtube-dl 2018.04.09

  • At least skimmed through the README, most notably the FAQ and BUGS sections

  • Searched the bugtracker for similar issues including closed ones

  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

  • Bug report (encountered problems with youtube-dl)

Description of your issue, suggested solution and other information

"-f best" defaults to 22 when available in manifest. Here is what I got:

youtube-dl.exe -F https://www.youtube.com/watch?v=XuoZqb6BDc4
[youtube] XuoZqb6BDc4: Downloading webpage
[youtube] XuoZqb6BDc4: Downloading video info webpage
[youtube] XuoZqb6BDc4: Extracting video information
[info] Available formats for XuoZqb6BDc4:
format code  extension  resolution note
249          webm       audio only DASH audio   60k , opus @ 50k, 6.01MiB
250          webm       audio only DASH audio   79k , opus @ 70k, 8.00MiB
171          webm       audio only DASH audio  124k , vorbis@128k, 13.07MiB
140          m4a        audio only DASH audio  131k , m4a_dash container, mp4a.40.2@128k, 14.83MiB
251          webm       audio only DASH audio  155k , opus @160k, 15.61MiB
278          webm       256x144    144p  123k , webm container, vp9, 30fps, video only, 10.76MiB
160          mp4        256x144    144p  127k , avc1.4d400c, 30fps, video only, 11.00MiB
330          webm       256x144    144p60 HDR  248k , vp9.2, 60fps, video only, 26.74MiB
133          mp4        426x240    240p  270k , avc1.4d4015, 30fps, video only, 23.76MiB
242          webm       426x240    240p  295k , vp9, 30fps, video only, 24.13MiB
331          webm       426x240    240p60 HDR  507k , vp9.2, 60fps, video only, 54.86MiB
243          webm       640x360    360p  710k , vp9, 30fps, video only, 45.19MiB
134          mp4        640x360    360p  737k , avc1.4d401e, 30fps, video only, 63.23MiB
332          webm       640x360    360p60 HDR 1084k , vp9.2, 60fps, video only, 116.87MiB
244          webm       854x480    480p 1135k , vp9, 30fps, video only, 82.50MiB
135          mp4        854x480    480p 1360k , avc1.4d401f, 30fps, video only, 120.53MiB
333          webm       854x480    480p60 HDR 2027k , vp9.2, 60fps, video only, 222.19MiB
247          webm       1280x720   720p 2412k , vp9, 30fps, video only, 166.87MiB
136          mp4        1280x720   720p 2709k , avc1.4d401f, 30fps, video only, 242.97MiB
302          webm       1280x720   720p60 3877k , vp9, 60fps, video only, 286.24MiB
298          mp4        1280x720   720p60 4076k , avc1.4d4020, 60fps, video only, 354.96MiB
248          webm       1920x1080  1080p 4380k , vp9, 30fps, video only, 295.98MiB
334          webm       1280x720   720p60 HDR 4590k , vp9.2, 60fps, video only, 500.71MiB
137          mp4        1920x1080  1080p 4862k , avc1.640028, 30fps, video only, 439.09MiB
299          mp4        1920x1080  1080p60 6855k , avc1.64002a, 60fps, video only, 627.10MiB
303          webm       1920x1080  1080p60 6916k , vp9, 60fps, video only, 482.83MiB
335          webm       1920x1080  1080p60 HDR 7113k , vp9.2, 60fps, video only, 765.54MiB
271          webm       2560x1440  1440p 9783k , vp9, 30fps, video only, 927.91MiB
308          webm       2560x1440  1440p60 14714k , vp9, 60fps, video only, 1.37GiB
336          webm       2560x1440  1440p60 HDR 16574k , vp9.2, 60fps, video only, 1.77GiB
313          webm       3840x2160  2160p 18023k , vp9, 30fps, video only, 1.87GiB
315          webm       3840x2160  2160p60 27267k , vp9, 60fps, video only, 2.82GiB
337          webm       3840x2160  2160p60 HDR 29302k , vp9.2, 60fps, video only, 3.14GiB
17           3gp        176x144    small , mp4v.20.3, mp4a.40.2@ 24k
36           3gp        320x180    small , mp4v.20.3, mp4a.40.2
43           webm       640x360    medium , vp8.0, vorbis@128k
18           mp4        640x360    medium , avc1.42001E, mp4a.40.2@ 96k
22           mp4        1280x720   hd720 , avc1.64001F, mp4a.40.2@192k (best)

D:\_learning>youtube-dl.exe -f 22 https://www.youtube.com/watch?v=XuoZqb6BDc4
[youtube] XuoZqb6BDc4: Downloading webpage
[youtube] XuoZqb6BDc4: Downloading video info webpage
[youtube] XuoZqb6BDc4: Extracting video information
[download] Resuming download at byte 487131


ERROR: Did not get any data blocks

totally something shady server side, wget gets same connection dropped


youtube-dl.exe -f 22 -g https://www.youtube.com/watch?v=XuoZqb6BDc4  -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-f', '22', '-g', 'https://www.youtube.com/watch?v=XuoZqb6BDc4', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2018.04.09
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.15063
[debug] exe versions: ffmpeg N-87327-g18821e3, ffprobe N-87327-g18821e3
[debug] Proxy map: {}
https://r4---sn-f5f7lnee.googlevideo.com/videoplayback?key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27sn7k&id=o-ACNwbGa1yF9EFR2WeT1uUlP9gBiIeFj63YI2aAx839KD&lmt=1523310968601281&nh=%2CIgpwcjAzLndhdzAyKgkxMjcuMC4wLjE&ei=OO3LWqzuAcig7ATUvKvICA&fvip=14&requiressl=yes&dur=960.911&pl=13&itag=22&source=youtube&c=WEB&expire=1523335576&mime=video%2Fmp4&mm=31%2C26&ratebypass=yes&ipbits=0&initcwndbps=1162500&signature=CBC8910E27853E7A90BD16E19AF665D63929F2FB.30BE7CA0FD17FD9D6F066F3D50E1F789B820D91E

D:\_learning>wget.exe "https://r4---sn-f5f7lnee.googlevideo.com/videoplayback?key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27sn7k&id=o-ACNwbGa1yF9EFR2WeT1uUlP9gBiIeFj63YI2aAx839KD&lmt=1523310968601281&nh=%2CIgpwcjAzLndhdzAyKgkxMjcuMC4wLjE&ei=OO3LWqzuAcig7ATUvKvICA&fvip=14&requiressl=yes&dur=960.911&pl=13&itag=22&source=youtube&c=WEB&expire=1523335576&mime=video%2Fmp4&mm=31%2C26&ratebypass=yes&ipbits=0&initcwndbps=1162500&signature=CBC8910E27853E7A90BD16E19AF665D63929F2FB.30BE7CA0FD17FD9D6F066F3D50E1F789B820D91E"
--2018-04-10 01:46:54--  https://r4---sn-f5f7lnee.googlevideo.com/videoplayback?key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27sn7k&id=o-ACNwbGa1yF9EFR2WeT1uUlP9gBiIeFj63YI2aAx839KD&lmt=1523310968601281&nh=%2CIgpwcjAzLndhdzAyKgkxMjcuMC4wLjE&ei=OO3LWqzuAcig7ATUvKvICA&fvip=14&requiressl=yes&dur=960.911&pl=13&itag=22&source=youtube&c=WEB&expire=1523335576&mime=video%2Fmp4&mm=31%2C26&ratebypass=yes&ipbits=0&initcwndbps=1162500&signature=CBC8910E27853E7A90BD16E19AF665D63929F2FB.30BE7CA0FD17FD9D6F066F3D50E1F789B820D91E
Resolving r4---sn-f5f7lnee.googlevideo.com (r4---sn-f5f7lnee.googlevideo.com)... 74.125.111.186
Connecting to r4---sn-f5f7lnee.googlevideo.com (r4---sn-f5f7lnee.googlevideo.com)|74.125.111.186|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 270258567 (258M) [video/mp4]
Saving to: 'videoplayback@key=yt6&ip=89.66.160.149&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27s'

videoplayback@key=yt6&ip=89.6   0%[                                                  ] 475.71K  --.-KB/s    in 0.05s

2018-04-10 01:46:55 (10.1 MB/s) - Connection closed at byte 487131. Retrying.

Meanwhile after searching for this bug on the net for ~30 minutes:

D:\_learning>wget.exe "https://r4---sn-f5f7lnee.googlevideo.com/videoplayback?key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27sn7k&id=o-ACNwbGa1yF9EFR2WeT1uUlP9gBiIeFj63YI2aAx839KD&lmt=1523310968601281&nh=%2CIgpwcjAzLndhdzAyKgkxMjcuMC4wLjE&ei=OO3LWqzuAcig7ATUvKvICA&fvip=14&requiressl=yes&dur=960.911&pl=13&itag=22&source=youtube&c=WEB&expire=1523335576&mime=video%2Fmp4&mm=31%2C26&ratebypass=yes&ipbits=0&initcwndbps=1162500&signature=CBC8910E27853E7A90BD16E19AF665D63929F2FB.30BE7CA0FD17FD9D6F066F3D50E1F789B820D91E"
--2018-04-10 02:14:48--  https://r4---sn-f5f7lnee.googlevideo.com/videoplayback?key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27sn7k&id=o-ACNwbGa1yF9EFR2WeT1uUlP9gBiIeFj63YI2aAx839KD&lmt=1523310968601281&nh=%2CIgpwcjAzLndhdzAyKgkxMjcuMC4wLjE&ei=OO3LWqzuAcig7ATUvKvICA&fvip=14&requiressl=yes&dur=960.911&pl=13&itag=22&source=youtube&c=WEB&expire=1523335576&mime=video%2Fmp4&mm=31%2C26&ratebypass=yes&ipbits=0&initcwndbps=1162500&signature=CBC8910E27853E7A90BD16E19AF665D63929F2FB.30BE7CA0FD17FD9D6F066F3D50E1F789B820D91E
Resolving r4---sn-f5f7lnee.googlevideo.com (r4---sn-f5f7lnee.googlevideo.com)... 74.125.111.186
Connecting to r4---sn-f5f7lnee.googlevideo.com (r4---sn-f5f7lnee.googlevideo.com)|74.125.111.186|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 270258567 (258M) [video/mp4]
Saving to: 'videoplayback@key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27s'

videoplayback@key=yt6&ip= 100%[=================================================>] 257.74M  12.5MB/s    in 21s

2018-04-10 02:22:48 (12.4 MB/s) - 'videoplayback@key=yt6&ip=&mt=1523313907&mv=m&ms=au%2Conr&sparams=dur%2Cei%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cnh%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&mn=sn-f5f7lnee%2Csn-3c27s.1' saved [270258567/270258567]

Magically works 100% ok.

In #14866 user @dstftw said "Format 22 is not available at Youtube side.". This is not entirely true, YT lists this format, and partially downloaded files look ok, but server arbitrarily blocks downloading. It cant be a case of partially processed file (encoding error/crash) because file size is valid (270258567), so why close connection? File size is not even from clen tag, its returned by the server in content-length header, same server that closes connection on us! :0

I even tried MOSrkm0r-Co clip from #14866 and it still drops connection after ~700KB serving start of a valid 720 avc+mp4a file. I got curious and forced YT player to serve h264 files, but that clever beast just used separate 136 + 140 streams bypassing 22 altogether. #14733 is same problem.

Is YT suppressing this format? is 22 deprecated on YT backend and only for backward compatibility with some fringe old devices? Why would google mess with serving 22 while its clearly present server side (correct content-length header = file was there from the start).

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 10, 2018

Most likely processing was not finished at the time of downloading. Anyway address this question to Youtube not here,

@raszpl
Copy link
Author

@raszpl raszpl commented Apr 10, 2018

Like I already said in the report : It cant be a case of partially processed file (encoding error/crash) because file size was valid (270258567) WHILE it was dropping connections. You cant already know the resulting file size before finishing muxing. Size is from content-length header = file is sitting on the server visible to www deamon. Not to mention files from other reports are still inaccessible.

Are you saying youtube-dl.exe -f 22 https://www.youtube.com/watch?v=T6f1-nEjDdM works for you right now?

Its not a YT problem per se, videos work fine on their website, they seem to stopped using itag 22 internally. Its more of a YT induced problem touching youtube-dl users. youtube-dl -f best DEFAULTS to itag22. There are currently >10 bug reports from people with same problem, all running default youtube-dl -f best

Its almost as if Google started booby trapping itag 22, or one of the services responsible for it is failing invisibly to monitoring because nobody uses it anymore on the backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.