Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pbs] Failed to download Nova video #14305

Closed
pepoluan opened this issue Sep 23, 2017 · 12 comments
Closed

[pbs] Failed to download Nova video #14305

pepoluan opened this issue Sep 23, 2017 · 12 comments

Comments

@pepoluan
Copy link

@pepoluan pepoluan commented Sep 23, 2017

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.09.15. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2017.09.15
% youtube-dl --version
2017.09.15

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: ['--external-downloader', 'aria2c', '--prefer-ffmpeg', '--ffmpeg-location', '/usr/local/bin/ffmpeg']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'http://www.pbs.org/wgbh/nova/space/death-dive-to-Saturn.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.15
[debug] Python version 3.6.1 - CYGWIN_NT-10.0-2.9.0-0.318-5-3-x86_64-64bit-WindowsPE
[debug] exe versions: ffmpeg N-85750-ga75ef15, ffprobe N-85750-ga75ef15
[debug] Proxy map: {}
[debug] Using fake IP 3.84.59.216 (US) as X-Forwarded-For.
[pbs] Downloading JSON metadata
[pbs] death-dive-to-Saturn: Downloading webpage
[pbs] Downloading widget/partnerplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
[pbs] Downloading portalplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/pbs.py", line 585, in _real_extract
    self._sort_formats(formats)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 1072, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description of your issue, suggested solution and other information

Not sure if this is related to #7095 or not, but since #7095 is an "old" issue (opened 2015, last update Jan 2017) with a possible solution #b0db58b, I think a new issue is in order.

I also don't think this is related to #8538, but since youtube-dl totally failed finding any video, probably not.

If you need more info or things to try out, feel free to ask.

@pepoluan
Copy link
Author

@pepoluan pepoluan commented Sep 23, 2017

Okay, out of curiosity, I grabbed the embed code for the video:

<iframe width="512" height="376" src="http://player.pbs.org/viralplayer/3004606354/" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" seamless allowfullscreen></iframe>

Then I feed the URL directly to youtube-dl. And it seems to work! Verbose partial output (still in download at the moment) follows:

% youdl.sh --verbose http://player.pbs.org/viralplayer/3004606354/
youtube-dl less than 2 days old. Skipping update
[debug] System config: []
[debug] User config: ['--external-downloader', 'aria2c', '--prefer-ffmpeg', '--ffmpeg-location', '/usr/local/bin/ffmpeg']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'http://player.pbs.org/viralplayer/3004606354/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.15
[debug] Python version 3.6.1 - CYGWIN_NT-10.0-2.9.0-0.318-5-3-x86_64-64bit-WindowsPE
[debug] exe versions: ffmpeg N-85750-ga75ef15, ffprobe N-85750-ga75ef15
[debug] Proxy map: {}
[debug] Using fake IP 3.210.188.40 (US) as X-Forwarded-For.
[pbs] Downloading JSON metadata
[pbs] 3004606354: Downloading widget/partnerplayer page
[pbs] 3004606354: Downloading portalplayer page
[pbs] 3004606354: Downloading hls-1080p-16x9 video url info
[pbs] 3004606354: Downloading m3u8 information
[pbs] 3004606354: Downloading mp4-2500k-16x9 video url info
[pbs] 3004606354: Downloading hls-16x9-720p video url info
[pbs] 3004606354: Downloading m3u8 information
[pbs] 3004606354: Checking http-2000k video URL
[pbs] 3004606354: http-2000k video URL is invalid, skipping
[pbs] 3004606354: Checking http-6500k video URL
[pbs] 3004606354: http-6500k video URL is invalid, skipping
[pbs] 3004606354: Checking http-4500k video URL
[pbs] 3004606354: http-4500k video URL is invalid, skipping
[pbs] 3004606354: Checking http-3000k video URL
[pbs] 3004606354: Checking http-1100k video URL
[pbs] 3004606354: http-1100k video URL is invalid, skipping
[pbs] 3004606354: Checking http-730k video URL
[pbs] 3004606354: http-730k video URL is invalid, skipping
[pbs] 3004606354: Checking http-730k video URL
[pbs] 3004606354: http-730k video URL is invalid, skipping
[pbs] 3004606354: Checking http-3000k video URL
[pbs] 3004606354: Checking http-2000k video URL
[pbs] 3004606354: http-2000k video URL is invalid, skipping
[pbs] 3004606354: Checking http-1100k video URL
[pbs] 3004606354: http-1100k video URL is invalid, skipping
[pbs] 3004606354: http-1100k video URL is invalid, skipping
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8'
[download] Destination: NOVA - S44 Ep12 - Death Dive to Saturn-3004606354.mp4
[debug] ffmpeg command line: /usr/local/bin/ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Cookie: pbsol.station=WNET
X-Forwarded-For: 3.210.188.40
' -i https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc 'file:NOVA - S44 Ep12 - Death Dive to Saturn-3004606354.mp4.part'
ffmpeg version N-85750-ga75ef15 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 61.100 / 55. 61.100
  libavcodec     57. 93.100 / 57. 93.100
  libavformat    57. 72.101 / 57. 72.101
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 88.100 /  6. 88.100
  libswscale      4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
[tls @ 00000000006d2f20] Server closed the connection
[hls,applehttp @ 00000000006f24a0] HLS request for url 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00001.ts', offset 0, playlist 0
[h264 @ 0000000000791e60] Reinit context to 1920x1088, pix_fmt: yuv420p
Input #0, hls,applehttp, from 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8':

... a bunch of ffmpeg output follows ...

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[tls @ 00000000006fdf20] Server closed the connection:05.60 bitrate=6568.3kbits/s speed=2.72x
[hls,applehttp @ 00000000006f24a0] HLS request for url 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00002.ts', offset 0, playlist 0
[tls @ 000000000071c9c0] Server closed the connection:11.24 bitrate=6599.2kbits/s speed=1.09x
[hls,applehttp @ 00000000006f24a0] HLS request for url 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00003.ts', offset 0, playlist 0

... an bunch of incremental repetition (increasing .ts number) follows ...

<download still going on>
@disposableid
Copy link

@disposableid disposableid commented Sep 27, 2017

PBS/Nova has been broken for awhile. The value of "pbs_video_id_*" used to contain only digits and the pbs extractor code matches this. But at some point the value became a string ending with "==" , "IrutAxizvuIvkt__hRsh-A==", for example, for the death-dive-to-Saturn program. Somehow that turns into "3004606354". It looks like base64, but isn't. So I've reached the extent of my expertise. Hopefully someone else will recognize the format.

@gfabiano gfabiano mentioned this issue Sep 27, 2017
4 of 8 tasks complete
@pepoluan
Copy link
Author

@pepoluan pepoluan commented Sep 28, 2017

#14351 does NOT close this issue. Tested with exact same URL, still ended with error:

[debug] System config: []
[debug] User config: ['--external-downloader', 'aria2c', '--prefer-ffmpeg', '--ffmpeg-location', '/usr/local/bin/ffmpeg']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'http://www.pbs.org/wgbh/nova/space/death-dive-to-Saturn.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.24
[debug] Python version 3.6.1 - CYGWIN_NT-10.0-2.9.0-0.318-5-3-x86_64-64bit-WindowsPE
[debug] exe versions: ffmpeg N-87353-g183fd30, ffprobe N-87353-g183fd30
[debug] Proxy map: {}
[debug] Using fake IP 3.48.30.12 (US) as X-Forwarded-For.
[pbs] Downloading JSON metadata
[pbs] death-dive-to-Saturn: Downloading webpage
[pbs] Downloading widget/partnerplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
[pbs] Downloading portalplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 777, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/pbs.py", line 585, in _real_extract
    self._sort_formats(formats)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 1072, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
@pepoluan
Copy link
Author

@pepoluan pepoluan commented Sep 28, 2017

More investigation:

The strangely-encoded value of pbs_video_id_* seems to be fed without modification/decoding into the URL http://player.pbs.org/partnerplayer/${VIDEO_ID}/?w=680&h=430.2397&chapterbar=true&endscreen=false&topbar=true&wmode=transparent

Feeding that URL to youtube-dl, though, properly triggers the download of the video by .ts chunks using ffmpeg. Verbose partial report follows:

[debug] System config: []
[debug] User config: ['--external-downloader', 'aria2c', '--prefer-ffmpeg', '--ffmpeg-location', '/usr/local/bin/ffmpeg']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'http://player.pbs.org/partnerplayer/IrutAxizvuIvkt__hRsh-A==/?w=680&h=430.2397&chapterbar=true&endscreen=false&topbar=true&wmode=transparent']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.24
[debug] Python version 3.6.1 - CYGWIN_NT-10.0-2.9.0-0.318-5-3-x86_64-64bit-WindowsPE
[debug] exe versions: ffmpeg N-87353-g183fd30, ffprobe N-87353-g183fd30
[debug] Proxy map: {}
[debug] Using fake IP 3.125.104.16 (US) as X-Forwarded-For.
[pbs] Downloading JSON metadata
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading player page
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading widget/partnerplayer page
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading portalplayer page
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading hls-1080p-16x9 video url info
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading m3u8 information
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading mp4-2500k-16x9 video url info
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading hls-16x9-720p video url info
[pbs] IrutAxizvuIvkt__hRsh-A==: Downloading m3u8 information
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-2000k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-2000k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-6500k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-6500k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-4500k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-4500k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-3000k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-1100k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-1100k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-730k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-730k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-730k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-730k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-3000k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-2000k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-2000k video URL is invalid, skipping
[pbs] IrutAxizvuIvkt__hRsh-A==: Checking http-1100k video URL
[pbs] IrutAxizvuIvkt__hRsh-A==: http-1100k video URL is invalid, skipping
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8'
[download] Destination: NOVA - S44 Ep12 - Death Dive to Saturn-3004606354.mp4
[debug] ffmpeg command line: /usr/local/bin/ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Cookie: pbsol.station=WNET
X-Forwarded-For: 3.125.104.16
' -i https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc 'file:NOVA - S44 Ep12 - Death Dive to Saturn-3004606354.mp4.part'
ffmpeg version N-87353-g183fd30 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 7.2.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 76.100 / 55. 76.100
  libavcodec     57.106.101 / 57.106.101
  libavformat    57. 82.101 / 57. 82.101
  libavdevice    57.  8.101 / 57.  8.101
  libavfilter     6.105.100 /  6.105.100
  libswscale      4.  7.103 /  4.  7.103
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
[tls @ 000000000089ab40] Server closed the connection
[hls,applehttp @ 0000000001f01ca0] HLS request for url 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00001.ts', offset 0, playlist 0
[hls,applehttp @ 0000000001f01ca0] Opening 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00001.ts' for reading
[h264 @ 0000000001f55780] Reinit context to 1920x1088, pix_fmt: yuv420p
Input #0, hls,applehttp, from 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8':
  Duration: 00:54:31.67, start: 2.050078, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:NOVA - S44 Ep12 - Death Dive to Saturn-3004606354.mp4.part':
  Metadata:
    encoder         : Lavf57.82.101
    Stream #0:0: Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(left), 1920x1080 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[tls @ 0000000001f12fc0] Server closed the connection:04.90 bitrate=6413.5kbits/s speed=2.25x
[hls,applehttp @ 0000000001f01ca0] HLS request for url 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00002.ts', offset 0, playlist 0
[hls,applehttp @ 0000000001f01ca0] Opening 'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k_00002.ts' for reading
frame=  228 fps= 46 q=-1.0 size=    5888kB time=00:00:07.57 bitrate=6368.3kbits/s speed=1.54x
@gfabiano
Copy link
Contributor

@gfabiano gfabiano commented Sep 28, 2017

Are you sure my fix does not work?

python -m youtube_dl -v http://www.pbs.org/wgbh/nova/space/death-dive-to-Saturn.html
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://www.pbs.org/wgbh/nova/space/death-dive-to-Saturn.html']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2017.09.15
[debug] Git HEAD: cc80dbec9
[debug] Python version 2.7.12 - Windows-10-10.0.15063
[debug] exe versions: ffmpeg 3.3.3, ffprobe 3.3.3
[debug] Proxy map: {}
[debug] Using fake IP 3.192.130.229 (US) as X-Forwarded-For.
[pbs] Downloading JSON metadata
[pbs] death-dive-to-Saturn: Downloading webpage
[pbs] death-dive-to-Saturn: Downloading widget/partnerplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
[pbs] death-dive-to-Saturn: Downloading portalplayer page
WARNING: Unable to download webpage: HTTP Error 404: Not Found
[pbs] death-dive-to-Saturn: Downloading partnerplayer page
[pbs] death-dive-to-Saturn: Downloading hls-1080p-16x9 video url info
[pbs] death-dive-to-Saturn: Downloading m3u8 information
[pbs] death-dive-to-Saturn: Downloading mp4-2500k-16x9 video url info
[pbs] death-dive-to-Saturn: Checking http-2000k video URL
[pbs] death-dive-to-Saturn: http-2000k video URL is invalid, skipping
[pbs] death-dive-to-Saturn: Checking http-6500k video URL
[pbs] death-dive-to-Saturn: http-6500k video URL is invalid, skipping
[pbs] death-dive-to-Saturn: Checking http-4500k video URL
[pbs] death-dive-to-Saturn: http-4500k video URL is invalid, skipping
[pbs] death-dive-to-Saturn: Checking http-3000k video URL
[pbs] death-dive-to-Saturn: Checking http-1100k video URL
[pbs] death-dive-to-Saturn: http-1100k video URL is invalid, skipping
[pbs] death-dive-to-Saturn: Checking http-730k video URL
[pbs] death-dive-to-Saturn: http-730k video URL is invalid, skipping
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://ga.video.cdn.pbs.org/videos/nova/3968ea7e-9177-4ba9-aa82-b73c9ef90ee6/2000008498/hd-16x9-mezzanine-1080p/2xpja9ui_nova4412-16x9-1080p-1080p-6500k.m3u8'

I download without problems

--edit
In your log there isn't this line:
[pbs] death-dive-to-Saturn: Downloading partnerplayer page
you probably are not using my fix

@pepoluan
Copy link
Author

@pepoluan pepoluan commented Sep 29, 2017

Hmmm... I think I made a mistake when pulling. I'll try again.

@stinkteeth
Copy link

@stinkteeth stinkteeth commented Sep 30, 2017

The page source of:
http://www.pbs.org/video/death-dive-to-saturn-oiqvx2/
contains:

`window.PBS.playerConfig

= {

id: '3004606354',

slug: 'death\u002Ddive\u002Dto\u002Dsaturn\u002Doiqvx2',

embedURL: 'https://player.pbs.org/',

embedType: 'portalplayer/'`

By assembling the embedURL, embedType, and id values in sequence, this url is contructed:
https://player.pbs.org/portalplayer/3004606354
and within its souce there is this:

`PBS.videoData

= {

'alternate_encoding':

{

'url': 'https://urs.pbs.org/redirect/67b3794774b840a5ab73fd0bb52e390c/',

'eeid': 'mp4-16x9-720p'

},

recommended_encoding': {'url': 'https://urs.pbs.org/redirect/246e95287fe9452caf079b7b8c2dc053/',

eeid': 'hls-16x9-720p'`

which are redirects to http mp4 and hls, respectively.

However, if for the embedURL value, "viralplayer" is used instead of "portalplayer", like this:
http://player.pbs.org/viralplayer/3004606354/
then its source contains a different variant for the hls redirect.

Yet still, if the encoded value is used with an embedURL value of "partnerplayer, like this:
http://player.pbs.org/partnerplayer/IrutAxizvuIvkt__hRsh-A==/

then yet another hls redirect variant is revealed.

This last variant seems to reveal the higher hls bitrates directly, and it probably caused by being logged in somehow.

Also, the http mp4 naming convention has changed. http mp4 doesn't seem to go higher than the 3000k name for newer videos. Previously, http mp4 url names ranged up to 6500k. Those qualities now seem to be exclusively hls.

@gfabiano
Copy link
Contributor

@gfabiano gfabiano commented Sep 30, 2017

So the real id is 3004606354 and not IrutAxizvuIvkt__hRsh-A== , right? I will try to handle all formats and adjust scraping of video_id

@stinkteeth
Copy link

@stinkteeth stinkteeth commented Sep 30, 2017

3004606354 is the "real" public facing id, IrutAxizvuIvkt__hRsh-A== is an alternate id for the same video that provides the higher bitrate hls redirect, I suspect to "passport" subscribers while they are logged in.
I am not a passport subscriber, so I don't know if the alternate id is accessible in the public facing source or via a publicly accessible api or other method.

@gfabiano
Copy link
Contributor

@gfabiano gfabiano commented Oct 1, 2017

Now my pull request ( #14351 ) handle all of this, check it.

@pepoluan
Copy link
Author

@pepoluan pepoluan commented Oct 1, 2017

@stinkteeth one point to note is that I am not a "passport" subscriber, yet if I pull http://player.pbs.org/partnerplayer/IrutAxizvuIvkt__hRsh-A==/ using wget, I can see the URLs for the higher bitrate video.

@pepoluan
Copy link
Author

@pepoluan pepoluan commented Oct 1, 2017

@gfabiano I concur: #14351 indeed fixes this. Closing.

@pepoluan pepoluan closed this Oct 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.