Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NBC #18202

Closed
lincalinca opened this issue Nov 16, 2018 · 3 comments
Closed

NBC #18202

lincalinca opened this issue Nov 16, 2018 · 3 comments

Comments

@lincalinca
Copy link

@lincalinca lincalinca commented Nov 16, 2018

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.11.07. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2018.11.07
    (I wasn't, then I upgraded from 2018-10-05, and the issue persisted; the logs below are from after upgrading)

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser (I couldn't do this, because I was using a proxy, which I haven't activated - I have had people geolocated in the US check, and the video works

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

E:\OURS\The Good Place\nbcdl>youtube-dl --proxy "23.101.121.11:3128" -u ###.###@###.com -p ############ https://www.nbc.com/the-good-place/video/michaels-gambit-extended-cut/3454089 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--proxy', u'23.101.121.11:3128', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'https://www.nbc.com/the-good-place/video/michaels-gambit-extended-cut/3454089', u'--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2018.11.07
[debug] Python version 2.7.12 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg N-91646-g78d4b6bd43, ffprobe N-91646-g78d4b6bd43
[debug] Proxy map: {u'http': u'23.101.121.11:3128', u'https': u'23.101.121.11:3128'}
[NBC] 3454089: Downloading JSON metadata
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\Python27\Scripts\youtube-dl.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\youtube_dl\__init__.py", line 472, in main
    _real_main(argv)
  File "c:\python27\lib\site-packages\youtube_dl\__init__.py", line 462, in _real_main
    retcode = ydl.download(all_urls)
  File "c:\python27\lib\site-packages\youtube_dl\YoutubeDL.py", line 2001, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "c:\python27\lib\site-packages\youtube_dl\YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "c:\python27\lib\site-packages\youtube_dl\extractor\common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "c:\python27\lib\site-packages\youtube_dl\extractor\nbc.py", line 96, in _real_extract
    video_data = response['data'][0]['attributes']
IndexError: list index out of range

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):


Description of your issue, suggested solution and other information

Tried to download video from nbc.com. It worked for one video in front of the paywall, but not for any behind the paywall. I signed up so I could get 3x free episodes, and used the following command:
youtube-dl --proxy "23.101.121.11:3128" -u ###.###@###.com -p ############ https://www.nbc.com/the-good-place/video/michaels-gambit-extended-cut/3454089 --verbose
(my credentials redacted, of course). As I'm not in the US, I needed to run this through the proxy. This proxy worked for extracting the non-paywalled episodes.

@josejuan05
Copy link

@josejuan05 josejuan05 commented Nov 18, 2018

Edit: See below; this is a separate bug

I get the same error for http://www.nbc.com/saturday-night-live/video/november-17-steve-carell/3828729, which does not appear to be behind the paywall.

Looking at the _real_extract function that is failing:

    def _real_extract(self, url):
        permalink, video_id = re.match(self._VALID_URL, url).groups()
        permalink = 'http' + compat_urllib_parse_unquote(permalink)
        response = self._download_json(
            'https://api.nbc.com/v3/videos', video_id, query={
                'filter[permalink]': permalink,
                'fields[videos]': 'description,entitlement,episodeNumber,guid,keywords,seasonNumber,title,vChipRating',
                'fields[shows]': 'shortTitle',
                'include': 'show.shortTitle',
            })
        video_data = response['data'][0]['attributes']

The function is failing because the data list is empty.

If I run youtube-dl http://www.nbc.com/saturday-night-live/video/november-17-steve-carell/3828729 --dump-pages it dumps out the following for the response object:

{"data":[],"meta":{"count":0,"version":"v3.0.0"},"links":{"self":"https://api.nbc.com/v3/videos?filter%5Bpermalink%5D=http%3A//www.nbc.com/saturday-night-live/video/november-17-steve-carell/3828729&include=show%2Cshow.shortTitle&page%5Bnumber%5D=1"}}

If I curl the URL in response['links']['self'] it appears to return the object that is actually required by the video_data declaration.

There are other NBC videos that still work (e.g. https://www.nbc.com/the-good-place/video/the-ballad-of-donkey-doug/3814933). The fix shouldn't break those.

I added a line to the above function as shown in the following code, and it appears to fix my problem.

    def _real_extract(self, url):
        permalink, video_id = re.match(self._VALID_URL, url).groups()
        permalink = 'http' + compat_urllib_parse_unquote(permalink)
        response = self._download_json(
            'https://api.nbc.com/v3/videos', video_id, query={
                'filter[permalink]': permalink,
                'fields[videos]': 'description,entitlement,episodeNumber,guid,keywords,seasonNumber,title,vChipRating',
                'fields[shows]': 'shortTitle',
                'include': 'show.shortTitle',
            })
        if len(response['data']) is 0:
          response = self._download_json(response['links']['self'],video_id)

If I can get my git login fixed I'll make a pull request.

@josejuan05 josejuan05 mentioned this issue Nov 18, 2018
5 of 9 tasks complete
@josejuan05
Copy link

@josejuan05 josejuan05 commented Nov 18, 2018

The original post links to a video which is not actually available on the NBC site, even for logged-in users. It's not clear to me whether the invocation of credentials used in OP's bug report would even work for NBC. I tried -u,-p with a test set of NBC.com credentials for a different locked video and received notice that the video required Adobe Pass Multi-System Operator credentials, even though I can watch the video online with the NBC.com credentials. So my patch seems to be for an unrelated bug. I created a separate issue.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Nov 19, 2018

https://www.nbc.com/the-good-place/video/michaels-gambit-extended-cut/3454089

Sorry, video is no longer available.

http://www.nbc.com/saturday-night-live/video/november-17-steve-carell/3828729

youtube-dl http://www.nbc.com/saturday-night-live/video/november-17-steve-carell/3828729
[NBC] 3828729: Downloading JSON metadata
[ThePlatform] 3828729: Downloading SMIL data
[ThePlatform] 3828729: Downloading m3u8 information
[ThePlatform] 3828729: Downloading JSON metadata
[download] Destination: November 17 - Steve Carell-3828729.mp4
ffmpeg version n4.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20180831
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[hls,applehttp @ 0x56464204e4c0] Opening 'https://nbcmpx-vh.akamaihd.net/i/video/629/291/181112_3828729_November_17___Steve_Carell_,25,40,18,12,7,4,2,00.mp4.csmil/crypt.key?null=0&id=AgBFocdGpf4YioNf8lvZ130OQr9v8kx0ez9JzD0iiJlxhXlqYRANyxrLTAUUOREtzxsX4tEGFGCGOg%3d%3d&hdntl=exp=1542697219~acl=/i/video/629/291/181112_3828729_November_17___Steve_Carell_*~data=hdntl~hmac=05d463f6200a665a4cad1771e8bd1012c88a22679df08dda37980b1ffcd4ebcd' for reading
[hls,applehttp @ 0x56464204e4c0] Opening 'crypto+https://nbcmpx-vh.akamaihd.net/i/video/629/291/181112_3828729_November_17___Steve_Carell_,25,40,18,12,7,4,2,00.mp4.csmil/segment1_0_av.ts?null=0&id=AgBFocdGpf4YioNf8lvZ130OQr9v8kx0ez9JzD0iiJlxhXlqYRANyxrLTAUUOREtzxsX4tEGFGCGOg%3d%3d&hdntl=exp=1542697219~acl=/i/video/629/291/181112_3828729_November_17___Steve_Carell_*~data=hdntl~hmac=05d463f6200a665a4cad1771e8bd1012c88a22679df08dda37980b1ffcd4ebcd' for reading
Input #0, hls,applehttp, from 'https://nbcmpx-vh.akamaihd.net/i/video/629/291/181112_3828729_November_17___Steve_Carell_,25,40,18,12,7,4,2,00.mp4.csmil/index_0_av.m3u8?null=0&id=AgBFocdGpf4YioNf8lvZ130OQr9v8kx0ez9JzD0iiJlxhXlqYRANyxrLTAUUOREtzxsX4tEGFGCGOg%3d%3d&hdntl=exp=1542697219~acl=%2fi%2fvideo%2f629%2f291%2f181112_3828729_November_17___Steve_Carell_*~data=hdntl~hmac=05d463f6200a665a4cad1771e8bd1012c88a22679df08dda37980b1ffcd4ebcd':
  Duration: 01:08:37.76, start: 0.100511, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720, Closed Captions, 29.97 tbr, 90k tbn, 59.94 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:November 17 - Steve Carell-3828729.mp4.part':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720, q=2-31, 29.97 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
^Cframe=   60 fps=0.0 q=-1.0 Lsize=     457kB time=00:00:01.99 bitrate=1875.8kbits/s speed=3.72x    
video:440kB audio:14kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.783505%
@remitamine remitamine closed this Nov 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.