fox.com newest episodes won't download

[tony-izzo]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.10.29
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

PASTE VERBOSE LOG HERE
python3 /usr/local/bin/youtube-dl --verbose https://www.fox.com/watch/90ad35a8b99adf1b580082da447ad039/
[debug] System config: []
[debug] User config: ['--ap-mso', 'Charter_Direct', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.fox.com/watch/90ad35a8b99adf1b580082da447ad039/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.10.29
[debug] Python version 3.5.2 (CPython) - Linux-4.10.0-38-generic-x86_64-with-LinuxMint-18.3-sylvia
[debug] exe versions: avconv 2.8.15, avprobe 2.8.15, ffmpeg 2.8.15, ffprobe 2.8.15
[debug] Proxy map: {}
[FOX] Downloading JSON metadata
[FOX] 90ad35a8b99adf1b580082da447ad039: Downloading JSON metadata
ERROR: This video is only available via cable service provider subscription. You may want to use --cookies.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2237, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 67, in _call_api
    video_id, data=data, headers=headers)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/adobepass.py", line 1337, in _download_webpage_handle
    *args, **compat_kwargs(kwargs))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2237, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 67, in _call_api
    video_id, data=data, headers=headers)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/adobepass.py", line 1337, in _download_webpage_handle
    *args, **compat_kwargs(kwargs))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 96, in _real_extract
    video = self._call_api('vodplayer/' + video_id, video_id)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 76, in _call_api
    'subscription. You may want to use --cookies.', expected=True)
youtube_dl.utils.ExtractorError: This video is only available via cable service provider subscription. You may want to use --cookies.

Description

Newer TV episodes on fox.com, those for which cable provider credentials are required, do not download. The output suggests using --cookies, but that doesn't work either.

The URL I provided to reproduce the problem (https://www.fox.com/watch/90ad35a8b99adf1b580082da447ad039/) is for the episode airing on 2019-10-28. This episode will become available to download when it is old enough (seems to be between a week and 2 weeks). If you are unable to reproduce using this URL, please navigate to the show home page (https://www.fox.com/9-1-1/) and use the URL of the most recent episode.

[PlaceboPRS]

I used S03e05 as I haven't seen 06 and didn't want spoilers! But it works fine, use this:

youtube-dl.exe --no-check-certificate https://content-aeuf1.uplynk.com/2b0a42422b0c458ab6f5e14e1b50b8e4/k.m3u8?pbs=14c5d1fbc5b240488f7d1c4f9c1fb4f5

[PlaceboPRS]

Without getting the m3u8 this worked:

youtube-dl.exe --cookies D:\Videos\cookies.txt --no-check-certificate https://www.fox.com/watch/90ad35a8b99adf1b580082da447ad039/

[tony-izzo]

Great, thanks I'll try that out tonight!

Out of curiosity would you characterize this as a workaround for a bug that should still be fixed, in which case I'll leave the issue open? Or is this a non-bug in which case I'll close it?

I don't know what m3u8 is, but I did notice the failure on new episodes occurred when retrieving it. Seemed to me that we were making an invalid (or no longer valid) assumption that m3u8 was outside, or otherwise not protected by, the paywall.

[PlaceboPRS]

Out of curiosity would you characterize this as a workaround for a bug that should still be fixed, in which case I'll leave the issue open? Or is this a non-bug in which case I'll close it?

That would definitely be for someone else to answer I'm just a curious user of this wonderful little utility :)

[remitamine]

@tony-izzo most likely you're using an expired or invalid cookies.

[tony-izzo]

The workaround is not working for me. Looks like you're using Windows. I'm on Linux. Makes it seem like a third-party library problem. I'll dig in when I have time. Thanks for taking a look.