Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fox.com HTTP Error 403: Forbidden #20208

Closed
roadzy opened this issue Mar 1, 2019 · 14 comments
Closed

Fox.com HTTP Error 403: Forbidden #20208

roadzy opened this issue Mar 1, 2019 · 14 comments

Comments

@roadzy
Copy link

@roadzy roadzy commented Mar 1, 2019

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like this: [x])
  • Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.03.01. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2019.03.01

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2019.03.01
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.

Trying to download The Masked Singer. I tried with cookies and it still fails.

youtube-dl --cookies=ms.txt https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/ --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies=ms.txt', 'https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.03.01
[debug] Python version 3.6.1 (CPython) - Linux-3.10.0-862.3.2.el7.x86_64-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 3.4.5, ffprobe 3.4.5
[debug] Proxy map: {}
[FOX] Downloading JSON metadata
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 605, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

youtube-dl --version
2019.03.01

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 1, 2019

most likely an invalid cookies file, a valid mvpd-auth cookie should be present in the cookies file.

youtube-dl --cookies cookies.txt https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading JSON metadata
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading JSON metadata
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading m3u8 information
[download] Destination: Season Premiere - Mask On Face Off-54074345bfce9b3069c4523fb673a41a.mp4
ffmpeg version n4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20181127
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[hls,applehttp @ 0x55bf2537b3c0] Opening 'https://content-ausc2.uplynk.com/check2?b=7f6a5e9fa09c4da2a352aa6a7f72283f&v=bc7f45ebfc0042178bd8844a0f9c69ca&r=i&pbs=cc2603e892234e82a88e893bdf1e231d' for reading
[hls,applehttp @ 0x55bf2537b3c0] Opening 'crypto+https://content-ausc2.uplynk.com/vodredir4/bc7f45ebfc0042178bd8844a0f9c69ca/I00000000.ts?ptime=0&pbs=cc2603e892234e82a88e893bdf1e231d&seq=0&o=https%3A//x-live-fox-stgec.uplynk.com/ause/slices/7f6/4f59ea65df184a2580ad8fc80d3eaa32/7f6a5e9fa09c4da2a352aa6a7f72283f/I00000000.ts%3Fpbs%3Dcc2603e892234e82a88e893bdf1e231d%26si%3D0' for reading
[mpegts @ 0x55bf259d5e40] Invalid timestamps stream=1, pts=0, dts=8589930814, size=53195
Input #0, hls,applehttp, from 'https://content-ausc2.uplynk.com/bc7f45ebfc0042178bd8844a0f9c69ca/j.m3u8?pbs=cc2603e892234e82a88e893bdf1e231d':
  Duration: 00:54:12.36, start: 0.000000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 60 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:2: Data: timed_id3 (ID3  / 0x20334449)
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Season Premiere - Mask On Face Off-54074345bfce9b3069c4523fb673a41a.mp4.part':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (HE-AAC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=   60 fps=0.0 q=-1.0 size=     256kB time=00:00:02.47 bitrate= 847.6kbits
@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 2, 2019

first check that the cookie line for mvpd-auth cookie is not preceded by #HTTPOnly_ prefix.

@roadzy
Copy link
Author

@roadzy roadzy commented Mar 10, 2019

@remitamine

The cookie has the following
www.fox.com FALSE / FALSE 1583739011 mvpd-auth
and above it is
www.fox.com FALSE / FALSE 0 pp-auth-token

I don't see and HTTPOnly anywhere in the file.

youtube-dl --cookies ms.txt https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/ --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'ms.txt', 'https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.03.09
[debug] Python version 3.6.1 (CPython) - Linux-3.10.0-862.3.2.el7.x86_64-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 3.4.5, ffprobe 3.4.5
[debug] Proxy map: {}
[FOX] Downloading JSON metadata
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 619, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

The thing I don't get is out of 10 different videos some work and other don't.

Here is the list I'm trying:

Failed youtube-dl --cookies ms.txt https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/
Failed youtube-dl --cookies ms.txt https://www.fox.com/watch/2a6fd26abb3bd6a07544962815e308fa/
Failed youtube-dl --cookies ms.txt https://www.fox.com/watch/779acabe11f4db0b1bbf3f89dd5ae3a1/
Failed youtube-dl --cookies ms.txt https://www.fox.com/watch/83901f8388e3e22f4112e76b75faca87/
Failed youtube-dl --cookies ms.txt https://www.fox.com/watch/ca4a5508cb6cce357a34e5c23c1b6a0a/
Worked youtube-dl --cookies ms.txt https://www.fox.com/watch/903b56b407eb30598797448290329afa/
Worked youtube-dl --cookies ms.txt https://www.fox.com/watch/c397cd3b15e93a5a5b37e9f66e85abcf/
Worked youtube-dl --cookies ms.txt https://www.fox.com/watch/5a73b942f987ef842c7c73f68a0d21cb/
Worked youtube-dl --cookies ms.txt https://www.fox.com/watch/cff4661b61fbe9c225e71373d58a6f8c/

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 10, 2019

The thing I don't get is out of 10 different videos some work and other don't.

the working videos are free don't require an account.
all the URLs you have posted work without a problem, not sure what's wrong on you're side.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 10, 2019

i have pushed some code to detect some of the errors, you can report back when you test the new version.

remitamine added a commit that referenced this issue Mar 10, 2019
@roadzy
Copy link
Author

@roadzy roadzy commented Mar 11, 2019

Can you walk me through testing this new code? Do i have do a git pull or should i just back up a file and do a copy and paste of the new code you added. (sorry I'm new to github)

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 11, 2019

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 18, 2019

you can check again with the latest version and post the error you're getting.

@roadzy
Copy link
Author

@roadzy roadzy commented Mar 18, 2019

youtube-dl --cookies ms.txt https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'ms.txt', 'https://www.fox.com/watch/54074345bfce9b3069c4523fb673a41a/', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.03.18
[debug] Python version 3.6.1 (CPython) - Linux-3.10.0-862.3.2.el7.x86_64-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 3.4.5, ffprobe 3.4.5
[debug] Proxy map: {}
[FOX] Downloading JSON metadata
[FOX] 54074345bfce9b3069c4523fb673a41a: Downloading JSON metadata
ERROR: This video is only available via cable service provider subscription. You may want to use --cookies.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/fox.py", line 67, in _call_api
video_id, data=data, headers=headers)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 891, in _download_json
expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 869, in _download_json_handle
expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/adobepass.py", line 1332, in _download_webpage_handle
*args, **compat_kwargs(kwargs))
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 659, in _download_webpage_handle
urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 644, in _request_webpage
raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2225, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/fox.py", line 67, in _call_api
video_id, data=data, headers=headers)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 891, in _download_json
expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 869, in _download_json_handle
expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/adobepass.py", line 1332, in _download_webpage_handle
*args, **compat_kwargs(kwargs))
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 659, in _download_webpage_handle
urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 644, in _request_webpage
raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 794, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 529, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/fox.py", line 96, in _real_extract
video = self._call_api('vodplayer/' + video_id, video_id)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/fox.py", line 76, in _call_api
'subscription. You may want to use --cookies.', expected=True)
youtube_dl.utils.ExtractorError: This video is only available via cable service provider subscription. You may want to use --cookies.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 18, 2019

invalid cookie file, the mvpd-auth cookie is missing, expired or has malformed value.

@roadzy
Copy link
Author

@roadzy roadzy commented Mar 18, 2019

hmm i've tried clearing my cache multiple times... i've logged in to my provider several times. Only other thing is I'll have to find out where the cookies are stored and delete them manually.

@roadzy
Copy link
Author

@roadzy roadzy commented Mar 18, 2019

hmm i've tried clearing my cache multiple times... i've logged in to my provider several times. Only other thing is I'll have to find out where the cookies are stored and delete them manually.

Ok I found where google chrome is storing cookies... made sure to remove all cookies manually. Restarted browser and then logged in again. Same cookie message.

See this is the issue. Tried this video with the same cookie file:

https://www.fox.com/watch/cff4661b61fbe9c225e71373d58a6f8c/

Works fine. Any idea why one video works with a cookie but the other doesn't? Both videos in question work fine in the browser.

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Mar 18, 2019

as I said before #20208 (comment), free videos do not require using a cookie file at all.

something wrong with the cookies that you're exporting, the way I export the cookies is using cookies.txt extension for Firefox if this configuration did not work for you than i'm afraid that I can't help you.

@roadzy
Copy link
Author

@roadzy roadzy commented Mar 18, 2019

as I said before #20208 (comment), free videos do not require using a cookie file at all.

something wrong with the cookies that you're exporting, the way I export the cookies is using cookies.txt extension for Firefox if this configuration did not work for you than i'm afraid that I can't help you.

I just tried it with out cookies and it seems to be working. I guess I assumed they all required cookies to watch. Thanks for the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.