[NoodleMagazine] Unable to download JSON metadata: HTTP Error 403: Forbidden

[tamham23]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm reporting that yt-dlp is broken on a supported site
• I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

UK

Provide a description that is worded well enough to be understood

Getting the 403 forbidden error when downloading from noodlemagazine.com. Works fine from browsers. Login is not necessary. Tried with vpn, used --user-agent and --cookies parameters, updated yt_dlp using master branch, but all to no avail.

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36', '--cookies', '.\\cookies.txt', 'https://noodlemagazine.com/watch/-221537140_456239050']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481]
[debug] Lazy loading extractors is disabled
[debug] Python 3.11.4 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1u  30 May 2023)
[debug] exe versions: ffmpeg 2023-07-19-git-efa6cec759-full_build-www.gyan.dev (setts), ffprobe 2023-07-19-git-efa6cec759-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1863 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
yt-dlp is up to date (stable@2023.07.06)
[NoodleMagazine] Extracting URL: https://noodlemagazine.com/watch/-221537140_456239050
[NoodleMagazine] -221537140_456239050: Downloading webpage
[NoodleMagazine] -221537140_456239050: Downloading JSON metadata
ERROR: [NoodleMagazine] -221537140_456239050: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: Forbidden>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 715, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\noodlemagazine.py", line 41, in _real_extract
    playlist_info = self._download_json(f'https://adult.noodlemagazine.com/playlist/{video_id}?m={key}', video_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 1069, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 1033, in download_handle
    res = self._download_webpage_handle(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 903, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 860, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_urllib.py", line 438, in _send
    res = opener.open(urllib_req, timeout=float(request.extensions.get('timeout') or self.timeout))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\YoutubeDL.py", line 4059, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\common.py", line 114, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_helper.py", line 203, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\common.py", line 325, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_urllib.py", line 443, in _send
    raise HTTPError(UrllibResponseAdapter(e.fp), redirect_loop='redirect error' in str(e)) from e
yt_dlp.networking.exceptions.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 847, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\YoutubeDL.py", line 4078, in urlopen
    raise _CompatHTTPError(e) from e
yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 403: Forbidden

[bashonly]

Looks like Cloudflare anti-bot fingerprinting is the culprit here. Try with fresh cookies -- you need to have loaded the webpage in browser and exported cookies within the past 30 minutes: https://github.com/yt-dlp/yt-dlp/wiki/FAQ#im-getting-http-error-403-and-the-site-has-an-open-issue-on-the-tracker-thats-labeled-cloudflare-related-what-can-i-do

[tamham23]

Already tried that with fresh cookies, but still the same error.

[bashonly]

Then the Cloudflare protection is strong with this one. Probably needs #7595

[gamer191]

Then the Cloudflare protection is strong with this one. Probably needs #7595

The API it's calling is https://adult.noodlemagazine.com/playlist/-221537140_456239050?m=de6bbd3099077e8f0ca0c194235d47b6. Even once you get past cloudflare's captcha (which it sends browsers as well) you just get an empty json ({}).

I'm not gonna actually visit that website, but my guess is that their current API isn't cloudflare protected, and for some reason they've decided to cloudflare-protect their old API, instead of shutting it down

[bashonly]

May will be fixed by #7830

[tamham23]

Yes, #7830 fixes the issue, but need to provide any non-whitespace value for the --referer header, as mentioned in it.