DouyuTV live - HTTP error 403 Forbidden

[Lonely233233]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

• I'm reporting that yt-dlp is broken on a supported site
• I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

No response

Provide a description that is worded well enough to be understood

Douyu's live link is invalid

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• If using API, add 'verbose': True to YoutubeDL params instead
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

C:\Users\Administrator>yt-dlp -vU http://www.douyu.com/5168755
[debug] Command-line config: ['-vU', 'http://www.douyu.com/5168755']
[debug] Portable config "E:\Tencent\Mplayer\yt-dlp.conf": ['--output', 'E:\\Tencent\\N_m3u8DL-CLI\\Downloads\\%(title)s-%(section_number)s-%(section_title)s.%(ext)s', '--console-title', '--progress-template', 'download-title:%(info.id)s-%(progress.eta)s', '--merge-output-format', 'mp4', '--no-write-subs', '--external-downloader', 'aria2c', '--cookies-from-browser', 'edge', '--embed-thumbnail', '--embed-metadata']
[debug] Encodings: locale cp936, fs utf-8, pref cp936, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19044-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 6.0-full_build-www.gyan.dev (setts), ffprobe 6.0-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[Cookies] Extracting cookies from edge
[debug] Extracting cookies from: "C:\Users\Administrator\AppData\Local\Microsoft\Edge\User Data\Profile 1\Network\Cookies"
[debug] Found local state file at "C:\Users\Administrator\AppData\Local\Microsoft\Edge\User Data\Local State"
[Cookies] Extracted 544 cookies from edge
[debug] cookie version breakdown: {'v10': 544, 'other': 0, 'unencrypted': 0}
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a9421
yt-dlp is up to date (stable@2023.03.04)
[DouyuTV] Extracting URL: http://www.douyu.com/5168755
[DouyuTV] 5168755: Downloading room info
[DouyuTV] 5168755: Downloading m3u8 information
ERROR: [DouyuTV] 5168755: Failed to download m3u8 information: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 694, in extract
  File "yt_dlp\extractor\douyutv.py", line 115, in _real_extract
  File "yt_dlp\extractor\common.py", line 1941, in _extract_m3u8_formats_and_subtitles
  File "yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 839, in _request_webpage

  File "yt_dlp\extractor\common.py", line 821, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 3742, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

[Lonely233233]

I can't reproduce the issue. Have you modified the source code? I don't see how this error could occur otherwise. These have been lines 107-108 of douyutv.py since the last stable release:

            f'http://www.douyutv.com/api/v1/room/{room_id}', video_id,
            note='Downloading room info', query=params)['data']

Has yours been modified from ['data'] to [data]?

f'http://www.douyutv.com/api/v1/room/{room_id}', video_id,
note='Downloading room info', query=params)['data'] no change

video_url = urljoin('https://openhls-tct.douyucdn2.cn/', self._search_regex(r'(live/.*)', room['hls_url'], 'URL'))
formats, subs = self._extract_m3u8_formats_and_subtitles(video_url.replace('/playlist.m3u8', '.m3u8'), room_id)

Only changed here because hls3-akm cannot be used in China

[Lonely233233]

can you show a verbose log with this error? add -v

Douyu now extracts the live source link and cannot directly use "http://openhls-tct.douyucdn2.cn/dyliveflv1/252140rz7aIq14cx_1024p.flv" must bring txSecret=fea7f993f949de8fae88a25f6a31ca55&txTime=6489e855

Because Douyu now has all the short link formats invalid

[c-basalt]

I've ported a solution from https://github.com/WhiteMinds/DouYuRecorder/, and added PhantomJs to the extractor to get authenticated on the new API:
https://github.com/c-basalt/yt-dlp/tree/douyu

---

yt-dlp/yt_dlp/extractor/douyutv.py

Lines 113 to 116 in 613dbce

	video_url = urljoin('https://hls3-akm.douyucdn.cn/', self._search_regex(r'(live/.*)', room['hls_url'], 'URL'))
	formats, subs = self._extract_m3u8_formats_and_subtitles(video_url, room_id)

This video_url in the current master branch appears to be no longer working. Do we know if this might work in some cases or should we just get rid of these?

---

Also, when I was testing the patch using https://www.douyu.com/998, I noticed that ffmpeg missed a lot of ts fragments from m3u8 link, because the single-threaded download speed of ffmpeg couldn't catch up and playlist reloading was too slow.

One way I figured out is to use streamlink with --stream-segment-threads. A command like this worked for me: streamlink $(py.exe -m yt_dlp -v "https://www.douyu.com/998" --get-url) best --loglevel trace -o "$(date +%s)-test.ts" --logfile "$(date +%s)-streamlink.log" --stream-segment-threads 10

Streamlink without --stream-segment-threads will result in an ever-growing download queue, until all queued items start to receive HTTP 404 and streamlink terminates itself.

Is there any plan that future yt-dlp will have some concurrency for m3u8 livestream (perhaps a native m3u8 downloader for live stream, since I don't feel ffmpeg was designed for live stream)? This can be important in a situation like this.

---

Add: DouyuShow appears to be broken as well

[bashonly]

This video_url in the current master branch appears to be no longer working. Do we know if this might work in some cases or should we just get rid of these?

@c-basalt See the patch here: #6074 (comment)

[c-basalt]

@c-basalt See the patch here: #6074 (comment)

The patch from May 11th gives new m3u8 links like this: https://openhls-tct.douyucdn2.cn/live/288016rlols5.m3u8?wsSecret=3a17750d0910ce6f0b73cd888255eeef&wsTime=1689724357

However, it seems those openhls-tct.douyucdn2.cn links are also giving me HTTP 403. I wonder if something has changed on the server side after that.