bilibili video related to festivel event cannot be downloaded

[matsumotokenji]

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

• I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

• I'm reporting a broken site
• I've verified that I'm running yt-dlp version 2023.01.06 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

Malaysia

Provide a description that is worded well enough to be understood

Bilibili Videos related to special festival events cannot be downloaded as the page is directed to a different templete. On previous version of yt-dlp (22.09.01) low quality version can still be downloadable but with latest version nothing can be downloaded.

example video
https://www.bilibili.com/video/BV1ay4y1d77f/
https://www.bilibili.com/video/BV1hD4y117je/

Provide verbose output that clearly demonstrates the problem

• Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
• Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

>yt-dlp.exe  --cookies bilibili_cookies.txt https://www.bilibili.com/video/BV1ay4y1d77f/ -vU
[debug] Command-line config: ['--cookies', 'bilibili_cookies.txt', 'https://www.bilibili.com/video/BV1ay4y1d77f/', '-vU']
[debug] Encodings: locale cp932, fs utf-8, pref cp932, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.06 [6becd25] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 2022-01-17-git-dcc9454ab9-full_build-www.gyan.dev (setts), ffprobe 2022-01-17-git-dcc9454ab9-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.16.0, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1760 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.01.06, Current version: 2023.01.06
yt-dlp is up to date (2023.01.06)
[BiliBili] Extracting URL: https://www.bilibili.com/video/BV1ay4y1d77f/
[BiliBili] 1ay4y1d77f: Downloading webpage
ERROR: [BiliBili] 1ay4y1d77f: Unable to extract play info; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 680, in extract
  File "yt_dlp\extractor\bilibili.py", line 288, in _real_extract
  File "yt_dlp\extractor\common.py", line 1242, in _search_json
  File "yt_dlp\extractor\common.py", line 1228, in _search_regex

[ZekunZh]

Same as above. Any updates ?

[gamer191]

I don't know any Japanese, but according to a git bisect, prior to ad97487 an 18 minute video downloaded, and after that commit, no video downloaded.

That means that an 18 minute video will download on version 2022.10.04, but not on later versions

Here is a verbose log of the first bad commit:

[debug] Command-line config: ['https://www.bilibili.com/video/BV1ay4y1d77f/', '--verbose']
[debug] User config "C:\Users\jaybu\AppData\Roaming\yt-dlp\config.txt": ['--ffmpeg-location', 'C:\\Users\\jaybu\\ffmpeg\\bin', '-P', 'C:\\Users\\jaybu\\youtube.dl']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f27] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: ad9748760
[debug] Python 3.10.8 (CPython 64bit) - Windows-10-10.0.19045-SP0
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffmpeg" -bsfs
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffprobe" -bsfs
[debug] exe versions: ffmpeg N-106498-g854615adf2-20220405 (setts), ffprobe N-106498-g854615adf2-20220405, phantomjs 2.1.1
[debug] Optional libraries: certifi-2022.09.24, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1702 extractors
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV1ay4y1d77f/
[BiliBili] 1ay4y1d77f: Downloading webpage
ERROR: [BiliBili] 1ay4y1d77f: Unable to extract play info; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\common.py", line 674, in extract
    ie_result = self._real_extract(url)
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\bilibili.py", line 289, in _real_extract
    play_info = self._search_json(r'window.__playinfo__\s*=', webpage, 'play info', video_id)['data']
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\common.py", line 1242, in _search_json
    json_string = self._search_regex(
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\common.py", line 1228, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)

Here's a verbose log of the last good commit:

[debug] Command-line config: ['https://www.bilibili.com/video/BV1ay4y1d77f/', '--verbose']
[debug] User config "C:\Users\jaybu\AppData\Roaming\yt-dlp\config.txt": ['--ffmpeg-location', 'C:\\Users\\jaybu\\ffmpeg\\bin', '-P', 'C:\\Users\\jaybu\\youtube.dl']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f27] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: e091fb92d
[debug] Python 3.10.8 (CPython 64bit) - Windows-10-10.0.19045-SP0
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffmpeg" -bsfs
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffprobe" -bsfs
[debug] exe versions: ffmpeg N-106498-g854615adf2-20220405 (setts), ffprobe N-106498-g854615adf2-20220405, phantomjs 2.1.1
[debug] Optional libraries: certifi-2022.09.24, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1701 extractors
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV1ay4y1d77f/
[BiliBili] 1ay4y1d77f: Grabbing original ID via API
[BiliBili] 778246196: Downloading webpage
[BiliBili] 778246196: Extracting videos in anthology
[BiliBili] Downloading anthology 778246196 - add --no-playlist to just download video
[download] Downloading playlist: 2023崩坏3新春会
[BiliBili] Playlist 2023崩坏3新春会: Downloading 1 videos of 1
[download] Downloading video 1 of 1
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV1ay4y1d77f?p=1
[BiliBili] 1ay4y1d77f: Grabbing original ID via API
[BiliBili] 778246196: Downloading webpage
[BiliBili] 778246196: Downloading video info page
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
WARNING: [BiliBili] unable to extract title; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[BiliBili] 778246196: Extracting videos in anthology
WARNING: [BiliBili] unable to extract description; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [BiliBili] unable to extract og:image; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[BiliBili] 778246196: Downloading tags
WARNING: Extractor failed to obtain "title". Creating a generic title instead
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 778246196_part1: Downloading 1 format(s): 0
[debug] Invoking http downloader on "http://upos-hz-mirrorakam.akamaized.net/upgcxcode/64/03/974030364/974030364_nb3-1-16.mp4?e=ig8euxZM2rNcNbuVhwdVtWuVhwdVNEVEuCIv29hEn0lqXg8Y2ENvNCImNEVEUJ1miI7MT96fqj3E9r1qNCNEto8g2ENvN03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&deadline=1677912863&gen=playurl&nbs=1&oi=455093723&os=akam&platform=pc&trid=a78526bf0cb24a788038cf015ee6c13d&uipk=5&upsig=eb8bd70b7bca365ac25866130419ce76&uparams=e,deadline,gen,nbs,oi,os,platform,trid,uipk&hdnts=exp=1677912863~hmac=e6551decf10e16784d008b7670893325c0f4963ee3d3e91dd6f7e16e657d976f&mid=0"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: C:\Users\jaybu\youtube.dl\BiliBili video #778246196_part1 [778246196_part1].mp4
[download] 100% of   29.96MiB in 00:00:31 at 984.04KiB/s
[download] Finished downloading playlist: 2023崩坏3新春会

[gamer191]

PS: I don't know why @matsumotokenji passed cookies in their verbose log, but this issue definitely doesn't require cookies to reproduce

[pukkandan]

@lockmatrix

[lockmatrix]

It is a site-bug, these festival video pages is not same as other normal videos.
Current bilibili extractor can not process it.

I personally have no plans to fix it right now due to the small number of videos involved.

Volunteers are needed

[lockmatrix]

These URL is redirect to https://www.bilibili.com/festival/,
I suggest adding a new extractor for it, and add a check-and-call in BiliBiliIE

[gamer191]

For users: as a temporary workaround, run yt-dlp --update-to stable@2022.10.04 (or yt-dlp --update-to 2022.10.04) before you download the video, and run yt-dlp --update after you're done

For developers: ad97487 is a rather big commit, so I decided to run a bisect to see which individual commit of #4945 has the regression. Believe it or not, it is the first commit (8eee34c), although that commit gives a slightly different error message:

[debug] Command-line config: ['https://www.bilibili.com/video/BV1ay4y1d77f/', '--verbose']
[debug] User config "C:\Users\jaybu\AppData\Roaming\yt-dlp\config.txt": ['--ffmpeg-location', 'C:\\Users\\jaybu\\ffmpeg\\bin', '-P', 'C:\\Users\\jaybu\\youtube.dl']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.09.01 [5d7c7d656] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: 8eee34cf2
[debug] Python 3.10.8 (CPython 64bit) - Windows-10-10.0.19045-SP0
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffmpeg" -bsfs
[debug] Checking exe version: "C:\Users\jaybu\ffmpeg\bin\ffprobe" -bsfs
[debug] exe versions: ffmpeg N-106498-g854615adf2-20220405 (setts), ffprobe N-106498-g854615adf2-20220405, phantomjs 2.1.1
[debug] Optional libraries: certifi-2022.09.24, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1674 extractors
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV1ay4y1d77f/
[BiliBili] 1ay4y1d77f: Downloading webpage
ERROR: An extractor error has occurred. (caused by KeyError('bvid')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\common.py", line 672, in extract
    ie_result = self._real_extract(url)
  File "C:\GitHub\yt-dlp-OG\yt_dlp\extractor\bilibili.py", line 226, in _real_extract
    bv_id = video_data['bvid']
KeyError: 'bvid'

I also observed some other error messages at different commits, as I was running the bisect. Unfortunately I have closed the command prompt window now, and I don't have the patience to run another git bisect.

[lockmatrix]

For developers: ad97487 is a rather big commit, so I decided to run a bisect to see which individual commit of #4945 has the regression. Believe it or not, it is the first commit (8eee34c), although that commit gives a slightly different error message:

That commit is a complete refactoring (to be precise, it was rewritten), using a different way of getting video information.

If you can write python, you can try to create a patch for https://www.bilibili.com/festival

[dirkf]

Beginnings of a solution at ytdl-org/youtube-dl#31661.

[dirkf]

The approach linked above seems not to work in the yt-dlp extractor because there's no window.__playinfo__. However punting to the old yt-dl extractor gets a giant FLV, as in the quoted log there.

Not entirely on-topic, but relevant to back-porting new IEs from the yt-dlp module:

• the title text isn't quite the same now in test_BilibiliSpacePlaylist
• similarly uploader for test_BilibiliAudio
• similarly title for test_BilibiliAudioAlbum
• similarly title for test_BiliIntl_3 (ends Bilibili, not Bstation).

Obvs may be others in these tests since only the first disagreement is reported in the download tests.

[qbnu]

I submitted a PR (#6547) but I could only test up to 480p. Also I don't know if festival videos can be anthologies, so I couldn't test that.