Add support for Bilibili international (and download subtitles) #29279

minhngoc25a · 2021-06-12T17:55:07Z

Checklist

I'm reporting a new site support request
I've verified that I'm running youtube-dl version 2021.06.06
I've checked that all provided URLs are alive and playable in a browser
I've checked that none of the provided URLs violate any copyrights
I've searched the bug tracker for similar site support requests including closed ones

Example URLs

Single video: https://www.biliintl.com/en/play/35452
Homepage: https://www.biliintl.com/

Description

Bilibili is like Youtube but in China. It also has an international site, which is used for Chinese animation and anime. The international version doesn't have a search bar, and you can't switch the subtitle on your own. "What's Wrong with My Big Brother" is the series name of the example URLs. It was used as an example since it's not geo-restricted. To get to my point, the Bilibili app on Android and iOS has more features such as switching subtitles, commenting and searching series. I've looked for many ways to switch the subtitle language to another one to download it, but it always ends up with the top one (when sorted by A-Z). Therefore, along with video downloading, I would like someone with great knowledge in this field to implement a multilanguage subtitle download solution. Note that Bilibili subtitles are usually in the JSON format (there is a converter for this) or CC and BCC (related repo).

SuperSonicHub1 · 2021-06-16T15:09:05Z

@minhngoc25a Doing some early research, the single video link isn't a single video, it's a season. This is a single video: https://www.biliintl.com/en/play/35452/357787

minhngoc25a · 2021-06-16T15:35:33Z

@SuperSonicHub1 I've just recently found out that Subtitle Edit support Bilibili JSON subtitle format (.bcc or .json).

SuperSonicHub1 · 2021-06-16T15:37:53Z

@minhngoc25a

I've just recently found out that Subtitle Edit support Bilibili JSON subtitle format (.bcc or .json).

youtube-dl should still be able to parse and convert it without outside help, and we should also include the original JSON just for good measure.

SuperSonicHub1 · 2021-06-16T17:05:57Z

@minhngoc25a The site also supports Siamese, using the URL scheme https://www.biliintl.com/th/

SuperSonicHub1 · 2021-06-16T18:21:14Z

How to Crack Bilibili International

Parse URL

https://www.biliintl.com/en/play/35452/357787
https://www.biliintl.com/th/play/35452/357787
https://www.biliintl.com/<lang>/play/<season_id>/<episode_id:optional>

== OR ==

https://www.biliintl.com/en/play/35452/357787?s_locale=en_US
https://www.biliintl.com/en/play/35452/357787?s_locale=th_TH
https://www.biliintl.com/en/play/<season_id>/<episode_id:optional>?s_locale=<locale>

Season

Get youtube-dl metadata from HTML page

title => meta[itemprop=name] || meta[property='og:title'] (remove `_bilibili`)
description => meta[itemprop=description] || meta[property='og:type']

Download JSON from https://api.biliintl.com/intl/gateway/web/view/ogv_collection?s_locale=<locale>&season_id=<season_id>
- if lang == 'th': locale == 'th_TH'
- if lang == 'en': locale == 'en_US'
Iterate over each episode in root["data"]["episodes"]
- use episode['ep_id'] and episode['long_title'] to create a URL result
  - InfoExtractor.url_result("https://www.biliintl.com/{lang}/play/{season_id}/{episode_id:optional}", video_title=long_title, video_id=ep_id)

Episode

Get youtube-dl metadata through methods referenced in the season section
- This website's API and HTML are surprisingly lean, so there isn't a lot of meat to pick off the bone.
Download JSON from https://api.biliintl.com/intl/gateway/web/playurl?ep_id=<episode_id>&s_locale=<locale>&device=wap&platform=web&qn=64&tf=0&type=0
- Look to the season section for determining locale
Set playurl to root['data']['playurl']
Get youtube-dl metadata from playurl
- "duration"
Get audio streams by iterating over playurl['audio_resource']
- "url"
- metadata
  - "codecs"
  - "mimetype"
  - "size"
Get video streams by iterating over playurl['video']
- "video_resource"
  - "url" (sometimes)
  - "id"
    - Combine the path with domain upos-bstar-mirrorakam.akamaized.net to get a URL
    - Requires the URL param hdnts; I don't get how it works: exp=1623873893~hmac=b6f9ae587367bce35a28498735e7dcdf68795003b3f61ebf4f8387ac171c7dd2
  - "codecs"
  - "mimetype"
  - "size"
  - "width"
  - "height"
- "stream_info"
  - "desc_words", "desc_text"

youtube-dl will have to mux the video and audio streams in post using a post-processor, a la the YouTube extractor.

Input would be heavily appreciated on generating URLs for the other video streams. I'll probably not make the extractor myself, but I hope this is helpful to whoever does.

SuperSonicHub1 · 2021-06-16T18:23:32Z

Oh, totally forgot subtitles! Those can be requested with https://api.biliintl.com/intl/gateway/m/subtitle?ep_id=<episode_id>&s_locale=<locale>. They're in a proprietary JSON format; look to OP for parsing them.

SuperSonicHub1 · 2021-06-16T18:27:05Z

Another amendment; in order to get the proper subtitle URL, you'll need to iterate over each item in root["data"]["subtitles"] to get the language (item["key"]) you want and request item["url"]. item["title"] gives a more formal subtitle name. Didn't want to forget this since downloading subtitles is the reason @minhngoc25a made this request in the first place!

SuperSonicHub1 · 2021-06-16T18:40:13Z

Yet another amendment; when doing that season request, if a "show" on BiliIntl has more than one season, there will be a seasons array within root["data"] giving us all the season IDs of a show.

{
    "seasons": [
      {
        "title": "S1",
        "season_id": 37653
      },
      {
        "title": "S2",
        "season_id": 1005556
      }
    ]
}

The episodes array will only contain the episodes relevant to the season ID requested. When a youtube-dl user gives us a URL that contains more than one season, should we download only the season from the season ID in the URL, or should we iterate over the seasons array and make multiple requests?

minhngoc25a · 2021-06-26T07:13:34Z

I think it should be only the season ID in the URl.

meowthed · 2021-08-01T02:43:54Z

there's also global.bilibili.com, which is more of the same, best include that too.

Kuju29 · 2021-10-16T01:29:22Z

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

minhngoc25a · 2021-10-16T02:03:41Z

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

It was implemented in yt-dlp already. This repo is currently less active, maybe because the devs lost interest or IRL issues.

AdvMaple · 2021-10-16T02:06:35Z

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

You can use this script I wrote to download video, audio, sub: https://github.com/AdvMaple/biliintl_subtitle_download_plugin

I haven't implement language choosing, but You can change it in the code.

If you have difficulty installing or changing the code, just open an issue in the repo.

Kuju29 · 2021-10-16T02:08:47Z

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

It was implemented in yt-dlp already. This repo is currently less active, maybe because the devs lost interest or IRL issues.

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

You can use this script I wrote to download video, audio, sub: https://github.com/AdvMaple/biliintl_subtitle_download_plugin

I haven't implement language choosing, but You can change it in the code.

If you have difficulty installing or changing the code, just open an issue in the repo.

.tv not .com

AdvMaple · 2021-10-16T02:16:11Z

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

It was implemented in yt-dlp already. This repo is currently less active, maybe because the devs lost interest or IRL issues.

is it possible to add support for download subtitles from Bilibili.tv (.tv not .com)

You can use this script I wrote to download video, audio, sub: https://github.com/AdvMaple/biliintl_subtitle_download_plugin
I haven't implement language choosing, but You can change it in the code.
If you have difficulty installing or changing the code, just open an issue in the repo.

.tv not .com

I just tested it, still work, change change the "match" URL to fit your need.

At line 6 to "bilibili.tv":

Kuju29 · 2021-10-16T02:18:10Z

เป็นไปได้ไหมที่จะเพิ่มการรองรับคำบรรยายดาวน์โหลดจาก Bilibili.tv (.tv ไม่ใช่ .com)

มันถูกนำไปใช้ใน yt-dlp แล้ว ขณะนี้ repo นี้มีการใช้งานน้อยลง อาจเป็นเพราะนักพัฒนาสูญเสียความสนใจหรือปัญหา IRL

เป็นไปได้ไหมที่จะเพิ่มการรองรับคำบรรยายดาวน์โหลดจาก Bilibili.tv (.tv ไม่ใช่ .com)

คุณสามารถใช้สคริปต์นี้ที่ฉันเขียนเพื่อดาวน์โหลดวิดีโอ เสียง ย่อย: https://github.com/AdvMaple/biliintl_subtitle_download_plugin
ฉันยังไม่ได้ใช้การเลือกภาษา แต่คุณสามารถเปลี่ยนได้ในโค้ด
หากคุณมีปัญหาในการติดตั้งหรือเปลี่ยนรหัส ให้เปิดปัญหาในที่เก็บ

.tv ไม่ใช่ .com

ฉันเพิ่งทดสอบมัน ยังคงใช้งานได้ เปลี่ยนเปลี่ยน URL "ที่ตรงกัน" เพื่อให้เหมาะกับความต้องการของคุณ

ที่บรรทัดที่ 6 ถึง "bilibili.tv":

Thank you very much 💯

AdvMaple · 2021-10-16T02:41:11Z

How do I add a plugin? I use windows

Go open issue in my repo. I don't want to clutterr this youtube-dl issue with other script problems. Sorry for causing you hassle.

softsubs · 2021-10-19T12:25:14Z

@AdvMaple help me, I have installed tampermonkey+script and changed the domain to bilibili.tv but tampermonkey when I check 'no scripts is running' the script is enabled, how to make this script active when I open bilibili.tv . thank you

AdvMaple · 2021-10-19T12:44:46Z

@AdvMaple help me, I have installed tampermonkey+script and changed the domain to bilibili.tv but tampermonkey when I check 'no scripts is running' the script is enabled, how to make this script active when I open bilibili.tv . thank you

Open on issue in my repo, I don't want to clutter yt-dl with my script problem

softsubs · 2021-10-20T01:23:07Z

@AdvMaple help me, I have installed tampermonkey+script and changed the domain to bilibili.tv but tampermonkey when I check 'no scripts is running' the script is enabled, how to make this script active when I open bilibili.tv . thank you

Open on issue in my repo, I don't want to clutter yt-dl with my script problem

@AdvMaple yes, I have checked your repo and the results are the same, can you share the steps in the form of a video, it really helped me, you can share it on youtube or something :)

AdvMaple · 2021-10-20T01:48:00Z

https://user-images.githubusercontent.com/46006210/138014304-c4fa2476-bbc5-4f7e-8edc-3ccc91b474c5.mp4
I will add a voice-over youtube video if this isn't enough

AdvMaple · 2021-10-20T02:11:51Z

https://user-images.githubusercontent.com/46006210/138014304-c4fa2476-bbc5-4f7e-8edc-3ccc91b474c5.mp4 I will add a voice-over youtube video if this isn't enough

@AdvMaple help me, I have installed tampermonkey+script and changed the domain to bilibili.tv but tampermonkey when I check 'no scripts is running' the script is enabled, how to make this script active when I open bilibili.tv . thank you

Open on issue in my repo, I don't want to clutter ytdl with my script problem

@AdvMaple yes, I have checked your repo and the results are the same, can you share the steps in the form of a video, it really helped me, you can share it on youtube or something :)

Bilibili also change their URL from: www.biliint.com to www.bilibili.tv . I've updated my script to match their change. You probably test my script when I haven't known about these changes.

softsubs · 2021-10-20T11:35:01Z

https://user-images.githubusercontent.com/46006210/138014304-c4fa2476-bbc5-4f7e-8edc-3ccc91b474c5.mp4 I will add a voice-over youtube video if this isn't enough

@AdvMaple help me, I have installed tampermonkey+script and changed the domain to bilibili.tv but tampermonkey when I check 'no scripts is running' the script is enabled, how to make this script active when I open bilibili.tv . thank you

Open on issue in my repo, I don't want to clutter ytdl with my script problem

@AdvMaple yes, I have checked your repo and the results are the same, can you share the steps in the form of a video, it really helped me, you can share it on youtube or something :)

Bilibili also change their URL from: www.biliint.com to www.bilibili.tv . I've updated my script to match their change. You probably test my script when I haven't known about these changes.

@AdvMaple yes, this is very valuable for a money :) . Thank you for your hard work and the results are very satisfying. I have tried and it works :)

minhngoc25a added the site-support-request Add extractor(s) for a new domain label Jun 12, 2021

minhngoc25a mentioned this issue Aug 2, 2021

Add support for Bilibili international (and download subtitles) yt-dlp/yt-dlp#611

Closed

5 tasks

minhngoc25a closed this as completed Oct 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Bilibili international (and download subtitles) #29279

Add support for Bilibili international (and download subtitles) #29279

minhngoc25a commented Jun 12, 2021

SuperSonicHub1 commented Jun 16, 2021

minhngoc25a commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021 •

edited

SuperSonicHub1 commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021 •

edited

SuperSonicHub1 commented Jun 16, 2021

minhngoc25a commented Jun 26, 2021

meowthed commented Aug 1, 2021

Kuju29 commented Oct 16, 2021

minhngoc25a commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 •

edited

Kuju29 commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 •

edited

Kuju29 commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 •

edited

softsubs commented Oct 19, 2021

AdvMaple commented Oct 19, 2021

softsubs commented Oct 20, 2021

AdvMaple commented Oct 20, 2021 •

edited

AdvMaple commented Oct 20, 2021

softsubs commented Oct 20, 2021

Add support for Bilibili international (and download subtitles) #29279

Add support for Bilibili international (and download subtitles) #29279

Comments

minhngoc25a commented Jun 12, 2021

Checklist

Example URLs

Description

SuperSonicHub1 commented Jun 16, 2021

minhngoc25a commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021 • edited

SuperSonicHub1 commented Jun 16, 2021

How to Crack Bilibili International

Parse URL

Season

Episode

SuperSonicHub1 commented Jun 16, 2021

SuperSonicHub1 commented Jun 16, 2021 • edited

SuperSonicHub1 commented Jun 16, 2021

minhngoc25a commented Jun 26, 2021

meowthed commented Aug 1, 2021

Kuju29 commented Oct 16, 2021

minhngoc25a commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 • edited

Kuju29 commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 • edited

Kuju29 commented Oct 16, 2021

AdvMaple commented Oct 16, 2021 • edited

softsubs commented Oct 19, 2021

AdvMaple commented Oct 19, 2021

softsubs commented Oct 20, 2021

AdvMaple commented Oct 20, 2021 • edited

AdvMaple commented Oct 20, 2021

softsubs commented Oct 20, 2021

SuperSonicHub1 commented Jun 16, 2021 •

edited

SuperSonicHub1 commented Jun 16, 2021 •

edited

AdvMaple commented Oct 16, 2021 •

edited

AdvMaple commented Oct 16, 2021 •

edited

AdvMaple commented Oct 16, 2021 •

edited

AdvMaple commented Oct 20, 2021 •

edited