[Bilibili] Extraction broken

[starkshaw]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.10.31. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2016.10.31

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

Description of your issue, suggested solution and other information

The download mission still can be done but the speed is limited at 10kB/s, but viewing from website is fully working. This issue appears shortly after Bilibili announced their membership plan early this month.

[yan12125]

They switched to a new appkey.

[starkshaw]

@yan12125 I remember Bilibili itself does not host videos. All videos are ultimately hosted on other websites, can we track the origin and trigger the download procedure from there?

[yan12125]

can we track the origin and trigger the download procedure from there?

If you can watch the video on the hosting website instead of bilibili, yes.

[Kugel-Blitz]

Bilibili now host some of their videos on their own servers instead of streaming from other servers

[starkshaw]

@yan12125 That's the problem... I am not sure how to track its source.

[starkshaw]

@ztytian What kind of videos are hosted on Bilibili-owned servers?

[yan12125]

I am not sure how to track its source

I don't know either. Here's how Bilibili was cracked the last time: #10375 (comment)

[starkshaw]

@yan12125 Many thanks. I just tried it out. But somehow the speed is still limited. Not too sure how Bilibili works now consider if they are now self-hosting videos. But here's what I've done:

That gist is written in C# but I'm using Mac. So here's what I got from compiled and ran in Mono:

$ ren UnBili UnBili.cs
$ mcs UnBili.cs
$ mono Unbili.exe
Bilibili Stream URL generation test by gdkchan
Enter a video URL to extract the Stream:
http://www.bilibili.com/video/av6973997/

Trying to download the webpage...
Trying to download JSON from "http://interface.bilibili.com/playurl?cid=11366577&appkey=6f90a59ac58a4123&otype=json&type=mp4&quality=2&sign=536c0517ee3691e6bb2d638dcc7d5d37"...

URLs:
http://ws.acgvideo.com/d/72/11366577-1hd.mp4?wsTime=1478619724&wsSecret2=a17fcb85ed3f55a9ee8c3a355bab02a4&oi=2510119029&rate=10

Press any key to exit

Then use Youtube-dl to download that video (but it is not from any big video website in China, such as Youku, Tudou, Sina, etc. Presumably it is a CDN-like website of Bilibili?), it still has that speed limitation of approximately 10kB/s, which is the same of running youtube-dl http://www.bilibili.com/video/av6973997/

In conclusion, I am not sure if that video is truly hosted by Bilibili, or its real address has been casted by Bilibili, and when user refer to the address above, Bilibili will trigger the bandwidth limitation. As far as what I thought, I think there's two ways to bypass this: 1) Find out the real URL. 2) Simulate youtube-dl like a real user (user-agent and cookie?).

[yan12125]

UnBili.cs is for the old appkey, which seems to be semi-blocked now. To get things work, someone need to find out how to use the new appkey.

[cnbeining]

I doubt whether the restriction is only related to appkey, as it seems that rate control is also considering IP frequently.

If the rate is limited, there's little 3rd softwares could do since the limit is hard coded in the URL. What we can do is to export the URLs and download with multiple threads: for now the limit is per thread. However, if the file does not exist at the CDN edge server(most likely would occur on WS servers for oversea users) we have to put up with this for the rate is limited by the bandwidth between China and the edge. One workaround is to use proxies (e.g., those provided by the ISPs) to get links from Bilibili 's own server but expect severe speed limit....

[yan12125]

Now they banned the old appkey. It's a shame of me for not knowing how to crack Emscripten.

[cnbeining]

Cannot find anything useful in the memory.

[starkshaw]

@yan12125 You mean the durl issue? Yeah I just noticed that...

[yan12125]

@starkshaw Yes. Now Bilibili returns this to youtube-dl:

{"code":-5002,"result":"appkey invalid"}

[cnbeining]

Please refer to You-get or Biligrab for a workaround. Yen Chi Hsuan <notifications@github.com>于Sun, Dec 11, 20169:21 AM写道：

…

@starkshaw <https://github.com/starkshaw> Yes. Now Bilibili returns this to youtube-dl: {"code":-5002,"result":"appkey invalid"} — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11077 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABfRgv3yLBPRgwDNAw6Vnbtk0dE4S2Svks5rHAbPgaJpZM4KkdV3> .

[yan12125]

Any reason call it a "workaround"?

[cnbeining]

I didn't manage to get the appsecret either. This method is from the SWF player. Yen Chi Hsuan <notifications@github.com>于Sun, Dec 11, 201611:45 AM写道：

…

Any reason call it a "workaround"? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11077 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABfRgs7dN9y3kblYnVxO1xONe2lMT3YFks5rHCisgaJpZM4KkdV3> .

[remitamine]

the bilibili code used in desktop website has been compiled using -s EMTERPRETIFY=1[1] which will generate a specific bytecode format and an interpreter included with the result js code[2].
[1] https://github.com/kripken/emscripten/wiki/Emterpreter
[2] https://github.com/kripken/emscripten/blob/master/tools/emterpretify.py

so there will be a need either to create a python interpreter for this bytecode or try to translate it into a more understandable format to be able to extract the app_key and secret.

some of the printable string used by the bilibili code:

l`aea>=<bd3?f9d=
yt&zp(txrzry-# &
VTTWQZY
%llx
https://
interface.bilibili.com/playurl?
bangumi.bilibili.com/player/web_api/playurl?
domain error
%u
%d
appkey
cid
otype
json
type
quality
%02x
&sign=
Illegal byte sequence
Domain error
Result not representable
Not a tty
Permission denied
Operation not permitted
No such file or directory
No such process
File exists
Value too large for data type
No space left on device
Out of memory
Resource busy
Interrupted system call
Resource temporarily unavailable
Invalid seek
Cross-device link
Read-only file system
Directory not empty
Connection reset by peer
Operation timed out
Connection refused
Host is down
Host is unreachable
Address in use
Broken pipe
I/O error
No such device or address
Block device required
No such device
Not a directory
Is a directory
Text file busy
Exec format error
Invalid argument
Argument list too long
Symbolic link loop
Filename too long
Too many open files in system
No file descriptors available
Bad file descriptor
No child process
Bad address
File too large
Too many links
No locks available
Resource deadlock would occur
State not recoverable
Previous owner died
Operation canceled
Function not implemented
No message of desired type
Identifier removed
Device not a stream
No data available
Device timeout
Out of streams resources
Link has been severed
Protocol error
Bad message
File descriptor in bad state
Not a socket
Destination address required
Message too large
Protocol wrong type for socket
Protocol not available
Protocol not supported
Socket type not supported
Not supported
Protocol family not supported
Address family not supported by protocol
Address not available
Network is down
Network unreachable
Connection reset by network
Connection aborted
No buffer space available
Socket is connected
Socket not connected
Cannot send after socket shutdown
Operation already in progress
Operation in progress
Stale file handle
Remote I/O error
Quota exceeded
No medium found
Wrong medium type
No error information
0123456789ABCDEF-+
inf
INF
nan
NAN

[Kimi-Arthur]

Cannot download. Getting this error (looks like no one is posting a similar log in this bug):

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://www.bilibili.com/video/av2044037/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.12.22
[debug] Python version 2.7.13 - Darwin-16.3.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2
[debug] Proxy map: {}
[BiliBili] 2044037: Downloading webpage
[BiliBili] 2044037: Downloading video info page
ERROR: An extractor error has occurred. (caused by KeyError(u'durl',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 358, in extract
return self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/bilibili.py", line 65, in _real_extract
for idx, durl in enumerate(video_info['durl']):
KeyError: u'durl'
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 364, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'durl',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[itsuwari]

It's may be better to take a look at their HTML5 version, switch UA to mobile devices or MacOS/Linux.
I can get download address, with ghost.py. I'm not sure how the download url generated.
And the newest version is not fixed.
youtube-dl http://www.bilibili.com/video/av7843600/index_3.html --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://www.bilibili.com/video/av7843600/index_3.html', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.05
[debug] Python version 2.7.9 - Linux-4.9.0-040900rc8-generic-x86_64-with-debian-8.6
[debug] exe versions: avconv 11.8-6, avprobe 11.8-6, ffmpeg N-82963-g9ec52a0, ffprobe N-82963-g9ec52a0
[debug] Proxy map: {}
[BiliBili] 7843600: Downloading webpage
[BiliBili] 7843600: Downloading video info page
ERROR: An extractor error has occurred. (caused by KeyError(u'durl',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 359, in extract
return self._real_extract(url)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/bilibili.py", line 65, in _real_extract
for idx, durl in enumerate(video_info['durl']):
KeyError: u'durl'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 694, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 365, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'durl',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.