Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cannot download from bilibili, "no videos found" #10375

Closed
chjlang opened this issue Aug 18, 2016 · 9 comments
Closed

cannot download from bilibili, "no videos found" #10375

chjlang opened this issue Aug 18, 2016 · 9 comments
Labels

Comments

@chjlang
Copy link

@chjlang chjlang commented Aug 18, 2016

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.08.17. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • I've verified and I assure that I'm running youtube-dl 2016.08.17

Before submitting an issue make sure you have:

  • At least skimmed through README and most notably FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

./youtube-dl --verbose http://www.bilibili.com/video/av5782830/index_1.html
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://www.bilibili.com/video/av5782830/index_1.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.08.17
[debug] Python version 2.7.1 - Linux-2.6.18-194.el5-x86_64-with-redhat-5.5-Final
[debug] exe versions: none
[debug] Proxy map: {}
[BiliBili] 5782830: Downloading webpage
[BiliBili] 9388980: Downloading video info page
ERROR: No videos found!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 691, in extract_info
    ie_result = ie.extract(url)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 347, in extract
    return self._real_extract(url)
  File "./youtube-dl/youtube_dl/extractor/bilibili.py", line 122, in _real_extract
    raise ExtractorError('No videos found!')
ExtractorError: No videos found!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
...
<end of log>

@yan12125 yan12125 added the broken-IE label Aug 18, 2016
yan12125 added a commit that referenced this issue Aug 24, 2016
Bilibili now uses emscripten, which is very difficult for reverse
engineering. I don't expect it to be fixed in near future, so I mark
it as broken.

Ref: #10375
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 24, 2016

For those who are interested in Bilibili: this website now uses emscripten, which (indirectly) compiles C/C++ to Javascript. I find it very difficult to analyze. If anyone know how to reverse emscripten-generated codes, please propose here. Thanks!

@remitamine
Copy link
Collaborator

@remitamine remitamine commented Aug 24, 2016

from the mobile website, it possible to extract lower quality using http://www.bilibili.com/m/html5?aid=<aid>&page=1.

@gdkchan
Copy link

@gdkchan gdkchan commented Aug 26, 2016

So I reversed the bilibili signature generation, and here you can find a C# source code that takes a Bilibili video url and returns the stream URL. Meat of the decoding function is on bilibiliPlayer.min.js. It haves a VM that interprets emscripten bytecode like was said here before. It calculates the MD5 Hash of a string to generate the signature. Relevant stuff on this string includes the Video CId, an AppKey (different from the other ones used on the page it seems), and another bigger key that seems to be calculated from one of the "external" (from the VM point of view) Javascript functions.

Worked with the few videos I tested here, if anyone try this out let me know if it's working too. Also if info on how to get the "constant" keys is needed too just let me know.

Note: I guess that the keys change within different player versions (maybe?), will need to observe for some time to see.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 26, 2016

Amazing! I'll try to translate C# to Python.

I guess that the keys change within different player versions (maybe?), will need to observe for some time to see.

It's common in Chinese websites. I have a set of scripts at https://github.com/yan12125/iqiyi-hack that extracts keys automatically by Selenium. It's originally written for Flash players, while it should not be difficult to add codes for Javascript-based players.

@gdkchan
Copy link

@gdkchan gdkchan commented Aug 26, 2016

@yan12125 The code is pretty simple and I added some comments too, should be easy to translate. Also interesting scripts you made, I'm not sure if the 128-bit key can be extracted "automatically" through, since it seems to be generated inside the VM (unless I missed something). It would probably be possible to extract it automatically by having a JavaScript engine running the player code, stopping the thing in the middle and grabbing the Key from the VM Heap, maybe.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 26, 2016

In previous cases, I patch the Flash player so that it posts the key to my server. Then I use an HTTP proxy to intercept requests to the main player. For Bilibili, I guess intercepting the MD5 function is a possible approach.

@yan12125 yan12125 closed this in 04b32c8 Aug 28, 2016
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Aug 28, 2016

Thanks to @gdkchan, BiliBili support will be back in the next version.

@yan12125 yan12125 mentioned this issue Nov 8, 2016
4 of 8 tasks complete
@weituotian
Copy link

@weituotian weituotian commented Apr 28, 2017

@yan12125 how to patch the Flash player?

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Apr 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.