Signature Extraction Support for html5fs player type

[kijalab]

Hi, I am getting the encrypted signature from the Youtube URL in my iPhone app and the player type is html5fs. I want to use your scripts to decrypt that signature. I was using your _static_decrypt_signature algorithm but it doesn't seem to work anymore. I'll appreciate if you can help.

Thanks.

[jezcomputer]

Hi, we also use the staticdecrypt_signature function directly also for our app - please could you keep that part active too ; we understand it's internal normally. Is it possible to expose the automatic decrypt directly if we just passing (s) into it ?

Many thanks and keep up the great work

[jhb50]

I have messaged you directly with the details, but I too extract the sig table from youtube.py, convert it to vbs format and use it to convert youtube urls prior to sending them to ffmpeg for playback via DLNA in realtime on devices such as TV's. Is it possible to generate the table as well, or continue to maintain it manually? Thanks again for your efforts.

[qrtt1]

I like the dynamic updating sig rules feature. I don't think to convert it to static code is good idea, because it got broken in uncertainly time. Hosting a decrypt-signature-server is another idea to do signature transforming.

And in the sever side, it could be update every hour by youtube-dl updater or pip updater.

[phihag]

youtube-dl's scope is clear: youtube-dl allows users to list, download, play and convert video and audio files. It can be used in a server or in a mobile application to do that. Only decrypting signatures is, however, out of scope.

What you could do is the following: Start a small open-source project that can be used as a server to extract signatures. This server can then be queried with a player URL and the length of the signature. Given these values, write a small Python function that imports youtube-dl and asks it to extract the signature function (or signature). Since we explicitly do not guarantee the internal workings of youtube-dl and may need or want to change them from time to time, you can then update this server in place.

Feel free to create a repository. I will not create one myself since I won't be using that, and will not support
it. However, if you agree to support it, I can contribute code for it. As long as the project stays well-maintained, we will also try to keep the relevant interfaces in youtube-dl fairly stable, or at least provide advance warning and changes to it.

[jezcomputer]

Today this is how I use the youtube-dl staticdecrypt_signature ; I run this on a server where every 1 hour the local copy of the youtube-dl source is synch'd using SVN ; then I have a php wrapper which is basically a http://usr.com?sig=xxxxxxxxx and simply returns the decrypted signature. The beauty of this approach is that master code i.e. here on github is 100% the same and a simple highlevel wrapper to use the py interface directly. (the PHP code just calling this python script handing down the signature as it goes) - I;m sure the authors of Youtube-DL doesn't realise they are the heart of these systems :-)

php and python snippets

<?php
    $signature = $_GET['signature'];
    $decrypted = shell_exec('python ytsighelper.py ' . escapeshellarg($signature));
    echo $decrypted;    
?>

e.g. (named ytsighelper.py)

#!/usr/bin/python

from sys import argv

#Pull in class from the Source Project and execute the _decrypt_signature method of that class directly
from youtube_dl.extractor.youtube import YoutubeIE


def decrypt(s):
  return YoutubeIE()._decrypt_signature(s, video_id=0, player_url=0, age_gate=0)


if __name__ == "__main__":
    first,second = argv
    print decrypt(second)

[kijalab]

@phihag @jezcomputer Thanks guys. Any idea how to determine the algorithm to decrypt the signature in a static way? I am looking for the logic behind the decrypt function so I can modify the _static_decrypt_signature function as and when Youtube folks change it. Any help will be appreciated. Thanks.

[phihag]

@kijalab Modifying the _static _decrypt_signature function is useless, because that function is only useful for legacy versions of youtube-dl. Instead, you can directly add an output option for the suggested server (there could be a command-line version as well) that uses a similar format.

[kijalab]

@phihag Thanks. What I meant was similar to what you suggested. I am not proposing to modify the _static_decrypt_signature function in youtube-dl. I am maintaining my own python script that just does the static conversion of signature. But I would like to know the logic behind the algorithms to decrypt the signatures statically. Any pointers will be helpful.

[jezcomputer]

@phlhag The automatic decrypt function ; can that be pretty self contained ie. just pass some arguments to fn() and it just some magic between the URL/id and the signature (no other page loading) or does it need a bunch of HTML text to scrap through ? (sorry for not knowing how you approach this - up until this point we been relying on you _static _decrypt_signature function and just passing in the signature as explained in my previous comment)

[phihag]

@kijalab Please read my above post. In short, feel free to start an open-source project that implements what you want.

@jezcomputer The automatic signature decryption needs the swf or HTML5 player code, or the URL of the player. If given the code, it does not do any network interaction. If given the URL, it downloads the URL and then proceeds to operate on the code.

[jhb50]

@phihag Thanks for your server suggestion and offer of assistance. I have no experience in that area so it would all be new to me. Currently with a 2 line mod to youtube-py, I can output a table of decoded signature urls for every available format for a given youtube video id, so I'm thinking a simpler approach would be to have my youtube plugin running under Serviio in windows invoke that modified youtube-py and access the decoded url table for the requested resolution. That way I can continue to use your app with little risk of change, and continue to access encoded signature videos. I will also look at py2exe to see if I can package this for distribution with the plugin.
Let me know if this violates any agreements. Thanks.

[phihag]

@jhb50 I'm a little bit puzzled why you would want to modify youtube-dl to get the URLs. Simply add the --get-url (or -g for short) parameter, like this:

youtube-dl -g http://www.youtube.com/watch?v=CvBfHwUxHIk --all-formats

Note that the other posters only want the algorithm and don't care about individual URLs. Since youtube-dl is licensed under Unlicense, you are free to modify and use it in any way you want.

[jhb50]

Simply add the --get-url LOL Thank you.

[kijalab]

@phihag The URL returned using '-g' option will have the IP address of the server where youtube-dl was executed. But in my case, I want to play the movie in my iPhone app, so, I need to get the URL corresponding to the IP address of my iPhone. Is there a way to get that? That's why, I am fetching the encrypted signature in my iPhone app and passing it to my server that'll decrypt and send back the correct signature which I then add it to the URL I got in my iPhone app code. That's the reason, I was using the algorithms you are using in static_decrypt_signature. Thoughts?

[qrtt1]

@kijalab it needs some hacking for youtube-dl, just moving the download action from server to your client.

[phihag]

@kijalab @qrtt1 I'm fully aware that -g is not sufficient for your needs, but it does solve jhb50's problem. Feel free to start and maintain an open-source project that does precisely what you want, i.e. only handle the signature. As I wrote above, you can simply import youtube-dl and then ask it to decipher a signature.

[jezcomputer]

@phihag Thanks for your feedback - I would like support/start a server wrapper project if possible ; where the server project can drive your code - therefore keeping/re-use to the highest and maintenance of a single area for the updates of the extractors. So developers like myself and the others in the comments can access access the required functionality without causing your project burden. In the end I'm happy to provide a server + php wrappers to drive this. Be great to get a few new public methods to e.g. Pass in the SWF/HTML5 text to avoid any network activity on your module - as explained by @kijalab - our IP's will be different ; so in the end we would can provide the required data to a function call ; resulting in a list of URL's / etc coming back with the correct signatures.

The wrapper idea - mean's that only one core project is then needing updates and the server wrapper just as it says exposes/wraps this.

What do think ? If your happy then I'm sure a few of us are more than happy to make this work - as said ; I have the server and happy to host that type of service a helper service; my server/s already wrap you code with PHP for the static signature extraction today and maybe a good starting point ?

[kijalab]

@phihag Thanks. I'll open a separate project that'll import youtube-dl and if you can support 'youtube_print_sig_code' as one of the parameters (it's there in the code, but no support from the command line parameters) and return just the formula, that'll be very helpful. I can then use the returned formula from youtube-dl to decrypt the signature that I get from my iPhone app.

Also, I have a lame question -- I downloaded the source code of youtube-dl. But how do I keep it updated? Are there any scripts (similar to -U) that'll pull all latest changes from the source code? Should I use git to keep it updated? I just downloaded the tar.gz file.

[phihag]

@jezcomputer The only network activity you're going to need is player download, and players do not vary across IPs. Therefore, it should be perfectly fine to pass in the player URL, to, say, extract_signature_function. Note that I don't really care about hosting the server, I just want it to be open-sourced so that we can improve its functionality, make sure to commit changes in advance when the internals of youtube-dl change, and so that other people can run their own instances.

@kijalab --youtube-print-sig-code is supported from the command line just fine, although we may remove that option. However, that would (like youtube-dl) need a video URL, which as explained above, is not really necessary - all we need for signature decryption is the player URL and signature length. To keep the code updated, clone it via git instead of downloading the tarball and then type git pull.

Should anyone of you decide to create a repository, please drop a link here.

[astazed]

Has anyone managed to get the _decrypt_signature or _extract_signature_function ... functions working out of the box?

The problem I am facing is that I am getting this error AttributeError: 'NoneType' object has no attribute ... for every function call that is outside YoutubeIE (ex: self._downloader.params, self._download_webpage...)
Modifying the code is not a viable option as I would have to rewrite everything with every youtube-dl update.
I wanted to know if there was a way to make things work without initializing the whole project.

I have been using the _static_decrypt_signature function on serverside for quite some time, but lately I started to experience some problems with it not being able to decrypt all signatures. I am glad that there are some who share the same problems, so maybe we can figure out how to make things work again.

Thanks

[qrtt1]

@nagiobeid You can do something like this: https://gist.github.com/qrtt1/6946660

(Official note) As I wrote above, the youtube-dl project will support unusual usages like this one only if they are in a maintained open-source repository. This code does a lot of crazy things which are not advisable, not future-proof, and will not be maintained by youtube-dl. You have been warned. -- @phihag

[astazed]

@qrtt1 Wow! Thanks that really helped

[jhb50]

@phihag Just a big thank you for your work. Invoking youtube-dl all-formats as you suggested directly from my Serviio Youtube plugin returns the decoded signatures perfectly! No more updates required!

[jezcomputer]

Has the --youtube-print-sig-code option been removed ?

[phihag]

@jezcomputer No.

[jezcomputer]

My bad ; was having issue with Vevo video - and didn't see sig-code coming out ; looks like they be remove the sig stuff ?

[cujo30227]

Since 10 hours or so I'm not getting an encrypted signature anymore on any videos. I'm always getting an unencrypted "sig" entry, which can directly be used in the URL... Is youtube preparing something else?!?

Op 16 okt. 2013 om 22:13 heeft jezcomputer notifications@github.com het volgende geschreven:

My bad ; was having issue with Vevo video - and didn't see sig-code coming out ; looks like they be remove the sig stuff ?

—
Reply to this email directly or view it on GitHub.

[phihag]

In the past, signatures were temporarily disabled from time to time. This shouldn't be a problem.

[jezcomputer]

Was trying to use the code posted by qrtt1 - to do the signature stuff ; passing in the signature + videoid ; then drove this via .php script from external app ; was working perfect - but for some strange reason since the signatures are gone cannot get a URL that will work when the client app is in a different country from the server ; however if I try the same country as the server i.e. UK - all is good. Is there some IP magic that goes on ?

[phihag]

@jezcomputer As I wrote above, the youtube-dl project will only support decryption-only functionality only if it happens from a maintained open-source repository. The code posted by @qrtt1, while being well-intentioned, is buggy and not future-proof in many places. That's precisely why I required a maintained open-source project (where such bugs could and would be fixed, and where we could the necessary interfaces static, or adapt them in the project).

Therefore, this is the wrong forum for problems with @qrtt1's code.

[jezcomputer]

@philhag Understand - time to start that open source project I think ;-) Now where to start - I've not been involved with any open source project startup from the start - any ptr's ?

[qrtt1]

@jezcomputer the code is just a proof to leverage the youtube-dl's signature method. If you plan to host a server and use the youtube-dl, just call the youtube-dl to generate a playable url with the customized url-handler which pass the url content from client's data.

[astazed]

@qrtt1 I think that it would only be fair if you started a new repository with the code you provided in this thread. In any other case I wouldn't mind doing so with the code adapted from your post.

I would really want to see a well maintained solution to this problem that isn't bound to break in the future.

[astazed]

Well if anyone is still interested here is a small repo containing my interpretation of decrypting the signatures based on @qrtt1 's implementation.

https://github.com/astazed/YSD

[jezcomputer]

@astazed Yes - I'm interested to support this project also.

[phihag]

Excellent. In the coming days, I'll suggest a couple of minor changes, but otherwise, we'll take care so that your project doesn't break unnecessarily.

[astazed]

@phihag @jezcomputer that is cool. thanks

[M-Shahbaz]

@jezcomputer @astazed I need only signature decipher from youtube-dl. I tried a lot to implement both of your methods but with current version of youtube-dl it seems that is not working.
Can you give an update how can I only do signature decryption from current version of youtube-dl ?
Regards,