Direct cipher signature & n-transform functions to circumvent throttling.

[gatecrasher777]

Change sig.js to run base.js code directly for signature decipher and n-parameter transform. Less code too. :)
Since Google's base.js is browser code it should not pose a security risk.
sig-test.js also changed - some of the tests were no longer useful with this methodology.

Fixes #964

Many projects are struggling with this throttling issue.

[gatecrasher777]

Bedtime. Will add tests to improve code coverage tomorrow.

[lgtm-com]

This pull request fixes 7 alerts when merging e0e90c8 into 40d0c54 - view on LGTM.com

fixed alerts:

• 7 for Incomplete string escaping or encoding

[lgtm-com]

This pull request fixes 7 alerts when merging 36750c1 into 40d0c54 - view on LGTM.com

fixed alerts:

• 7 for Incomplete string escaping or encoding

[mh4ck]

I've tried it, works but doesn't really fix with throttling. The downloads are still slowly, or have i missed a thing?

[gatecrasher777]

Thanks for feedback. Will do some additional testing.

[gatecrasher777]

With the fix:
`
C:\Dev\node-ytdl-core>node dl 2QZAfcC-OTc
100%
complete at 3.342688783612893 mB/s

C:\Dev\node-ytdl-core>node dl MZb2uCFNjwQ
100%
complete at 7.814144400570114 mB/s

C:\Dev\node-ytdl-core>node dl DXdz-Zl1cdk
100%
complete at 6.827378049943889 mB/s

C:\Dev\node-ytdl-core>node dl ipS2Mas3OJw
100%
complete at 6.84887724351718 mB/s

C:\Dev\node-ytdl-core>node dl W2Xb2GFK2yc
100%
complete at 5.903212656407182 mB/s

C:\Dev\node-ytdl-core>node dl 6nC0f1185WI
100%
complete at 7.124622217951667 mB/s

C:\Dev\node-ytdl-core>node dl z3cjksFFKAQ
100%
complete at 6.055402285250079 mB/s

C:\Dev\node-ytdl-core>node dl 0WGVgfjnLqc
100%
complete at 2.7921356909432165 mB/s

C:\Dev\node-ytdl-core>node dl zmbmrw07qBc
100%
complete at 2.1377727646613645 mB/s

C:\Dev\node-ytdl-core>node dl dJv0OvFnVXU
100%
complete at 2.4884090047889034 mB/s

C:\Dev\node-ytdl-core>node dl 24kpAClYmmQ
100%
complete at 6.226202339822745 mB/s

C:\Dev\node-ytdl-core>node dl Zt5qJC1xQ8A
100%
complete at 4.662531029558929 mB/s
`
No slowdowns at all. 2-8 mB/s on 100mbps fibre connection.

Now, without the fix:
`
C:\Dev\nyc>node dl 2QZAfcC-OTc
100%
complete at 0.07246436337297874 mB/s

C:\Dev\nyc>node dl MZb2uCFNjwQ
16%
`
First two files are already throttled. Can't even wait for the 2nd one. It is going to take a long, long time to finish. And life is too short.

Here is my dl.js code:

const fs = require('fs');
const ytdl = require ('./lib/index.js');

let id = process.argv[2];
let start;
let end;
let dl = ytdl(id,{quality: 'highestvideo'});
dl.on(
    'progress',
    (a,b,c) => {
        if (!start) {
            start = Date.now();
        }
        process.stdout.write( `${Math.floor(100*b/c)}% \r`);
        if (b == c) {
            end = Date.now();
            console.log(`\n complete at ${c*1000/(end-start)/1024/1024} mB/s`);
        }
    }
);
dl.pipe(fs.createWriteStream('test.mp4'));

[gatecrasher777]

2 more finished.
`
C:\Dev\nyc>node dl MZb2uCFNjwQ
100%
complete at 0.07245288566336598 mB/s

C:\Dev\nyc>node dl DXdz-Zl1cdk
100%
complete at 0.07308124182656603 mB/s
`

The 4th file is throttled too, and taking a long time to finish.
Being a statistician, I'm 99% confident that this PR fixes the throttling.
But anyone else can also test. Just run the dl.js from the main project folder using the lib/sig.js in this PR vs the current lib/sig.js.

[TimeForANinja]

Since Google's base.js is browser code it should not pose a security risk.

🤔 bold claim - in the end we execute code straight from the internet and hope google is playing nice

[gatecrasher777]

Using vm module instead. No access to node.js environment. Better?
Its time for a Ninja. :)

[drinkspiller]

For the dummies: Do you think there will be an update to solve the problem?

Use the fix in PR #1022 now by following these instructions

[LuanRT]

FWIW I found I could could get full access to the all node-js modules from within vm running an empty context... just by using a this.constructor inside the sandbox. So require('fs') and delete all is quite possible. That's why vm2 is preferred. No known way to break out of the sandbox.

I would concede that a reliable means of n-transform without running any google code directly is first prize. But it might just be inherently unreliable by design, through the constant introduction of novel transformations. That seems to have been the experience so far of those trying to emulate it, although I can't be certain about that. Perhaps it will stabilize.

Any progress on this? If it helps, I have developed a script that parses the JavaScript code and emulates the necessary transformations on the n token— it was not based on VLC's approach but I'm constantly debugging it. So far I have tested it with 20+ different players and only ran into minor parsing problems on one of them, however they were easily fixed. I guess one could port it to ytdl-core if executing the n transform function directly is a problem.
(do note that my library is also written in JavaScript, so it should be pretty simple to port).

[Michal-Szczepaniak]

Works for me but didn't test thoroughly

[peepo5]

👍

[Michal-Szczepaniak]

Tested by 3 people, not only it improves performance of getting url by a lot, it fixed throttling completely, this patch is amazing can we merge it asap?

[gazugafan]

Working for me so far. Going to get tested more thoroughly tonight.

[KuraimiKun]

how can i do that

[TimeForANinja]

thanks again for the pr - should be released on npm in a second
i opened #1043 as a reminder / discussion to remove that vm-dependency
for now this looks like the best fix