Skip to content

Commit

Permalink
fix: get more reliable format data from player_response
Browse files Browse the repository at this point in the history 
Squashed commit of the following:

commit 7518c31545288765d507657d7526201cd88fcfbf
Author: fent <933490+fent@users.noreply.github.com>
Date:   Sun Nov 24 15:30:53 2019 -0500

    fix: add back formats meta for playlist formats

commit 327482f30ee4bf4bb4dac8641cc2dc83f600caf6
Author: fent <933490+fent@users.noreply.github.com>
Date:   Sun Nov 24 03:37:38 2019 -0500

    refactor: clean up dead code

commit e8a48ab3cd3be2559736d167193437ba90eac490
Author: fent <933490+fent@users.noreply.github.com>
Date:   Wed Nov 20 20:58:27 2019 -0500

    test: update more tests

commit dbf2d0bfbac7aef7c90f3f937935323e81a96691
Author: fent <933490+fent@users.noreply.github.com>
Date:   Sun Oct 27 23:42:25 2019 -0400

    update util tests

commit f44dcb690a2478c1288eeb5a75f58761d967eee5
Author: fent <933490+fent@users.noreply.github.com>
Date:   Sun Oct 27 23:37:06 2019 -0400

    Fixes

commit b56ddd1ba09b3ec99f67c09dc71893f6c20a68b1
Merge: 2402167 0c1e18f
Author: fent <933490+fent@users.noreply.github.com>
Date:   Wed Oct 23 10:16:36 2019 -0400

    Merge branch 'master' of github.com:RocoGreen/node-ytdl-core into RocoGreen-master

commit 0c1e18f
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 20 08:00:16 2019 +0200

    Update info.js

commit 2a864e4
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 20 07:54:58 2019 +0200

    Removing some spaces again

commit 16952bc
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 20 07:53:24 2019 +0200

    Restoring index.js

commit f73de3d
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Fri Oct 18 18:37:00 2019 +0200

    Remove some spaces

commit bdd7877
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Fri Oct 18 18:31:26 2019 +0200

    Fix sig parameter

commit c93b372
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Fri Oct 18 18:30:37 2019 +0200

    Change mimeType to type and delete s

commit 5ae6c45
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Fri Oct 18 18:27:53 2019 +0200

    Fix format.url

    YouTube changed their api and i fixed that !
    Some videos is touched by that and others not

commit a146b25
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Thu Oct 17 17:50:55 2019 +0200

    Delete index.js

    I will do a pull request to miniget for ratelimit it will be better

commit fed8354
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 13 13:35:30 2019 +0200

    Bug fix : ContentLength is not defined

    I dunno but if format.contentLength is undefined, sometimes contentLength is undefined so i replaced it by format.contentLength and progress event is working perfectly, i hope this is the last bug !

commit f1ee0d1
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 13 11:40:11 2019 +0200

    Fix progress event if format isDashMPD and emit full response

commit d9b0752
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sun Oct 13 09:33:14 2019 +0200

    Addind chunk declaration

commit cb9f681
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sat Oct 12 18:55:27 2019 +0200

    Fix progress event

commit cf12a13
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sat Oct 12 18:15:20 2019 +0200

    Fix a problem with content length

commit 6cd9084
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sat Oct 12 17:03:45 2019 +0200

    Add files via upload

commit d9c798c
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sat Oct 12 16:58:09 2019 +0200

    Change typings for new formats info

commit 5313ddd
Author: RocoGreen <52676010+RocoGreen@users.noreply.github.com>
Date:   Sat Oct 12 16:57:17 2019 +0200

    Changing rate limit and using formats from player_response
  • Loading branch information
@fent
fent committed Nov 25, 2019
1 parent e590a50 commit eb1491f
Show file tree
Hide file tree
Showing 33 changed files with 23,065 additions and 23,857 deletions.
804 changes: 292 additions & 512 deletions lib/formats.js
View file

Large diffs are not rendered by default.

65 changes: 15 additions & 50 deletions lib/info.js
View file
Expand Up @@ -5,19 +5,13 @@ const request = require('miniget');
const util = require('./util');
const extras = require('./info-extras');
const sig = require('./sig');
const FORMATS = require('./formats');


const VIDEO_URL = 'https://www.youtube.com/watch?v=';
const EMBED_URL = 'https://www.youtube.com/embed/';
const VIDEO_EURL = 'https://youtube.googleapis.com/v/';
const INFO_HOST = 'www.youtube.com';
const INFO_PATH = '/get_video_info';
const KEYS_TO_SPLIT = [
'fmt_list',
'fexp',
'watermark'
];


/**
Expand Down Expand Up @@ -98,15 +92,14 @@ exports.getBasicInfo = (id, options, callback) => {
*/
const parseFormats = (info) => {
let formats = [];
if (info.url_encoded_fmt_stream_map) {
formats = formats.concat(info.url_encoded_fmt_stream_map.split(','));
}
if (info.adaptive_fmts) {
formats = formats.concat(info.adaptive_fmts.split(','));
if (info.player_response.streamingData) {
if (info.player_response.streamingData.formats) {
formats = formats.concat(info.player_response.streamingData.formats);
}
if (info.player_response.streamingData.adaptiveFormats) {
formats = formats.concat(info.player_response.streamingData.adaptiveFormats);
}
}
formats = formats.map((format) => querystring.parse(format));
delete info.url_encoded_fmt_stream_map;
delete info.adaptive_fmts;
return formats;
};

Expand Down Expand Up @@ -144,41 +137,25 @@ const gotConfig = (id, options, additional, config, fromEmbed, callback) => {
request(url, options.requestOptions, (err, res, body) => {
if (err) return callback(err);
let info = querystring.parse(body);
const player_response = config.args.player_response || info.player_response;

if (!info.url_encoded_fmt_stream_map && !info.adaptive_fmts &&
!info.config && (config.args.fmt_list ||
config.args.url_encoded_fmt_stream_map || config.args.adaptive_fmts)) {
info = config.args;
info.no_embed_allowed = true;
} else if (info.status === 'fail') {
if (info.status === 'fail') {
return callback(
Error(`Code ${info.errorcode}: ${util.stripHTML(info.reason)}`));
}

const player_response = config.args.player_response || info.player_response;
try {
} else try {
info.player_response = JSON.parse(player_response);
} catch (err) {
return callback(
Error('Error parsing `player_response`: ' + err.message));
}

let playability = info.player_response.playabilityStatus;
if (playability && playability.status === 'UNPLAYABLE') {
return callback(Error(playability.reason));
}

// Split some keys by commas.
KEYS_TO_SPLIT.forEach((key) => {
if (!info[key]) return;
info[key] = info[key]
.split(',')
.filter((v) => v !== '');
});

info.fmt_list = info.fmt_list ?
info.fmt_list.map((format) => format.split('/')) : [];

info.formats = parseFormats(info);

// Add additional properties to info.
Object.assign(info, additional, {
video_id: id,
Expand Down Expand Up @@ -236,16 +213,7 @@ exports.getFullInfo = (id, options, callback) => {
if (results[0]) { mergeFormats(info, results[0]); }
if (results[1]) { mergeFormats(info, results[1]); }

if (options.debug) {
info.formats.forEach((format) => {
const itag = format.itag;
if (!FORMATS[itag]) {
console.warn(`No format metadata for itag ${itag} found`);
}
});
}

info.formats.forEach(util.addFormatMeta);
info.formats = info.formats.map(util.addFormatMeta);
info.formats.sort(util.sortFormats);
info.full = true;
callback(null, info);
Expand All @@ -266,12 +234,9 @@ exports.getFullInfo = (id, options, callback) => {
*/
const mergeFormats = (info, formatsMap) => {
info.formats.forEach((f) => {
if (!formatsMap[f.itag]) {
formatsMap[f.itag] = f;
}
formatsMap[f.itag] = f;
});
info.formats = [];
for (let itag in formatsMap) { info.formats.push(formatsMap[itag]); }
info.formats = Object.values(formatsMap);
};


Expand Down
9 changes: 7 additions & 2 deletions lib/sig.js
View file
@@ -1,5 +1,6 @@
const url = require('url');
const request = require('miniget');
const url = require('url');
const request = require('miniget');
const querystring = require('querystring');


// A shared cache to keep track of html5player.js tokens.
Expand Down Expand Up @@ -265,6 +266,10 @@ exports.setDownloadURL = (format, sig, debug) => {
*/
exports.decipherFormats = (formats, tokens, debug) => {
formats.forEach((format) => {
if (format.cipher) {
Object.assign(format, querystring.parse(format.cipher));
delete format.cipher;
}
const sig = tokens && format.s ? exports.decipher(tokens, format.s) : null;
exports.setDownloadURL(format, sig, debug);
});
Expand Down
35 changes: 13 additions & 22 deletions lib/util.js
View file
Expand Up @@ -27,20 +27,12 @@ const videoEncodingRanks = {
* @param {Object} b
*/
exports.sortFormats = (a, b) => {
const ares = a.resolution ? parseInt(a.resolution.slice(0, -1), 10) : 0;
const bres = b.resolution ? parseInt(b.resolution.slice(0, -1), 10) : 0;
const ares = a.qualityLabel ? parseInt(a.qualityLabel.slice(0, -1)) : 0;
const bres = b.qualityLabel ? parseInt(b.qualityLabel.slice(0, -1)) : 0;
const afeats = ~~!!ares * 2 + ~~!!a.audioBitrate;
const bfeats = ~~!!bres * 2 + ~~!!b.audioBitrate;

const getBitrate = (c) => {
if (c.bitrate) {
let s = c.bitrate.split('-');
return parseFloat(s[s.length - 1], 10);
} else {
return 0;
}
};

const getBitrate = (c) => parseInt(c.bitrate) || 0;
const audioScore = (c) => {
const abitrate = c.audioBitrate || 0;
const aenc = audioEncodingRanks[c.audioEncoding] || 0;
Expand Down Expand Up @@ -94,10 +86,7 @@ exports.chooseFormat = (formats, options) => {

let format;
const quality = options.quality || 'highest';
const getBitrate = (f) => {
let s = f.bitrate.split('-');
return parseFloat(s[s.length - 1], 10);
};
const getBitrate = (f) => parseInt(f.bitrate);
switch (quality) {
case 'highest':
format = formats[0];
Expand All @@ -119,7 +108,7 @@ exports.chooseFormat = (formats, options) => {
break;

case 'lowestaudio':
formats = exports.filterFormats(formats, 'audio')
formats = exports.filterFormats(formats, 'audio');
format = null;
for (let f of formats) {
if (!format
Expand All @@ -141,7 +130,7 @@ exports.chooseFormat = (formats, options) => {
break;

case 'lowestvideo':
formats = exports.filterFormats(formats, 'video')
formats = exports.filterFormats(formats, 'video');
format = null;
for (let f of formats) {
if (!format
Expand Down Expand Up @@ -331,16 +320,18 @@ exports.validateURL = (string) => {

/**
* @param {Object} format
* @return {Object}
*/
exports.addFormatMeta = (format) => {
const meta = FORMATS[format.itag];
for (let key in meta) {
format[key] = meta[key];
}

format = Object.assign({}, FORMATS[format.itag], format);
format.container = format.mimeType ?
format.mimeType.split(';')[0].split('/')[1] : null;
format.codecs = format.mimeType ?
exports.between(format.mimeType, 'codecs="', '"') : null;
format.live = /\/source\/yt_live_broadcast\//.test(format.url);
format.isHLS = /\/manifest\/hls_(variant|playlist)\//.test(format.url);
format.isDashMPD = /\/manifest\/dash\//.test(format.url);
return format;
};


Expand Down
4 changes: 2 additions & 2 deletions test/files/refresh.js
View file
Expand Up @@ -111,12 +111,12 @@ const videos = [
{
page: 'get_video_info',
saveAs: 'no-formats',
fn: body => body.replace(/fmt/g, '')
fn: body => body.replace(/streamingData/g, 'no')
},
{
page: 'watch.html',
saveAs: 'no-formats',
fn: body => body.replace(/fmt/g, '')
fn: body => body.replace(/streamingData/g, 'no')
}
]
},
Expand Down
2 changes: 1 addition & 1 deletion test/files/videos/_HSylqgVYQI-regular/get_video_info
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion test/files/videos/_HSylqgVYQI-regular/get_video_info-unknown-format
View file

Large diffs are not rendered by default.

0 comments on commit eb1491f

Please sign in to comment.