Obtaining compressed/uncompressed bytes transferred from ytdl.getInfo requests #899
Replies: 4 comments 2 replies
-
Sorry to be impatient. I played around some and got a fairly nice solution: In my code: let transferred;
let uncompressed = 0;
let compressed = 0;
let zip = null;
ytdl.getInfo( url, {
reqCallback : (req) => {
req.on('data',(chunk)=>{
uncompressed += chunk.length;
});
req.on('end',()=>{
if (zip != null) compressed += zip.bytesWritten;
zip = null;
});
},
requestOptions: {
headers: {
'cookie': this.cookie,
'accept-encoding': 'gzip, deflate, br'
},
acceptEncoding: {
gzip: () => { zip = zlib.createGunzip(); return zip; },
deflate: () => { zip = zlib.createDeflate(); return zip; },
br: () => { zip = zlib.createBrotliDecompress(); return zip; }
}
}
}).then(info => {
(compressed) ? transferred += compressed : transferred += uncompressed;
...
console.log('uncompressed: '+uncompressed);
console.log('compressed: '+compressed);
}, err => {
(compressed) ? transferred += compressed : transferred += uncompressed;
...
console.log('uncompressed: '+uncompressed);
console.log('compressed: '+compressed);
}); In the modified info.js, each miniget request (about 6 of them) would expose the request stream. For example: let req = miniget(jsonUrl, reqOptions);
options.reqCallback(req);
let body = await req.text(); instead of let body = await miniget(jsonUrl, reqOptions).text(); Typical output from the above looks like this:
So essentially, I'm suggesting a reqCallback option. Exposing the miniget requests like this might have other applications no-one has thought of yet. |
Beta Was this translation helpful? Give feedback.
-
That sounds very interesting and should be implemented by default (without a 'reqCallback') if it is like you say.
Is it only the info request or even videos can be received compressed? |
Beta Was this translation helpful? Give feedback.
-
all miniget events are forwarded to the ytdl stream, that includes the |
Beta Was this translation helpful? Give feedback.
-
If we can get the request stream on a getInfo request it is also helpful to stop/handle broken proxy connections. In my case it happens sometimes that the connection to the proxy hangs up and the request never finish/error or run into a timeout. If we get the request stream we can manually stop the connection and try again with another proxy. |
Beta Was this translation helpful? Give feedback.
-
I have a github project that uses ytdl-core when stream urls are ciphered. I would like it to return the bytes transferred. Before I fall back onto ytdl-core I use something very vanilla that can handle lots of simultaneous requests like this:
Similarly, when urls are missing/ciphered, and I fall back onto ytdl-core I can still implement br compression:
but extracting the actual bytes transferred does not seem reliable. I can get the bytes transferred on the last request ytdl-core makes with
and then querying zip.bytesWritten but its not a useful number if several requests are pipelined in ytdl-core prior to the last request.
I know the bytes transferred to get video info is not important in the typical ytdl-core use case, but to be able to reliably monitor overall bandwidth in my project would be a big plus.
I also don't want to use a modified ytdl-core in my project because ytdl-core updates are both often and crucial for reliable download urls.
I envision/hope for something simple and non-invasive like this:
Perhaps if I work on a pull request that achieves this and doesn't break anything, it might be considered?
Thanks for a great project.
Beta Was this translation helpful? Give feedback.
All reactions