Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Feature Request: improve youtube-dl for connections with packet-loss #11161
Comments
|
Use |
|
i've seen this here when searching for similar issues: |
|
also, if |
|
Caching won't work in general since data may be bound to concrete session, IP, expire after some period of time and so on. |
|
so can we maybe reopen this and see if someone else has an interesting suggestion or idea, @dstftw ? |
From this sentence, I guess what you need is a new SSL feature called client-side SSL session management, which will be included in CPython 3.6. See http://bugs.python.org/issue19500 |
Before submitting an issue make sure you have:
What is the purpose of your issue?
Description of your issue, suggested solution and other information
Sometimes when there is very bad weather, my 3G link has a lot of packet loss - up to 50%.
This is not a huge problem per se - for example regular HTTP stuff etc works well because of retransmission of lost packets. HOWEVER since these days everything is httpS, under these conditions the initial connection handshake to make a new TLS connection is very unlikely to succeed. Even less unlikely is that all of the up-to-five HTTPS requests made by youtube-dl to gather information about a youtube video succeed.
So it takes up to 10 minutes to have the good luck that all 5 downloads (youtube webpage, information webpage, player webpage, metadata webpage, etc) succeed in a row so that youtube-dl can start downloading. ONCE IT DOWNLOADS, EVERYTHING IS FINE (almost). because really the only problematic thing is the handshake. if the connection once is open, retransmission of lost packets will make sure that even under these bad conditions i can download with over 1 MBit/s. What then eventually happens, is that the connection gets interrupted completely for a moment - and youtube-dl hangs until it finally hits its read-timeout (which seems to be rather huge - in the order of several minutes).
Then the Fun begins again (after hitting CTRL-C and restarting the job).
Even worse when downloading several videos from a playlist, because youtube-dl will always start with the first video from the list, download all 5 informational pages, just to see then that the file is already there, then to the next... so if you're stuck on the 5th video, there are zero percent changes of getting so far that the download can resume.
My proposal now would be that, until a job is 100% complete (so if downloading a playlist, until all videos are downloaded) that youtube-dl caches all of the information it downloads previous to downloading the actual videos, and finally when everything is complete all the files in the cache can be removed.
so in the case of downloading a playlist that was previously interrupted at the 5th video, youtube-dl would read the cached playlist file, then read all the cached information about the first video, see that it is already completely, then read all of the cached information of the second etc until the fifth, and then here would actually for the first time make a connection to the internet.