Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Download does not automatically recover when the connection times out #4240
Comments
|
Can confirm the issue. |
|
I was just searching to see if this issue already existed before reporting it. I can confirm it happens for me as well This is exactly the sort of situation retrying is supposed to handle. However, it appears that http.py isn't expecting SSLError to be recoverable, when often they are. Not all SSLError will be cryptographic problems, but it appears the only nod to this is that the catch of socket.error (SSLError's superclass) retries ECONNRESET. I would recommend handling SSLError separately, since only the subtype CertificateError is unrecoverable (i.e. not network-based). Rather than handling SSLError as a socket.error, which assumes that the only recoverable socket error is ECONNRESET, the fact that many SSLError are transient network problems should allow a more nuanced handling of of SSLError. Specifically, the OpenSSL docs state explicitly that the following errno are retriable ("The TLS/SSL I/O function should be called again later."): SSL_ERROR_WANT_READ, SSL_ERROR_WANT_WRITE, SSL_ERROR_WANT_CONNECT, SSL_ERROR_WANT_ACCEPT, SSL_ERROR_WANT_X509_LOOKUP, SSL_ERROR_WANT_ASYNC SSL_ERROR_SYSCALL is possibly retriable, but the Python stdlib provides no way to inspect the error more deeply. Still, this should probably be retried by default as long as retries != infinite, since if it's not retriable (say EMFILE at the OS level), it will fail quickly and run through the retries with no harm done. SSL_ERROR_SSL is the only one that is probably not retriable, since it would likely be a crypto problem due to mismatched certs or a misconfigured server. Still, the error has enough detail that it is (I believe) broken out into a couple different exception types in Python, one of which being CertificateError. |
|
youtube-dl obviously has a retry mechanism. It just doesn't catch everything |
|
For now, as a workaround, you can do something like the following in the terminal. The loop attempts to run the command in the getvids function up to 10 times or until it finishes successfully. |
|
jeffymahoney how to run this loop? I got SyntaxError: invalid syntax |
|
@daewizal That's not Python that he's using. If you're on windows, you can create a batch script to "retry". For example, put the following in a file called youtube-dl-retry.cmd This will keep on running the youtube-dl command until it succeeds. |
|
@IGTHORN thank you! |
|
@IGTHORN : could you also please provide shell script for same |
|
@nipunsadvilkar The script posted by jeffymahoney is a shell script. You can create a file called youtube-dl-retry.sh and paste in his script. Just change the line with youtube-dl so that it contains the video you want to download. Jeff's script will only retry 10 times, so if you want it to retry more than that, change the for loop line to: |
|
@IGTHORN , @jeffymahoney : Thank you guys! |
I set youtube-dl to download a youtube playlist overnight. During the night the connection times out and youtube-dl sits idle for a long time until eventually the connection times out.
This might be due to some minor packet loss on my line:
Ping statistics for 8.8.8.8: Packets: Sent = 200, Received = 193, Lost = 7 (3% loss),I ran it last night with --verbose and got the following output:
I captured another instance, verbose output trimmed to the stack trace and single video: