Add Sleep Before Metadata Extraction

[cwfitzgerald]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.10.05. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.10.05

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

Description of your issue, suggested solution and other information

I am attempting to download an entire organization's videos for them from vimeo, which requires downloading the original quality video. In order to get the link for the original quality video, youtube-dl has to get the JSON metadata. JSON metadata on vimeo appears to be rate limited to something on the order of once per 5 seconds or so. This means if I were to resume the download of all the videos, when it is going through all the ones that it has already downloaded it will break the rate limit for the JSON metadata, preventing me from getting any original videos. --sleep-interval and --max-sleep-interval work for preventing rate limiting on downloads, but there needs to be a sleep that occurs before metadata extraction, so even when youtube-dl goes through all the videos that it has already gotten, it can be spaced out to comply with rate-limiting.

[dstftw]

No such rate limits have been seen on vimeo so far. Provide concrete verbose logs demonstrating that.

[mtone]

FWIW, I had the same issue with Pluralsight.

1. Download new content with long sleep, no issue for days.
2. Attempted to resume a few partial courses (playlists). Because some files already exist, youtube-dl skips and requests metadata with only a few seconds timeout between requests. A few dozen of those and I was getting HTTP 403 errors and a captcha when logging to web site.
3. Wait a few days, try again. I no longer resume a partial course, simply delete the whole thing and restart to ensure proper sleep. No issues whatsoever since.

Providing concrete logs of this would put my account at risk, but my anecdotal data clearly points to the lack of a longer metadata sleep as the cause.

(PS I was using --embed-subs which consistently fails on some videos, I believe it happens when .srt timestamps are outside the video duration, which happens here and there. Fixed by using --ignore-errors from now to keep going)

[gyaaniguy]

sleep should take BEFORE any connections made. Because in some cases the server responds to 403 during metadata extraction. Getting the issue with the following

youtube-dl --sleep-interval 10 https://www.sonyliv.com/details/episodes/6022124397001/Ep.-92---Viraj-With-A-Plan---Ladies-Special---3-April-2019