handle throttling automatically

[tzarebczan]

Today we need to shutdown servers, wait, fix yt sync records so they are picked up by other channels, etc.

[tzarebczan]

23 recently got stuck at eleting original file /tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.f140 (pass -k to keep) ERRO[694297] 2604:2dc0:101:200::e9f set to throttled DEBU[694297] slack: ytsync-status: :sos: 2604:2dc0:101:200::e9f set to throttled INFO[694297] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694297] /tmp/ytsync318973938/QTWjcTZI5ko DEBU[694297] QTWjcTZI5ko deleted from disk for 'due to error' (/tmp/ytsync318973938/QTWjcTZI5ko/jennifer-love-hewitt-stories.mp4) INFO[694312] Running command youtube-dl --no-progress -o/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories --merge-output-format mp4 --rm-cache-dir --postprocessor-args -movflags faststart --abort-on-unavailable-fragment --fragment-retries 1 --cookies cookies.txt --max-filesize 2048M --match-filter duration <= 7200 --source-address 2604:2dc0:101:200::e9f https://www.youtube.com/watch?v=NE6xHQGFJSc -fbestvideo[ext=mp4][height<=1080]+bestaudio[ext!=webm] --user-agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36 DEBU[694315] Removing cache dir /home/lbry/.cache/youtube-dl .. [youtube] NE6xHQGFJSc: Downloading webpage [youtube] NE6xHQGFJSc: Downloading API JSON [youtube] NE6xHQGFJSc: Downloading MPD manifest [youtube] NE6xHQGFJSc: Downloading API JSON [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 [download] Download completed [download] Destination: /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 [download] Download completed [ffmpeg] Merging formats into "/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4" Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f135 (pass -k to keep) Deleting original file /tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.f140 (pass -k to keep) INFO[694315] Command finished with error: WARNING: Unable to download webpage: HTTP Error 429: Too Many Requests INFO[694315] /tmp/ytsync318973938/NE6xHQGFJSc DEBU[694315] NE6xHQGFJSc deleted from disk for 'due to error' (/tmp/ytsync318973938/NE6xHQGFJSc/jennifer-love-hewitt-stories.mp4)

it detects it - it should stop syncing so that another server can pick it up and then wait a day.