Gracefully Handle IA Unavailability at Startup

[brandongalbraith]

requests.exceptions.RetryError: Error retrieving metadata from https://archive.org/metadata/youtube-*REDACTED*, (MaxRetryError("HTTPSConnectionPool(host='archive.org', port=443): Max retries exceeded with url: /metadata/youtube-REDACTED (Caused by ResponseError('too many 503 error responses'))"), 'https://archive.org/metadata/youtube-*REDACTED*')

When tubeup starts up, it makes a query to the Internet Archive API to determine if the item already exists. If the Internet Archive is down for maintenance or otherwise cannot service this metadata request, we should handle this gracefully.

[brandongalbraith]

I am of two minds regarding this bug. If IA is unavailable, should it bail out immediately (as the artifacts retrieved won't be able to be uploaded to IA)? Or should it still attempt to retrieve target artifacts (to upload to IA when it is available again)? This will likely be a component in the tubeup refactor to decouple the retrieval process from the archiving process.

[vxbinaca]

I am of two minds regarding this bug. If IA is unavailable, should it bail out immediately (as the artifacts retrieved won't be able to be uploaded to IA)? Or should it still attempt to retrieve target artifacts (to upload to IA when it is available again)? This will likely be a component in the tubeup refactor to decouple the retrieval process from the archiving process.

Wouldn't it just be if S3 is available? The front end can go down and usually, sans last October, S3 is available.

[brandongalbraith]

Wouldn't it just be if S3 is available? The front end can go down and usually, sans last October, S3 is available.

It's a fair question. We could attempt the metadata query to determine if it already exists, and if that lookup fails, still attempt to upload artifact objects, and then bail out if those S3 operations fail as well (either because IA S3 is down or the objects already exist in S3). Thoughts?

[vxbinaca]

I mean when we did the lookup during the hack downtime, the lookup would fail and Tubeup would bail. Isn't that in effect the same thing as you want?

[mrpapersonic]

Wouldn't it just be if S3 is available? The front end can go down and usually, sans last October, S3 is available.

It's a fair question. We could attempt the metadata query to determine if it already exists, and if that lookup fails, still attempt to upload artifact objects, and then bail out if those S3 operations fail as well (either because IA S3 is down or the objects already exist in S3). Thoughts?

I think when S3 is available it ought to work as I proposed here: #317 (comment) (never really went anywhere, as yt's stupid anti-rip stuff really prevented any testing).

However once any connection to S3 fails, tubeup should enter a "S3 unavailable" mode where it mass downloads all of the videos, then uploading them with the same giant amount of retries as we do now. Though, this may not be really necessary, since S3 seems to be a lot more reliable now than it was circa 2019-2021.

[vxbinaca]

Hack something up and we can add it after testing (somehow). But to my mind this seems unnecessary there's only rare S3 downtimes.

S3 search for the item as a safety, would fail during the hack and this would cause Tubeup to bail and would become a unintentional safety.

[brandongalbraith]

@vxbinaca @mrpapersonic I haven't seen this reoccur, so I'm going to close and punt on this work for now.