Skip DMCA'd repos which return a 451 response #454
Merged
+201
−29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Log a warning and the link to the DMCA notice. Continue backing up other repositories instead of crashing. Closes josegonzalez#163
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #163 - Handle HTTP 451 (DMCA takedown) responses
When encountering a 451 due to DMCA'd repositories, log a warning with the DMCA notice URL and continue with other repositories, instead of terminating with the error.
To avoid multiple messages and repeated wasted requests for each component (eg issues, prs) the tool raises an exception so multiple activities for the repo can be skipped to move onto the next one.
Example of skipping DMCA'd repo
127MU/phigros-html5Exit code is 0.
Added pytest on the 451 logic (written by AI).
Side note
DMCA'd repositories will have an empty directory created (e.g.,
repositories/phigros-html5/issues/). This occurs because directory creation happens before the first API request. It would be a bigger refactor to change the order of when the directory is created. Considered writing an UNAVAILABLE.txt to the directory, but that gets messy as a repo can be un-DMA'd (e.g. youtube-dl) and a subsequent run might work and populate the directories.