Added commit mode #139
Merged
Added commit mode #139
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aandergr
requested changes
Nov 6, 2018
| @@ -276,6 +276,10 @@ def main(): | |||
| help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a ' | |||
| 'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry ' | |||
| 'infinitely.') | |||
| g_how.add_argument('--commit-mode', action='store_true', | |||
There was a problem hiding this comment.
Please also add the flag to docs/cli-option.rst
instaloader/__main__.py
Outdated
| @@ -309,6 +313,9 @@ def main(): | |||
| raise SystemExit("--no-captions and --post-metadata-txt or --storyitem-metadata-txt given; " | |||
| "That contradicts.") | |||
|
|
|||
| if args.commit_mode and args.no_metadata_json: | |||
| raise SystemExit('Commit mode requires JSON metadata to be saved.') | |||
There was a problem hiding this comment.
Suggested change
| raise SystemExit('Commit mode requires JSON metadata to be saved.') | |
| raise SystemExit('--commit-mode cannot be used together with --no-metadata-json.') |
Naming the exact flags might it make easier for the user to understand and solve the problem (however, this is just a suggestion).
instaloader/exceptions.py
Outdated
| @@ -51,3 +51,7 @@ class ConnectionException(InstaloaderException): | |||
|
|
|||
| class TooManyRequestsException(ConnectionException): | |||
| pass | |||
|
|
|||
|
|
|||
| class NeedMetadataException(InstaloaderException): | |||
There was a problem hiding this comment.
I think there is no need to introduce a new exception. Normally we would just use InvalidArgumentException.
instaloader/instaloader.py
Outdated
| @@ -128,6 +130,12 @@ def __init__(self, | |||
| else post_metadata_txt_pattern | |||
| self.storyitem_metadata_txt_pattern = '' if storyitem_metadata_txt_pattern is None \ | |||
| else storyitem_metadata_txt_pattern | |||
| self.commit_mode = commit_mode | |||
| if self.commit_mode and not self.save_metadata: | |||
| raise NeedMetadataException("Commit mode requires JSON metadata to be saved.") | |||
There was a problem hiding this comment.
Suggested change
| raise NeedMetadataException("Commit mode requires JSON metadata to be saved.") | |
| raise InvalidArgumentException("Commit mode requires JSON metadata to be saved.") |
I think there is no need to introduce a new exception for this
added 3 commits
December 9, 2018 02:12
The commit mode ensures pictures are not corrupted when Instaloader is unexpectedly interrupted. In the case that the last picture is corrupted because of an interruption, Instaloader will redownload the picture. Since the metadata is the last object saved to disk, we can consider a post as "committed" if its json metadata file exists and is not malformed. Instaloader should download any posts which are not committed. Downside is commit mode requires metadata to be saved.
This is in addition to the other commit logic.
|
I have rebased to 4.1.1 and incorporated the changes you suggested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I found that when Instaloader is interrupted (ctr-c, computer crash, etc) in the middle of a run, it may not correctly save the last picture. This is a commit mode that I scrapped up to make sure Instaloader will redownload the corrupted picture in its next run.
The commit mode ensures pictures are not corrupted when Instaloader is
unexpectedly interrupted. In the case that the last picture is corrupted
because of an interruption, Instaloader will redownload the picture
in the next run. Since the metadata is the last object saved to disk,
we can consider a post as "committed" if its json metadata file exists and
is not malformed. Instaloader should download any posts which are not
committed. Downside is commit mode requires metadata to be saved.