Avoid rewriting unchanged JSON files for labels, milestones, releases… #455
+283
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…, hooks, followers, and following
This change reduces unnecessary writes when backing up metadata that changes
infrequently. The implementation compares existing file content before writing
and skips the write if the content is identical, preserving file timestamps.
Key changes:
- Added json_dump_if_changed() helper that compares content before writing
- Uses atomic writes (temp file + rename) for all metadata files
- NOT applied to issues/pulls (they use incremental_by_files logic)
- Made log messages consistent and past tense ("Saved" instead of "Saving")
- Added informative logging showing skip counts
Fixes josegonzalez#133
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #133 - Avoid rewriting unchanged JSON files
When backing up labels, milestones, releases, hooks, followers, and following, compare existing file content before writing and skip the write if content is identical, preserving file timestamps.
Added
json_dump_if_changed()helper that reads existing files, compares content, and only writes if changed using atomic operations (temp file + rename). If the comparison can't be performed, it conservatively writes anyway.As issues/pulls already use incremental logic and always fetch fresh data they are untouched. No point reading the file when we know it will be written anyway.
Example running backup twice on unchanged repository:
First run:
Second run (nothing changed):
Added pytest test suite (9 tests) covering the new functionality.
Side note
Made log messages consistent and past tense ("Saved" instead of "Saving"/"Writing"). In my case I mount a s3fuse for saving to so there is overhead to read. However, testing against large repositories showed these files are small (largest: 47KB for 9,000+ followers, typical: <10KB), so the read cost is negligible compared unnecessary writes and not preserving timestamp.