refactor(general): move index cleanup out to refreshAttempt #3603
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
julio-lopez
force-pushed
the
draft/no-index-cleanup-on-load
branch
from
c406533
to
065cf05
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #3603 +/- ##
==========================================
+ Coverage 75.86% 77.14% +1.27%
==========================================
Files 470 470
Lines 37301 28440 -8861
==========================================
- Hits 28299 21940 -6359
+ Misses 7071 4570 -2501
+ Partials 1931 1930 -1 ☔ View full report in Codecov by Sentry. |
julio-lopez
force-pushed
the
draft/no-index-cleanup-on-load
branch
from
065cf05
to
b7a65ce
Compare
julio-lopez
changed the title
julio-lopez
changed the title
julio-lopez
commented
Feb 7, 2024
Comment on lines
+474
to
+478
| e.maybeGenerateNextRangeCheckpointAsync(ctx, cs, p) | ||
| e.maybeStartCleanupAsync(ctx, cs, p) | ||
| e.maybeOptimizeRangeCheckpointsAsync(ctx, cs) |
There was a problem hiding this comment.
This is code movement only. These were at the end of refreshAttemptLocked so putting them here has no functional changes, they just happened to be called right after refreshAttemptLocked returns, instead of the last calls steps refreshAttemptLocked.
Comment on lines
-698
to
-700
| e.maybeGenerateNextRangeCheckpointAsync(ctx, cs, p) | ||
| e.maybeStartCleanupAsync(ctx, cs, p) | ||
| e.maybeOptimizeRangeCheckpointsAsync(ctx, cs) |
There was a problem hiding this comment.
Moved to the only caller, see comment above.
| @@ -948,7 +960,7 @@ func (e *Manager) getIndexesFromEpochInternal(ctx context.Context, cs CurrentSna | |||
| uncompactedBlobs = ue | |||
| } | |||
|
|
|||
| if epochSettled { | |||
| if epochSettled && e.allowWritesOnLoad() { | |||
There was a problem hiding this comment.
This is a functional change. It avoids advancing epoch when writes are disallowed.
Before: it attempted to advance the epoch (by writing a blob to the storage), even on read-only storage.
After: prevents advancing epochs when writes are disallowed on index loads, for example, when using read-only storage.
| @@ -678,12 +704,6 @@ func (e *Manager) refreshAttemptLocked(ctx context.Context) error { | |||
| len(ues[cs.WriteEpoch+1]), | |||
| cs.ValidUntil.Format(time.RFC3339Nano)) | |||
|
|
|||
| if !e.st.IsReadOnly() && shouldAdvance(cs.UncompactedEpochSets[cs.WriteEpoch], p.MinEpochDuration, p.EpochAdvanceOnCountThreshold, p.EpochAdvanceOnTotalSizeBytesThreshold) { | |||
There was a problem hiding this comment.
Moved to caller.
This was done "almost" at the end, but before returning an error in some cases.
It should not make a difference in practice though.
| return e.maybeCompactAndCleanupLocked(ctx, p) | ||
| } | ||
|
|
||
| func (e *Manager) maybeCompactAndCleanupLocked(ctx context.Context, p *Parameters) error { |
There was a problem hiding this comment.
The goal of this function is to group the epoch cleanup tasks that are performed during index loads, so it is simpler to manage.
The long term objective is to remove this code path from index loading altogether.
|
Before: After: |
Introduces an `allowWritesOnLoadHelper` to check whether or not writes can be performed when loading the indexes. Currently this is only a function of whether the storage is in read-only mode. In the near future, an explicit flag will be added to control this behavior.
Functional change: prevents compacting single epochs when writes are disallowed, that is when using read-only storage. Currently, the epoch manager will attempt to perform single-epoch compactions for all eligible epochs, even on read-only storage. Ref: kopia#3225
redgoat650
approved these changes
Feb 13, 2024
This was referenced Feb 13, 2024
julio-lopez
force-pushed
the
draft/no-index-cleanup-on-load
branch
from
7b28438
to
84da027
Compare
6 tasks
|
@ashmrtn FYI |
julio-lopez
added a commit
to kastenhq/kopia
that referenced
this pull request
Mar 12, 2024
Extracted from kopia#3651. Thanks to @plar and @redgoat650 for the suggestions. Ref: - kopia#3603 - kopia#3645 - kopia#3638 - kopia#3639
julio-lopez
added a commit
that referenced
this pull request
Mar 12, 2024
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 18, 2024
Refactor: move index compaction and cleanup out of refreshAttemptLocked Introduces an `allowWritesOnLoadHelper` to check whether or not writes can be performed when loading the indexes. Currently this is only a function of whether the storage is in read-only mode. In the near future, an explicit flag will be added to control this behavior. Fix epoch manager: avoid single-epoch compaction when writes are disallowed. Functional change: prevents compacting single epochs when writes are disallowed, that is when using read-only storage. Currently, the epoch manager will attempt to perform single-epoch compactions for all eligible epochs, even on read-only storage. Ref: - kopia#3224 - kopia#3225 - kopia#3638 - kopia#3639
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 22, 2024
Refactor: move index compaction and cleanup out of refreshAttemptLocked Introduces an `allowWritesOnLoadHelper` to check whether or not writes can be performed when loading the indexes. Currently this is only a function of whether the storage is in read-only mode. In the near future, an explicit flag will be added to control this behavior. Fix epoch manager: avoid single-epoch compaction when writes are disallowed. Functional change: prevents compacting single epochs when writes are disallowed, that is when using read-only storage. Currently, the epoch manager will attempt to perform single-epoch compactions for all eligible epochs, even on read-only storage. Ref: - kopia#3224 - kopia#3225 - kopia#3638 - kopia#3639
julio-lopez
added a commit
that referenced
this pull request
Mar 27, 2024
…3651) Perform index epoch compaction and cleanup during repository maintenance * refactor: rename maintenance task for deleting superseded indexes * maintenance task to cleanup epoch markers * maintenance task to advance write index epoch * maintenance task to compact epoch ranges * quick maintenance task to compact a single (index) epoch * full maintenance task to compact a single (index) epoch Ref: - #3638 - #3639 Followup to: - #3603 - #3645 Related helpers and changes: - #3721 - #3735 - #3709 - #3728 - #3727 - #3726
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 29, 2024
…opia#3651) Perform index epoch compaction and cleanup during repository maintenance * refactor: rename maintenance task for deleting superseded indexes * maintenance task to cleanup epoch markers * maintenance task to advance write index epoch * maintenance task to compact epoch ranges * quick maintenance task to compact a single (index) epoch * full maintenance task to compact a single (index) epoch Ref: - kopia#3638 - kopia#3639 Followup to: - kopia#3603 - kopia#3645 Related helpers and changes: - kopia#3721 - kopia#3735 - kopia#3709 - kopia#3728 - kopia#3727 - kopia#3726
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactoring related to #3638 to simplify later PRs as much as possible.
These changes are mostly code movement except for a functional change noted below.
Changes
Move index compaction and cleanup out of
refreshAttemptLocked, and perform it inrefreshLockedafterrefreshAttempLocksucceeds. See inline comments.Refactor: Introduce an
allowWritesOnLoadHelperto check whether or not writes can be performed when loading the indexes. Currently this is only a function of whether the storage is in read-only mode. In the near future, an explicit flag will be added to control this behavior.Functional change: Avoid single epoch compaction when writes are disallowed, that is when using read-only storage. Currently, the epoch manager will attempt to perform single epoch compactions, when it is time to do so, even on read-only storage.
Ref: