refactor(general): allow disabling writes on index loads #3645
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #3645 +/- ##
==========================================
+ Coverage 75.86% 77.15% +1.29%
==========================================
Files 470 470
Lines 37301 28446 -8855
==========================================
- Hits 28299 21948 -6351
+ Misses 7071 4570 -2501
+ Partials 1931 1928 -3 ☔ View full report in Codecov by Sentry. |
julio-lopez
changed the base branch from
master
to
draft/no-index-cleanup-on-load
This was referenced Feb 15, 2024
|
@ashmrtn FYI |
julio-lopez
commented
Feb 16, 2024
| @@ -1015,16 +1019,28 @@ func rangeCheckpointBlobPrefix(epoch1, epoch2 int) blob.ID { | |||
| return blob.ID(fmt.Sprintf("%v%v_%v_", RangeCheckpointIndexBlobPrefix, epoch1, epoch2)) | |||
| } | |||
|
|
|||
| func allowWritesOnIndexLoad() bool { | |||
There was a problem hiding this comment.
Note: this env variable mechanism to control the behavior is temporary.
We will remove it once the corresponding code paths are explicitly hooked in maintenance.
julio-lopez
changed the base branch from
draft/no-index-cleanup-on-load
to
master
Conditionally disables epoch index maintenace operations when loading indexes. This prevents (potentially expensive) cleanup write operations on the index read path. The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad` field. Refs: - kopia#3174 - kopia#3224 - kopia#3225
julio-lopez
force-pushed
the
fix/no-index-cleanup-on-load-env-var
branch
from
1eabe17
to
f903815
Compare
Shrekster
changed the title
julio-lopez
added a commit
to kastenhq/kopia
that referenced
this pull request
Mar 6, 2024
julio-lopez
added a commit
to kastenhq/kopia
that referenced
this pull request
Mar 7, 2024
julio-lopez
added a commit
that referenced
this pull request
Mar 7, 2024
julio-lopez
added a commit
to kastenhq/kopia
that referenced
this pull request
Mar 12, 2024
Extracted from kopia#3651. Thanks to @plar and @redgoat650 for the suggestions. Ref: - kopia#3603 - kopia#3645 - kopia#3638 - kopia#3639
julio-lopez
added a commit
that referenced
this pull request
Mar 12, 2024
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 18, 2024
…#3645) Conditionally disables epoch index maintenance operations when loading indexes. This prevents (potentially expensive) cleanup write operations on the index read path. The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad` field, which can be temporarily overridden via an environment variable. This override mechanism will be removed in the near future. Refs: - kopia#3174 - kopia#3224 - kopia#3225 - kopia#3638 - kopia#3639
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 22, 2024
…#3645) Conditionally disables epoch index maintenance operations when loading indexes. This prevents (potentially expensive) cleanup write operations on the index read path. The behavior is controlled via the `epoch.Manager.allowCleanupWritesOnIndexLoad` field, which can be temporarily overridden via an environment variable. This override mechanism will be removed in the near future. Refs: - kopia#3174 - kopia#3224 - kopia#3225 - kopia#3638 - kopia#3639
julio-lopez
added a commit
that referenced
this pull request
Mar 27, 2024
…3651) Perform index epoch compaction and cleanup during repository maintenance * refactor: rename maintenance task for deleting superseded indexes * maintenance task to cleanup epoch markers * maintenance task to advance write index epoch * maintenance task to compact epoch ranges * quick maintenance task to compact a single (index) epoch * full maintenance task to compact a single (index) epoch Ref: - #3638 - #3639 Followup to: - #3603 - #3645 Related helpers and changes: - #3721 - #3735 - #3709 - #3728 - #3727 - #3726
Lyndon-Li
pushed a commit
to Lyndon-Li/kopia
that referenced
this pull request
Mar 29, 2024
…opia#3651) Perform index epoch compaction and cleanup during repository maintenance * refactor: rename maintenance task for deleting superseded indexes * maintenance task to cleanup epoch markers * maintenance task to advance write index epoch * maintenance task to compact epoch ranges * quick maintenance task to compact a single (index) epoch * full maintenance task to compact a single (index) epoch Ref: - kopia#3638 - kopia#3639 Followup to: - kopia#3603 - kopia#3645 Related helpers and changes: - kopia#3721 - kopia#3735 - kopia#3709 - kopia#3728 - kopia#3727 - kopia#3726
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is only a temporary mechanism for changing the default behavior.
Future PRs will:
Followup to: #3603
Ref: