-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
verify uncompressed file does not exists under /xlog and /timeline [BF-1358] #592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kathia-barahona
commented
Jul 3, 2023
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
July 3, 2023 11:56
199b849
to
d850c84
Compare
kathia-barahona
commented
Jul 3, 2023
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
July 3, 2023 13:33
95efb01
to
a2aa969
Compare
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
July 4, 2023 08:28
a2aa969
to
500cb54
Compare
alexole
reviewed
Jul 4, 2023
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #592 +/- ##
==========================================
- Coverage 91.31% 91.29% -0.02%
==========================================
Files 32 32
Lines 4684 4709 +25
==========================================
+ Hits 4277 4299 +22
- Misses 407 410 +3
|
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
July 4, 2023 08:58
500cb54
to
3815837
Compare
alexole
reviewed
Jul 4, 2023
alexole
reviewed
Jul 4, 2023
alexole
reviewed
Jul 4, 2023
…_skip_compression_if_already_compressed
alexole
changed the title
verify uncompressed file does not exists under /xlog and /timeline (i…
verify uncompressed file does not exists under /xlog and /timeline [BF-1358]
Jul 6, 2023
alexole
reviewed
Jul 6, 2023
alexole
approved these changes
Jul 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
Skip re-compressing during startup_walk. This happens when restarting pghoard while the deletion queue has not been emptied. Meaning, that pghoard might had already compressed some files but not deleted them yet, which ends up on duplicated upload events.
For example during startup, WAL file
00001
exists in both/xlog
and/xlog_incoming
, pghoard will attempt to compress all files in/xlog_incoming
and later upload files in/xlog
. The transfer agent will end up adding 2 upload events for00001
compressed file, but the second event will fail since the compressed file might have been already deleted, so the transfer agent will keep retrying and end up getting stuck.Missing uploading compressed timeline files, noticed that during the startup walk, pghoard is only checking files in
xlog
path, but compressed timelines are stored under/timeline
.Resolves: #BF-1358