verify uncompressed file does not exists under /xlog and /timeline [BF-1358] #592
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kathia-barahona
commented
Jul 3, 2023
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
199b849
to
d850c84
Compare
kathia-barahona
commented
Jul 3, 2023
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
95efb01
to
a2aa969
Compare
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
a2aa969
to
500cb54
Compare
alexole
reviewed
Jul 4, 2023
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #592 +/- ##
==========================================
- Coverage 91.31% 91.29% -0.02%
==========================================
Files 32 32
Lines 4684 4709 +25
==========================================
+ Hits 4277 4299 +22
- Misses 407 410 +3
|
kathia-barahona
force-pushed
the
skip_recompression_in_startup
branch
from
500cb54
to
3815837
Compare
alexole
reviewed
Jul 4, 2023
alexole
reviewed
Jul 4, 2023
alexole
reviewed
Jul 4, 2023
…_skip_compression_if_already_compressed
alexole
changed the title
alexole
reviewed
Jul 6, 2023
alexole
approved these changes
Jul 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
Skip re-compressing during startup_walk. This happens when restarting pghoard while the deletion queue has not been emptied. Meaning, that pghoard might had already compressed some files but not deleted them yet, which ends up on duplicated upload events.
For example during startup, WAL file
00001exists in both/xlogand/xlog_incoming, pghoard will attempt to compress all files in/xlog_incomingand later upload files in/xlog. The transfer agent will end up adding 2 upload events for00001compressed file, but the second event will fail since the compressed file might have been already deleted, so the transfer agent will keep retrying and end up getting stuck.Missing uploading compressed timeline files, noticed that during the startup walk, pghoard is only checking files in
xlogpath, but compressed timelines are stored under/timeline.Resolves: #BF-1358