Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

verify uncompressed file does not exists under /xlog and /timeline [BF-1358] #592

Merged
merged 5 commits into from
Jul 6, 2023

Conversation

kathia-barahona
Copy link
Contributor

About this change - What it does

  • Skip re-compressing during startup_walk. This happens when restarting pghoard while the deletion queue has not been emptied. Meaning, that pghoard might had already compressed some files but not deleted them yet, which ends up on duplicated upload events.

    For example during startup, WAL file 00001 exists in both /xlog and /xlog_incoming, pghoard will attempt to compress all files in /xlog_incoming and later upload files in /xlog. The transfer agent will end up adding 2 upload events for 00001 compressed file, but the second event will fail since the compressed file might have been already deleted, so the transfer agent will keep retrying and end up getting stuck.

  • Missing uploading compressed timeline files, noticed that during the startup walk, pghoard is only checking files in xlog path, but compressed timelines are stored under /timeline.

Resolves: #BF-1358

pghoard/pghoard.py Outdated Show resolved Hide resolved
@kathia-barahona kathia-barahona marked this pull request as ready for review July 3, 2023 12:02
@kathia-barahona kathia-barahona requested review from a team as code owners July 3, 2023 12:02
pghoard/pghoard.py Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Jul 4, 2023

Codecov Report

Merging #592 (60d6bf6) into main (6dce5bf) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #592      +/-   ##
==========================================
- Coverage   91.31%   91.29%   -0.02%     
==========================================
  Files          32       32              
  Lines        4684     4709      +25     
==========================================
+ Hits         4277     4299      +22     
- Misses        407      410       +3     
Impacted Files Coverage Δ
pghoard/pghoard.py 84.99% <100.00%> (+0.12%) ⬆️

pghoard/pghoard.py Outdated Show resolved Hide resolved
pghoard/pghoard.py Outdated Show resolved Hide resolved
@alexole alexole changed the title verify uncompressed file does not exists under /xlog and /timeline (i… verify uncompressed file does not exists under /xlog and /timeline [BF-1358] Jul 6, 2023
@alexole alexole merged commit 87a01e9 into main Jul 6, 2023
9 of 10 checks passed
@alexole alexole deleted the skip_recompression_in_startup branch July 6, 2023 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants