Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't buffer postgres sources messages as aggressively #12557

Closed
guswynn opened this issue May 18, 2022 · 1 comment
Closed

Don't buffer postgres sources messages as aggressively #12557

guswynn opened this issue May 18, 2022 · 1 comment
Labels
A-STORAGE Topics related to the Storage layer C-refactoring Category: replacing or reorganizing code

Comments

@guswynn
Copy link
Contributor

guswynn commented May 18, 2022

In #11946, we buffer messages from the pg source in 2 places. We should strive to remove one or both of these:

  • We buffer all InProgress in the reclocking process. The ReclockOperator can likely be adjusted to hand out a timestamp but not seal off that timestamp for offsets that have not reached Finalized status cc @cjubb39
  • We buffer transactions in the postgres source itself, both in snapshots and replication. These can likely be made to aggressively emit InProgress messages and send Finalized messages at the end
@guswynn guswynn added the C-refactoring Category: replacing or reorganizing code label May 18, 2022
@guswynn guswynn added this to To Do in Storage (Old) May 19, 2022
@guswynn guswynn added the A-STORAGE Topics related to the Storage layer label May 19, 2022
@guswynn guswynn changed the title Don't buffer InProgress messaged Don't buffer postgres sources messages as aggressively May 19, 2022
@benesch
Copy link
Member

benesch commented Jul 11, 2022

Fixed by #13246.

@benesch benesch closed this as completed Jul 11, 2022
Storage (Old) automation moved this from To Do to Landed Jul 11, 2022
@nmeagan11 nmeagan11 removed this from Landed in Storage (Old) Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-STORAGE Topics related to the Storage layer C-refactoring Category: replacing or reorganizing code
Projects
None yet
Development

No branches or pull requests

2 participants