Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store-gateway overwrites the lazy-loaded.json snapshot while it's still loading blocks at startup #8255

Closed
pracucci opened this issue Jun 3, 2024 · 0 comments · Fixed by #8281
Assignees

Comments

@pracucci
Copy link
Collaborator

pracucci commented Jun 3, 2024

Today we experienced an issue where we observed store-gateway not correctly loading previously loaded blocks at startup. The issue happens when the store-gateway gets restarted while starting up, and is caused by the fact that lazy-loaded.json is overwritten with the currently loaded blocks while the store-gateway is still startup up.

For example, look at the following sequence of events:

  • T0: 100 blocks loaded in the store-gateway and written to lazy-loaded.json
  • T1: store-gateway is terminated
  • T2: store-gateway is starting, and progressively loading the 100 blocks
  • T3: store-gateway has loaded 50 out of 100 blocks, and lazy-loaded.json is updated with the 50 blocks loaded so far
  • T4: store-gateway is restarted again, while it was still loading the 100 blocks found in lazy-loaded.json at startup. At this point lazy-loaded.json contains the 50 blocks written at T3, so it lost track of the initial 100 blocks

What we want: we shouldn't update lazy-loaded.json until the initial store-gateway sync has been completed.

@pracucci pracucci changed the title Store-gateway overwrites the lazy-loaded.json snapshot while it's still lazy loading blocks at startup Store-gateway overwrites the lazy-loaded.json snapshot while it's still loading blocks at startup Jun 3, 2024
@narqo narqo self-assigned this Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants