Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't MarkForDeletion a chunk until all references to it have been swapped #1914

Merged

Conversation

2 participants
@lscpike
Copy link
Contributor

commented Apr 21, 2019

Don't MarkForDeletion a chunk until all references to it have been swapped out. This avoids it still being accessible for reads resulting in FileBeingDeletedException.

I'm 99% we've been hitting this whilst scavenging. The scenario is:

  • Previously scavenged and merged multiple chunks.
  • Scavenging multiple threads.
  • 1 thread finishes re-scavenging a merged chunk (chunk "#1-5") and swaps in in the DB (TfChunkManager).
  • TfChunkManager swaps Chunk index 1 for the new chunk and deletes the old chunk.
  • Note indices 2-5 still point at the deleted chunk.
  • Another thread is scavenging a different but close proximity chunk.
  • The second thread needs to check for stream deletion of a stream so reads the metadata which happens to be in chunk "#2".
  • BOOM! FileBeingDeletedException

This is obviously a problem for any read operation on the chunk, not just the scavenge process.

The fix is simple; don't MarkForDeletion until all references in the DB are swapped.

Don't MarkForDeletion a chunk until all references to it have been sw…
…apped out. This avoids it still being accessible for reads resulting in FileBeingDeletedException.
@shaan1337

This comment has been minimized.

Copy link
Member

commented Apr 30, 2019

@lscpike looks good. thanks for your contribution! we have seen at least one other case where the bug occurred.

@shaan1337 shaan1337 merged commit af32ca8 into EventStore:master Apr 30, 2019

9 checks passed

EventStore.EventStore Build #20190421.1 succeeded
Details
EventStore.EventStore (Centos 7 x64 Debug) Centos 7 x64 Debug succeeded
Details
EventStore.EventStore (Centos 7 x64 Release) Centos 7 x64 Release succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Debug) Ubuntu 14.04 x64 Debug succeeded
Details
EventStore.EventStore (Ubuntu 14.04 x64 Release) Ubuntu 14.04 x64 Release succeeded
Details
EventStore.EventStore (Windows x64 Debug) Windows x64 Debug succeeded
Details
EventStore.EventStore (Windows x64 Release) Windows x64 Release succeeded
Details
EventStore.EventStore (macOS x64 Debug) macOS x64 Debug succeeded
Details
EventStore.EventStore (macOS x64 Release) macOS x64 Release succeeded
Details

@lscpike lscpike deleted the lscpike:bugfix/deleted-chunk-still-accessible branch Apr 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.