Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix write volume over size MaxPossibleVolumeSize #5190

Merged
merged 1 commit into from Jan 12, 2024

Conversation

kungf
Copy link
Contributor

@kungf kungf commented Jan 11, 2024

What problem are we solving?

volume over size MaxPossibleVolumeSize
#5182

How are we solving the problem?

  1. when write, n.Size is not init yet before prepareWriteBuf, so n.Size.IsValid() alaways false!
  2. v.nm.ContentSize() only have write data use size not include delete use size, so if volume has many delete ops(n) before full, more than n*32 bytes can also write to volume, even volume true offset is large than MaxPossibleVolumeSize

How is the PR tested?

Checks

  • I have added unit tests if possible.
  • I will add related wiki document changes and link to this PR after merging.

@chrislusf
Copy link
Collaborator

chrislusf commented Jan 11, 2024

Thanks for the findings and the PR!

@kungf
Copy link
Contributor Author

kungf commented Jan 12, 2024

hi @chrislusf,I am glad to be able to do something for SeaweedFS!

  • How is this PR tested?

I will test it by these steps:

  1. write some files
  2. delete these files
  3. repeated 1,2 until volume is full
    I wrote a tool to check whether needs to set over flow。
  • Deletion can still work if over the size limit?

Yes, If we disable deletion after the volume is full, it will be unable to reclaim space。

when we only check size by “MaxPossibleVolumeSize < v.nm.ContentSize()+uint64(actualSize)”,and v.nm.ContentSize() only includes write data size but does not include delete op size, so if the volume had many deletions, even " MaxPossibleVolumeSize < v.nm.ContentSize()+uint64(actualSize)" is false, write needle offset can also exceed MaxPossibleVolumeSize。

This pr can fix size mismatch err when vacuuming。

@chrislusf chrislusf merged commit 0e8a54f into seaweedfs:master Jan 12, 2024
5 checks passed
@kungf kungf deleted the fix_write_oversize branch January 12, 2024 06:12
@kungf kungf mentioned this pull request Jan 15, 2024
2 tasks
kmlebedev pushed a commit to kmlebedev/seaweedfs that referenced this pull request Apr 8, 2024
kmlebedev pushed a commit to kmlebedev/seaweedfs that referenced this pull request Apr 8, 2024
fix: Revert "fix write volume over size MaxPossibleVolumeSize (seaweedfs#5190)"

See merge request infrastructure/utils/storage/seaweedfs!9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants