Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] StorageInfo isCorrupted state can't turn into true by LocalStorageChecker mark corrupted #675

Closed
3 tasks done
xumanbu opened this issue Mar 2, 2023 · 2 comments · Fixed by #677
Closed
3 tasks done

Comments

@xumanbu
Copy link
Contributor

xumanbu commented Mar 2, 2023

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

LocalStorageChecker checker storage can't be write when storage disk full LocalStorageChecker checker storage unhealthy,and mark storage is corrupted. the storage can not be used never, although the disk space release in future.

https://github.com/apache/incubator-uniffle/blob/master/server/src/main/java/org/apache/uniffle/server/LocalStorageChecker.java

      if (!storageInfo.checkStorageReadAndWrite()) {
        storageInfo.markCorrupted();
        corruptedDirs++;
        continue;
      }

fix code may like blow :

      if (!storageInfo.checkStorageReadAndWrite()) {
        storageInfo.markCorrupted(true);
        corruptedDirs++;
        continue;
      }else{
        storageInfo.markCorrupted(false);
      }

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@jerqi
Copy link
Contributor

jerqi commented Mar 2, 2023

If the disk have bad tracks, it may cause that we write successfully sometimes, it write unsuccessffully. So we don't turn the disk to uncorrupted although the disk can write successfully.

xumanbu added a commit to xumanbu/incubator-uniffle that referenced this issue Mar 3, 2023
@xumanbu
Copy link
Contributor Author

xumanbu commented Mar 3, 2023

agree.. disk space enough should be avioded in check storage read and write.

xumanbu added a commit to xumanbu/incubator-uniffle that referenced this issue Mar 3, 2023
@jerqi jerqi closed this as completed in #677 Mar 3, 2023
jerqi pushed a commit that referenced this issue Mar 3, 2023
### What changes were proposed in this pull request?

Filter no space exception in checkStorageReadAndWrite

### Why are the changes needed?

Fix: #675 

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT

Co-authored-by: jam.xu <jam.xu@vipshop.com>
advancedxy pushed a commit to advancedxy/incubator-uniffle that referenced this issue Mar 21, 2023
…te (apache#677)

### What changes were proposed in this pull request?

Filter no space exception in checkStorageReadAndWrite

### Why are the changes needed?

Fix: apache#675 

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT

Co-authored-by: jam.xu <jam.xu@vipshop.com>
xianjingfeng pushed a commit to xianjingfeng/incubator-uniffle that referenced this issue Apr 5, 2023
…te (apache#677)

### What changes were proposed in this pull request?

Filter no space exception in checkStorageReadAndWrite

### Why are the changes needed?

Fix: apache#675 

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
UT

Co-authored-by: jam.xu <jam.xu@vipshop.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants