Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] app localdisk folder remains when app is expired #1634

Closed
3 tasks done
zuston opened this issue Apr 10, 2024 · 0 comments · Fixed by #1635
Closed
3 tasks done

[Bug] app localdisk folder remains when app is expired #1634

zuston opened this issue Apr 10, 2024 · 0 comments · Fixed by #1635

Comments

@zuston
Copy link
Member

zuston commented Apr 10, 2024

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

I found some shuffle-servers basic dir has lots of app folders remains wich has been expired.

After digging this case, I think this is caused by the children shuffleIds unregistering. Please see here: https://github.com/apache/incubator-uniffle/blob/a7a0b4ce41a213f8f3e05a1e1198c41857706229/server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java#L778-#L781 .

Once one app's all shuffleIds has been unregistered by the ShufflePurgeEvent, the above code will be invalid, because shuffleToCachedBlockIds will be empty and them won't trigger the removeResource by app level. So the app folder still remains after expire.

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
zuston added a commit to zuston/incubator-uniffle that referenced this issue Apr 10, 2024
@zuston
[apache#1634] fix(server): remove app folder if app is expired
6f42d0d
@zuston zuston mentioned this issue Apr 10, 2024
Merged
jerqi pushed a commit that referenced this issue Apr 13, 2024
@zuston
[#1634] fix(server): remove app folder if app is expired (#1635)
6ea4300 
### What changes were proposed in this pull request?

Always executing the removeResources no matter whether blockIds remain.

### Why are the changes needed?

Fix: #1634 

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Using existing unit tests, but additionally checking the app folder after app is expired.
jerqi pushed a commit that referenced this issue Apr 30, 2024
@zuston
[#1634] fix(server): remove app folder if app is expired (#1635)
9e0cf9f 
### What changes were proposed in this pull request?

Always executing the removeResources no matter whether blockIds remain.

### Why are the changes needed?

Fix: #1634 

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Using existing unit tests, but additionally checking the app folder after app is expired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#1634] fix(server): remove app folder if app is expired zuston/incubator-uniffle
1 participant
@zuston