Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Shuffle server can not delete remote storage path in security cluster. #1111

Closed
3 tasks done
zhengchenyu opened this issue Aug 7, 2023 · 0 comments · Fixed by #1122
Closed
3 tasks done

[Bug] Shuffle server can not delete remote storage path in security cluster. #1111

zhengchenyu opened this issue Aug 7, 2023 · 0 comments · Fixed by #1122

Comments

@zhengchenyu
Copy link
Collaborator

zhengchenyu commented Aug 7, 2023 •
edited

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

In security cluster, shuffle server delete remote storage path failed, the logs:

[WARN] 2023-08-04 15:38:28,840 clearResourceThread HadoopShuffleDeleteHandler delete - Can't delete shuffle data for appId[appattempt_1691129746305_0051_000001] with 4 times
java.lang.Exception: User must be not null or empty
	at org.apache.uniffle.common.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:99)
	at org.apache.uniffle.common.filesystem.HadoopFilesystemProvider.getFilesystem(HadoopFilesystemProvider.java:57)
	at org.apache.uniffle.storage.handler.impl.HadoopShuffleDeleteHandler.delete(HadoopShuffleDeleteHandler.java:54)
	at org.apache.uniffle.server.storage.HadoopStorageManager.removeResources(HadoopStorageManager.java:113)
	at org.apache.uniffle.server.storage.MultiStorageManager.removeResources(MultiStorageManager.java:175)
	at org.apache.uniffle.server.ShuffleTaskManager.removeResources(ShuffleTaskManager.java:663)
	at org.apache.uniffle.server.ShuffleTaskManager.lambda$new$0(ShuffleTaskManager.java:162)
	at java.lang.Thread.run(Thread.java:748)

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zhengchenyu zhengchenyu changed the title [Bug] Shuffle server delete remote storage path failed. [Bug] Shuffle server delete remote storage path failed in security cluster. Aug 8, 2023
@zhengchenyu zhengchenyu changed the title [Bug] Shuffle server delete remote storage path failed in security cluster. [Bug] Shuffle server can not delete remote storage path in security cluster. Aug 8, 2023
zhengchenyu added a commit to zhengchenyu/incubator-uniffle that referenced this issue Aug 8, 2023
@zhengchenyu
[apache#1111] fix: Shuffle server can not delete remote storage path …
5050eb2 
…in security cluster.
@zhengchenyu zhengchenyu mentioned this issue Aug 8, 2023
Merged
zuston pushed a commit that referenced this issue Aug 11, 2023
@zhengchenyu
[#1111] fix: Shuffle server can not delete remote storage path of sec…
541d9dc 
…ured HDFS cluster (#1122)

### What changes were proposed in this pull request?

Fix the bug that shuffle server can not delete remote storage path in secured HDFS.

### Why are the changes needed?

Fix: #1111

### How was this patch tested?

unit test and test in cluster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@zhengchenyu