Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [tez] Fix the user of remote storage. #1109

Closed
3 tasks done
zhengchenyu opened this issue Aug 7, 2023 · 0 comments · Fixed by #1128
Closed
3 tasks done

[Bug] [tez] Fix the user of remote storage. #1109

zhengchenyu opened this issue Aug 7, 2023 · 0 comments · Fixed by #1128

Comments

@zhengchenyu
Copy link
Collaborator

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

When we use remote storage, we will flush data to hdfs, but the user of the storage is appid. See the storage:

[yarn@host dir]$ hdfs dfs -ls /uniffle-rss
Found 1 items
drwxrwx--x   - application_1691045129773_0174 yarn          0 2023-08-04 16:52 /uniffle-rss/appattempt_1691045129773_0174_000001

It seems reasonable that regard applicationid as user, because it is more safe. But we know the max number of hadoop user is 16777215, it is limited. Even though the related remote storage directory is deleted, this user string won't be delete from SerialNumberMap util restart namenode.

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zhengchenyu zhengchenyu mentioned this issue Aug 10, 2023
Merged
zuston pushed a commit that referenced this issue Aug 10, 2023
@zhengchenyu
[#1109] fix(tez): Fix the user of remote storage. (#1128)
0bde6ba 
### What changes were proposed in this pull request?

tez application should use login ugi to  communicate with shuffle sever, but not applicationid ugi.

### Why are the changes needed?

Fix: #1109

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

test in real cluster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#1109] fix(tez): Fix the user of remote storage. zhengchenyu/incubator-uniffle
1 participant
@zhengchenyu