Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Subtask] [Netty] Use off heap memory to read HDFS data #596

Closed
3 tasks done
Tracked by #133
jerqi opened this issue Feb 13, 2023 · 1 comment · Fixed by #806
Closed
3 tasks done
Tracked by #133

[Subtask] [Netty] Use off heap memory to read HDFS data #596

jerqi opened this issue Feb 13, 2023 · 1 comment · Fixed by #806

Comments

@jerqi
Copy link
Contributor

jerqi commented Feb 13, 2023 •
edited

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the subtask

We need to use off heap memory to read HDFS data to reduce GC.

Parent issue

#133

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@jerqi jerqi mentioned this issue Feb 22, 2023
Closed
21 tasks
@jerqi
Copy link
Contributor Author

jerqi commented Apr 6, 2023

Method read can use ByteBuffer to read data. https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html

jerqi added a commit to jerqi/incubator-uniffle that referenced this issue Apr 10, 2023
@jerqi
[apache#596] Use off heap memory to read HDFS data
e302695
@jerqi jerqi mentioned this issue Apr 10, 2023
Merged
jerqi added a commit that referenced this issue Apr 13, 2023
@jerqi
[#596] feat(netty): Use off heap memory to read HDFS data (#806)
c6cde5d 
### What changes were proposed in this pull request?
1. Use off heap memory to read HDFS data
2. remove some unused code
(to do: use off heap memory to read HDFS index data)

### Why are the changes needed?
Fix: #596 

### Does this PR introduce _any_ user-facing change?
Yes, add the document.

### How was this patch tested?
Pass origin tests.
jerqi added a commit to jerqi/incubator-uniffle that referenced this issue May 8, 2023
@jerqi
[apache#596][FOLLOWUP] Index data support offheap read
6357775
@jerqi jerqi mentioned this issue May 8, 2023
Merged
jerqi added a commit that referenced this issue May 10, 2023
@jerqi
[#596][FOLLOWUP] Index data support offheap read (#852)
0125fe3 
### What changes were proposed in this pull request?
We support to read off heap data in the #806, we support to read index in this pr.

### Why are the changes needed?
#596 follow up pr

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?
GA passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#596] feat(netty): Use off heap memory to read HDFS data jerqi/incubator-uniffle
1 participant
@jerqi