Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-4151. Skip the inputstream while offset larger than zero in s3g #1354

Merged
merged 2 commits into from
Aug 28, 2020
Merged

HDDS-4151. Skip the inputstream while offset larger than zero in s3g #1354

merged 2 commits into from
Aug 28, 2020

Conversation

maobaolong
Copy link
Member

What changes were proposed in this pull request?

Skip the inputstream while offset larger than zero in s3g.

What is the link to the Apache JIRA

HDDS-4151

How was this patch tested?

Download a big file like 10GB, and get the cost time, compare it than without this PR, this PR can get good performance back.

@maobaolong
Copy link
Member Author

@elek Please take a look at this PR, it can solve the performance issue which involved by #1212

bharatviswa504
bharatviswa504 reviewed Aug 26, 2020
View reviewed changes
Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maobaolong
Thank you for the PR, can you update Jira description and explain about the issue

@maobaolong
Copy link
Member Author

@bharatviswa504 Thanks for your review, I reply your comments inline. In fact the performance improvement comes from #843 , I just do some refactor and do some improvements by #1212 , and unfortunately, I forget to call seek or skip before the IOUtils.copyLarge, that's my bad, and I corrected it by this PR.

@bharatviswa504
Copy link
Contributor

@maobaolong
Thanks for the detailed explanation.
I have one comment, rest LGTM.

@maobaolong
Copy link
Member Author

@bharatviswa504 Thanks for your review and suggestion, I push a new commit to address your comment, PTAL.

@maobaolong maobaolong closed this Aug 28, 2020
@maobaolong maobaolong reopened this Aug 28, 2020
@ChenSammi
Copy link
Contributor

ChenSammi commented Aug 28, 2020 •
edited
Loading

@maobaolong , s3WrapperInputStream.copyLarge has implemented the skip and copy. You can just leverage this function.

@maobaolong
Copy link
Member Author

I have discussed with @ChenSammi Offline, s3WrapperInputStream.copyLarge is a duplicated code which copy from IOUtils.copyLarge, It has been improved by #1212.

bharatviswa504
bharatviswa504 approved these changes Aug 28, 2020
View reviewed changes
Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@bharatviswa504 bharatviswa504 merged commit 8e98977 into apache:master Aug 28, 2020
@bharatviswa504
Copy link
Contributor

Thank You @maobaolong for the contribution and @ChenSammi for the review.

@maobaolong
Copy link
Member Author

@bharatviswa504 Thanks for review and merge it for me.

rakeshadr pushed a commit to rakeshadr/hadoop-ozone that referenced this pull request Sep 3, 2020
@maobaolong @rakeshadr
HDDS-4151. Skip the inputstream while offset larger than zero in s3g (a…
c72680d 
…pache#1354)
errose28 added a commit to errose28/ozone that referenced this pull request Sep 11, 2020
@errose28
Merge branch 'master' into HDDS-4122
c2043a2 
* master: (26 commits)
  HDDS-4167. Acceptance test logs missing if fails during cluster startup (apache#1366)
  HDDS-4121. Implement OmMetadataMangerImpl#getExpiredOpenKeys. (apache#1351)
  HDDS-3867. Extend the chunkinfo tool to display information from all nodes in the pipeline. (apache#1154)
  HDDS-4077. Incomplete OzoneFileSystem statistics (apache#1329)
  HDDS-3903. OzoneRpcClient support batch rename keys. (apache#1150)
  HDDS-4151. Skip the inputstream while offset larger than zero in s3g (apache#1354)
  HDDS-4147. Add OFS to FileSystem META-INF (apache#1352)
  HDDS-4137. Turn on the verbose mode of safe mode check on testlib (apache#1343)
  HDDS-4146. Show the ScmId and ClusterId in the scm web ui. (apache#1350)
  HDDS-4145. Bump version to 1.1.0-SNAPSHOT on master (apache#1349)
  HDDS-4109. Tests in TestOzoneFileSystem should use the existing MiniOzoneCluster (apache#1316)
  HDDS-4149. Implement OzoneFileStatus#toString (apache#1356)
  HDDS-4153. Increase default timeout in kubernetes tests (apache#1357)
  HDDS-2411. add a datanode chunk validator fo datanode chunk generator (apache#1312)
  HDDS-4140. Auto-close /pending pull requests after 21 days of inactivity (apache#1344)
  HDDS-4152. Archive container logs for kubernetes check (apache#1355)
  HDDS-4056. Convert OzoneAdmin to pluggable model (apache#1285)
  HDDS-3972. Add option to limit number of items displaying through ldb tool. (apache#1206)
  HDDS-4068. Client should not retry same OM on network connection failure (apache#1324)
  HDDS-4062. Non rack aware pipelines should not be created if multiple racks are alive. (apache#1291)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharatviswa504 bharatviswa504 bharatviswa504 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@maobaolong @bharatviswa504 @ChenSammi