Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ABFS] Fix download by reading file correctly instead of extra 1 byte always with every chunk #3472

Merged
merged 1 commit into from
Sep 22, 2023

Conversation

agl29
Copy link
Collaborator

@agl29 agl29 commented Sep 22, 2023

What changes were proposed in this pull request?

  • We were reading extra 1 byte from the abfs file
  • Header['range'] = headers['range'] = 'bytes=x-y' in this x is starting point and y is end point and both are included hence we removed the 1 byte from y to make sure we are not duplicating the bytes.

How was this patch tested?

  • Locally and in live DH cluster

@Harshg999 Harshg999 enabled auto-merge (squash) September 22, 2023 10:50
Copy link
Collaborator

@Harshg999 Harshg999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Harshg999 Harshg999 merged commit b82b388 into master Sep 22, 2023
2 checks passed
@Harshg999 Harshg999 deleted the abfs_file_download3 branch September 22, 2023 10:58
tabraiz12 pushed a commit that referenced this pull request Oct 4, 2023
… always with every chunk (#3472)

- When trying to read the file, we are giving wrong endByte value as per https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations#format-2-bytesstartbyte-endbyte.

- We were reading 1 byte extra for every 1MB chunk size value i.e instead of 0-1023 we were sending 0-1024. This was messing up with the downloaded file and lead to truncation of end bytes to fit the downloaded size.

Header['range'] = headers['range'] = 'bytes=x-y' in this x is starting point and y is end point and both are included hence we removed the 1 byte from y to make sure we are not duplicating the bytes.

- We checked and this issue is only for ABFS, other FS in Hue should not show this behaviour.

(cherry picked from commit b82b388)
wing2fly pushed a commit that referenced this pull request Mar 6, 2024
…extra 1 byte always with every chunk (#3472)

- When trying to read the file, we are giving wrong endByte value as per https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations#format-2-bytesstartbyte-endbyte.

- We were reading 1 byte extra for every 1MB chunk size value i.e instead of 0-1023 we were sending 0-1024. This was messing up with the downloaded file and lead to truncation of end bytes to fit the downloaded size.

Header['range'] = headers['range'] = 'bytes=x-y' in this x is starting point and y is end point and both are included hence we removed the 1 byte from y to make sure we are not duplicating the bytes.

- We checked and this issue is only for ABFS, other FS in Hue should not show this behaviour.

(cherry picked from commit b82b388)
(cherry picked from commit 331315e)
Change-Id: I35a76662314099d51f2c5ff70a0d5a710fe2b9d3
(cherry picked from commit 8ffa705)
athithyaaselvam pushed a commit that referenced this pull request Mar 14, 2024
…extra 1 byte always with every chunk (#3472)

- When trying to read the file, we are giving wrong endByte value as per https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations#format-2-bytesstartbyte-endbyte.

- We were reading 1 byte extra for every 1MB chunk size value i.e instead of 0-1023 we were sending 0-1024. This was messing up with the downloaded file and lead to truncation of end bytes to fit the downloaded size.

Header['range'] = headers['range'] = 'bytes=x-y' in this x is starting point and y is end point and both are included hence we removed the 1 byte from y to make sure we are not duplicating the bytes.

- We checked and this issue is only for ABFS, other FS in Hue should not show this behaviour.

(cherry picked from commit b82b388)
(cherry picked from commit 331315e)
Change-Id: I35a76662314099d51f2c5ff70a0d5a710fe2b9d3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants