[AIRFLOW-1158] handle remaining bytes of the multipart upload#2337
Closed
stellah wants to merge 1 commit intoapache:masterfrom
Closed
[AIRFLOW-1158] handle remaining bytes of the multipart upload#2337stellah wants to merge 1 commit intoapache:masterfrom
stellah wants to merge 1 commit intoapache:masterfrom
Conversation
Author
|
This passed on all tests except for on Python 2.7 postgres. The fix doesn't have anything to do with this . I see that other PRs have this issue as well. |
Contributor
|
Please add tests (absence of tests is no excuse for not adding them) and update your commit message per guidelines. Btw I do think we have tests for s3. Didn't check though. |
Contributor
|
WRT tests, write tests that would break without this fix, and work with it. |
Author
|
I would need a test s3 bucket. Do you guys have one? I still don't see any s3 tests. |
Contributor
|
@stellah This is a good fix. Can you add some tests and rebase? |
Author
|
@r39132 It looks like this fix is no longer needed, as you guys use boto3 upload_file now instead of FileChunkIO. |
Contributor
|
@stellah Thx. I'll close this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dear Airflow maintainers,
This fixes multipart upload to S3 where the remaining bytes after multiples of 5*1024**3 bytes don't get uploaded.
JIRA
Tests
doesn't have tests