Skip to content

Fix S3 Select payload join#16189

Merged
potiuk merged 2 commits intoapache:mainfrom
TAKEDA-Takashi:fix-s3-select-payload-join
Jun 2, 2021
Merged

Fix S3 Select payload join#16189
potiuk merged 2 commits intoapache:mainfrom
TAKEDA-Takashi:fix-s3-select-payload-join

Conversation

@TAKEDA-Takashi
Copy link
Copy Markdown
Contributor

The original code has a potential bug that decodes fail if the payload is split in the middle of a multibyte character.
This can be avoided by joining and then decoding.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 1, 2021
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2021

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions Bot added the okay to merge It's ok to merge this PR as it does not require more tests label Jun 1, 2021
Copy link
Copy Markdown
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add regression test for that ?

Do I understand correctly that it fixes the problem where the records split is in the middle of multi-byte character? If so it should be fairly easy to add unit test for.

@TAKEDA-Takashi
Copy link
Copy Markdown
Contributor Author

Added commit.
The actual payload is about 65KB, which is long data, but the test data is short.

Copy link
Copy Markdown
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect! Thank you!

@potiuk potiuk merged commit 9fcdf3d into apache:main Jun 2, 2021
@TAKEDA-Takashi TAKEDA-Takashi deleted the fix-s3-select-payload-join branch June 11, 2021 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers okay to merge It's ok to merge this PR as it does not require more tests provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants