Merge pull request #11245 - Update the range for pyarrow to support new pyarrow version 0.16.0

[aaltay]

Update the range for pyarrow to support new pyarrow version 0.16.0

Is this a regression from a previous release?

Kind of. Latest version of pyarrow is not supported blocking a down stream users next release (TFX).

Is this a new feature or related to a new feature?

No.

Would this impact production workloads for users?

Maybe. Depends on what is pre-installed in containers. We do not need to change what is pre-installed and most users would not have an issue.

What percentage of users would be impacted by this issue if it is not fixed?

TFX users. (Probably >10% user base.)

Would it be possible for the impacted users to skip this version?

Yes, but they will need to delay their release until 2.21.0.

@amaliujia - Please review this. I am not sure if this is a clear cut case for a cherry pick, I will leave up to you to decide.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[brills]

Thanks @aaltay -

To add some context and to clarify a bit on the severity of the issue:

pyarrow 0.16 contains new features and bug fixes that we (TFX) care about (just to list a few: apache/arrow#6066
apache/arrow#6006). So it would be nice if TFX can start using it in the next version (we were planning to release it at the end of April). But by that time, the latest beam would be 2.20.

Note that this is not to say if this PR is not merged, any bad consequence will happen, because we can also decide to not start using pyarrow 0.16, and still rely on / add workarounds on our end. it's just more technical debt.

[aaltay]

Python pre commit failed with an unrelated error:

11:24:10 > Failed to list versions for com.google.errorprone:error_prone_core.

[aaltay]

Run Python PreCommit

[amaliujia]

LGTM. Will merge this PR after tests finish.

[aaltay]

I believe last test failed with another flake.

[aaltay]

Run Python PreCommit

[aaltay]

Run Python PreCommit

[amaliujia]

@aaltay I will take care of this PR to get it merged.

[amaliujia]

Run Python PreCommit

[aaltay]

@aaltay I will take care of this PR to get it merged.

Thank you!

[amaliujia]

Run Python PreCommit

[amaliujia]

Run Python PreCommit

[amaliujia]

Run Python PreCommit

[amaliujia]

I don't think the failed python precommit is related to this change. Will merge this PR.

[aaltay]

I don't think the failed python precommit is related to this change. Will merge this PR.

Thank you.