[SPARK-33216][INFRA][PYTHON][2.4] Set upper bound of Pandas version in GitHub Actions

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to pin Pandas version up to 1.1.0 released on July 28.

Why are the changes needed?

branch-2.4 GitHub Action was broken since Aug 26 due to Pandas API incompatibility.

• https://github.com/apache/spark/runs/1034685663

The following is just one example.

File "/home/runner/work/spark/spark/python/pyspark/sql/tests.py", line 5948, in change_col_order
    return pd.DataFrame.from_items([
AttributeError: type object 'DataFrame' has no attribute 'from_items'

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the GitHub Action.

[dongjoon-hyun]

@HyukjinKwon How do you think about this? branch-2.4 GitHub Action seems to be broken for a while.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34728/

[dongjoon-hyun]

@HyukjinKwon . I'll close this PR in favor of SPARK-30777.

[dongjoon-hyun]

Please recover branch-2.4~ Thank you in advance.

[HyukjinKwon]

Ohh okay got it. There are more tests being failed.

[HyukjinKwon]

Some of them require Arrow dep upgrade in pom which isn't possible. I thought I fixed it by setting upper bound of PyArrow. Hm, let me fix it by the same approach proposed here.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34728/

[HyukjinKwon]

I discussed offline with @dongjoon-hyun. It requires to port at least two PRs 07a9885 and 16990f9 + Arrow dependency upgrade in POM side which is not possible. Setting the upper bound looks correct approach. It was my bad that I thought it just needs one backport.

[dongjoon-hyun]

Thank you for investigating all the stuff and guiding this, @HyukjinKwon !

[HyukjinKwon]

LGTM: it can just set to whichever pandas version that makes the build in branch-2.4

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34729/

[HyukjinKwon]

@dongjoon-hyun, I will open another PR with picking your commits - seems it fails.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34729/

[SparkQA]

Test build #130121 has finished for PR 30127 at commit 260a5d5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #130120 has finished for PR 30127 at commit 9729565.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #130122 has finished for PR 30127 at commit 260a5d5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.