Skip to content

[SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test#50293

Closed
LuciferYang wants to merge 2 commits intoapache:masterfrom
LuciferYang:install-python311-for-macos
Closed

[SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test#50293
LuciferYang wants to merge 2 commits intoapache:masterfrom
LuciferYang:install-python311-for-macos

Conversation

@LuciferYang
Copy link
Contributor

@LuciferYang LuciferYang commented Mar 17, 2025

What changes were proposed in this pull request?

This pr enables macOS to use actions/setup-python to install Python 3.11, so that the relevant modules can be tested using Python 3.11.

Why are the changes needed?

Although the macOS daily test uses macOS-15, which comes with Python 3.9 to Python 3.13 installed by default, the python3 command points to Python 3.13 by default instead of Python 3.11, which is required for the tests. This leads to some unexpected behaviors during testing. For example, I encountered the following logs while testing:

- udaf/udaf-group-analytics.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-group-by-ordinal.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-group-by.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-grouping-set.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!

This is because the relevant modules were installed using Python 3.11, but the python3 is actually Python 3.13. Therefore, this pull request restores the installation process for Python 3.11 on macOS, so that the tests also use Python 3.11.

Does this PR introduce any user-facing change?

No, just for test

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the INFRA label Mar 17, 2025
@LuciferYang LuciferYang changed the title Test maven test Restore the installation of Python 3.11 in macOS daily test. Mar 17, 2025
@LuciferYang LuciferYang changed the title Restore the installation of Python 3.11 in macOS daily test. [INFRA]Restore the installation of Python 3.11 in macOS daily test. Mar 17, 2025
@LuciferYang LuciferYang changed the title [INFRA]Restore the installation of Python 3.11 in macOS daily test. [INFRA] Restore the installation of Python 3.11 in macOS daily test. Mar 17, 2025
@LuciferYang LuciferYang marked this pull request as draft March 17, 2025 05:35
@LuciferYang LuciferYang changed the title [INFRA] Restore the installation of Python 3.11 in macOS daily test. [SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test Mar 17, 2025
@LuciferYang LuciferYang marked this pull request as ready for review March 17, 2025 12:23
@LuciferYang
Copy link
Contributor Author

cc @HyukjinKwon @dongjoon-hyun

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @LuciferYang .

@dongjoon-hyun
Copy link
Member

Merged to master.

@LuciferYang
Copy link
Contributor Author

Thank you @dongjoon-hyun

@LuciferYang LuciferYang deleted the install-python311-for-macos branch March 21, 2025 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants