[SPARK-39861][PYTHON][DOCS] Deprecate Python 3.7 Support

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to deprecate Python 3.7 support at Apache Spark 3.4.0.

Why are the changes needed?

Apache Spark 3.4 will be released on February 2023 and will be supported for next 18 months.

• https://spark.apache.org/versioning-policy.html

Before Spark 3.4.1, Python 3.7 is going to reach End Of Support on 2023-06-27. Although Apache Spark 3.4 will work with Python 3.7 still for a while, there is no proper and official way to support Python 3.7 from that time.

• https://www.python.org/downloads/

Does this PR introduce any user-facing change?

Yes, but the users will see only a deprecation warning log and docs.

How was this patch tested?

Pass the CIs.

[dongjoon-hyun]

cc @HyukjinKwon , @ueshin , @zero323 , @xinrong-meng , @itholic

[dongjoon-hyun]

cc @viirya , @huaxingao , @sunchao , too

[viirya]

The timing looks good to me.

[viirya]

But it is better to open this for a while for others to take a look.

[sunchao]

just curious whether the version number could be something different from 3.7? if so perhaps the warning message could be something like Python support for version <= 3.7 is deprecated in Spark 3.4.

[viirya]

Same though at first glance. But as we only support Python3.7+, I think there might be other version check so here it is 3.7 for sure? Not sure. If not, then revised error message would be better.

[dongjoon-hyun]

Good question. It's unlikely because we did the following since Apache Spark 3.2 for a while. This is the first step and we did many efforts to kill 3.6 completely. The following code is removed at branch-3.3 after that. :)

spark/python/pyspark/context.py

Lines 233 to 239 in 0d11085

	if sys.version_info[:2] < (3, 7):
	with warnings.catch_warnings():
	warnings.simplefilter("once")
	warnings.warn(
	"Python 3.6 support is deprecated in Spark 3.2.",
	FutureWarning
	)

[dongjoon-hyun]

In addition, Spark 3.4 requires Python 3.7+ at the installation stage explicitly.

spark/python/setup.py

Line 277 in 8628c15

python_requires='>=3.7',

[sunchao]

I see, never mind then :)

[dongjoon-hyun]

Thank you, @viirya , @sunchao , @huaxingao , @ueshin . Merged to master for Apache Spark 3.4.

[itholic]

Thanks for the fix!