Change Documentation page "Python Package Management" url enclosed

[yetanotherlogonfail]

Proposed change to User Guide documentation page

"Python Package Management"
URL
https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html

Reason: The paragraph is unclear
user-facing change: Yes

Change
Paragraph: Using PySpark Native Features

From:
PySpark allows to upload Python files (.py), zipped Python packages (.zip), and Egg files (.egg) to the executors by:

Setting the configuration setting spark.submit.pyFiles

Setting --py-files option in Spark scripts

Directly calling pyspark.SparkContext.addPyFile() in applications

This is a straightforward method to ship additional custom Python code to the cluster. You can just add individual files or zip whole packages and upload them. Using pyspark.SparkContext.addPyFile() allows to upload code even after having started your job.

However, it does not allow to add packages built as Wheels and therefore does not allow to include dependencies with native code.

TO:
PySpark allows to upload Python files (.py), zipped Python packages (.zip), and Egg files (.egg) to the executors by:

Setting the configuration setting spark.submit.pyFiles
OR
Setting --py-files option in Spark scripts
OR
Directly calling pyspark.SparkContext.addPyFile() in applications

This is a straightforward method to ship additional custom Python code to the cluster. You can just add individual files or zip whole packages and upload them. Using pyspark.SparkContext.addPyFile() allows to upload code even after having started your job.

However, it does not allow to add packages built as Wheels and therefore does not allow to include dependencies with native code.

[AmplabJenkins]

Can one of the admins verify this patch?

[yetanotherlogonfail]

Hyukjin Kwon (Jira) ***@***.***> 02:39 (7 hours ago) to carltonrees [ https://issues.apache.org/jira/browse/SPARK-37134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435117#comment-17435117 ] Hyukjin Kwon commented on SPARK-37134: -------------------------------------- They are individual items so OR is correct. Feel free to create a PR to clarify them.

…

On Thu, 28 Oct 2021 at 10:13, UCB AMPLab ***@***.***> wrote: Can one of the admins verify this patch? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#34422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRBKWW74D6IOMDCZMY2AY3UJEH2XANCNFSM5G4I3TLA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Regards, Carl "Chance has put in our way a most singular and whimsical problem, and its solution is its own reward."