Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-46894][PYTHON] Move PySpark error conditions into standalone JSON file #44920

Closed
wants to merge 11 commits into from
Closed

[SPARK-46894][PYTHON] Move PySpark error conditions into standalone JSON file #44920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
4fd3a2e
more errors into JSON file
nchammas Jan 28, 2024
d136d28
graft all of pyspark
nchammas Jan 29, 2024
7f4029f
use importlib to get the JSON file
nchammas Jan 29, 2024
d2db8ce
fix comment
nchammas Jan 29, 2024
24f15e1
update location of errors in PySpark docs
nchammas Jan 29, 2024
1f7fa8e
Merge branch 'master' into pyspark-error-json
nchammas Mar 24, 2024
0192f65
anticipate global renaming of error class to condition
nchammas Mar 24, 2024
79fd115
fix syntax highlighting
nchammas Mar 29, 2024
010714d
Merge branch 'master' into pyspark-error-json
nchammas Mar 29, 2024
b18b5d2
Merge branch 'master' into pyspark-error-json
nchammas Apr 30, 2024
c0cb503
migrate away from deprecated importlib method
nchammas Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 7 additions & 2 deletions python/MANIFEST.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.

global-exclude *.py[cod] __pycache__ .DS_Store
# Reference: https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html

graft pyspark
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nchammas Seems like this ending up with adding all tests as well. Could we just include that json file alone?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, graft pulls everything.

We can try to just include what we think we need, but it's probably safer (and easier) in the long run to instead exclude what we don't want to package, like tests. Would that work for you?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it's safer so we don't miss something out ... but let's just add json file alone .. I think it's more import to get rid of unrelated files ..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Are you planning to address this in #46331 (or some other PR), or would you like me to take care of it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would appreciate if you make another PR :-)

recursive-include deps/jars *.jar
graft deps/bin
recursive-include deps/sbin spark-config.sh spark-daemon.sh start-history-server.sh stop-history-server.sh
recursive-include deps/data *.data *.txt
recursive-include deps/licenses *.txt
recursive-include deps/examples *.py
recursive-include lib *.zip
recursive-include pyspark *.pyi py.typed
include README.md

# Note that these commands are processed in the order they appear, so keep
# this exclude at the end.
global-exclude *.py[cod] __pycache__ .DS_Store
4 changes: 2 additions & 2 deletions python/docs/source/getting_started/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,15 +145,15 @@ PySpark is included in the distributions available at the `Apache Spark website
You can download a distribution you want from the site. After that, uncompress the tar file into the directory where you want
to install Spark, for example, as below:

.. parsed-literal::
.. code-block:: bash

tar xzvf spark-\ |release|\-bin-hadoop3.tgz

Ensure the ``SPARK_HOME`` environment variable points to the directory where the tar file has been extracted.
Update ``PYTHONPATH`` environment variable such that it can find the PySpark and Py4J under ``SPARK_HOME/python/lib``.
One example of doing this is shown below:

.. parsed-literal::
.. code-block:: bash

cd spark-\ |release|\-bin-hadoop3
export SPARK_HOME=`pwd`
Expand Down