Skip to content

Commit

Permalink
docs: spark troubleshooting
Browse files Browse the repository at this point in the history
  • Loading branch information
sbrugman committed Dec 21, 2021
1 parent 60967c9 commit 1dc0319
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions docs/source/configuration.rst
Expand Up @@ -264,3 +264,15 @@ Now that spark is installed, restart the runtime.
.config("spark.sql.session.timeZone", "GMT")
.getOrCreate()
)
Troubleshooting Spark
~~~~~~~~~~~~~~~~~~~~~

The following section documents error that you may run into using spark, and how they can be resolved.

TypeError: 'JavaPackage' object is not callable

This error occurs when pyspark cannot find the required jars. Ensure that the location of "spark.jars" or "spark.jars.packages" is correct (see examples above). Stop the spark session and restart it with the exact location. You can use the ones hosted on github using the "spark.jars.pacakges" example or place the jars locally and use "spark.jars", depending on whichever is easiest in your setup.

If you are running in a jupyter notebook, then the kernel needs to be restarted.

0 comments on commit 1dc0319

Please sign in to comment.