Make create_temporary_dir work with pyspark-extension only (#222)

Calling create_temporary_dir and install_pip_package (which uses the former) can now be called
in Python without the need to have the spark-extension jar in the classpath. This allows using
those methods with only pyspark-extension installed.

PYSPARK-DEPS.md

@@ -6,6 +6,9 @@ Such a deployment can be cumbersome, especially when running in an interactive n
 The `spark-extension` package allows installing Python packages programmatically by the PySpark application itself (PySpark ≥ 3.1.0).
 These packages are only accessible by that PySpark application, and they are removed on calling `spark.stop()`.
 
+Either install the `spark-extension` Maven package, or the `pyspark-extension` PyPi package (on the driver only),
+as described [here](README.md#using-spark-extension).
+
 ## Installing packages with `pip`
 
 Python packages can be installed with `pip` as follows:

python/gresearch/spark/__init__.py

@@ -17,6 +17,7 @@
 import shutil
 import subprocess
 import sys
+import tempfile
 import time
 from contextlib import contextmanager
 from pathlib import Path
@@ -426,9 +427,8 @@ def create_temporary_dir(spark: Union[SparkSession, SparkContext], prefix: str)
     if isinstance(spark, SparkSession):
         spark = spark.sparkContext
 
-    package = spark._jvm.uk.co.gresearch.spark.__getattr__("package$").__getattr__("MODULE$")
-    mktempdir = package.createTemporaryDir
-    return mktempdir(prefix)
+    root_dir = spark._jvm.org.apache.spark.SparkFiles.getRootDirectory()
+    return tempfile.mkdtemp(prefix=prefix, dir=root_dir)
 
 
 SparkSession.create_temporary_dir = create_temporary_dir