Remove the hard dependency on the pyspark

[alexott]

Right now, the chispa package has a hard dependency on the pyspark making it hard to use with Databricks runtime, or other compatible Spark runtime. Instead, this package should either rely on implicit dependency completely, or use something like findspark package, something like done in spark-testing-base or in pytest-spark

[MrPowers]

@alexott - thanks for reporting this & sorry for the delayed response.

I wasn't able to replicate your issue (was able to successfully attach chispa v0.6.0 to a Databricks cluster via PyPi & manually attaching the wheel), but I'm a Python n00b and I'm sure your point is valid.

In the Scala world, we add the Spark dependency like libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.4" % "provided" and the "provided" part is what makes Spark a "soft" dependency.

I updated PySpark to be a "dev-dependency" rather than a regular dependency:

[tool.poetry.dependencies]
python = ">2.7"

[tool.poetry.dev-dependencies]
pytest = "3.2.2"
pytest_describe = "^1.0.0"
pyspark = ">2.0.0"

I published chispa v0.7.0. Can you please try it out and let me know if it fixes your issue? Thanks!

[alexott]

Thank you very much Matthew! I'll check on Monday, when I get to my work laptop...

[alexott]

Thank you! I just checked, it works just fine now

[MrPowers]

@alexott - thanks for confirming!

If you ever have any additional recommendations for this library, just let me know, thanks!