[INF] Support Pyspark but not require it?

[hectormz]

I'm trying to catch up on the pyspark conversation. But I was wondering if we were supporting pyspark, but not requiring it @ericmjl , @zjpoh ?

I updated my local pyjanitor and had to download pyspark as the requirements changed, and it was about 215MB. I think that's fine if it's required for development, but might not expected for users that only anticipate using pandas. I believe rdkit is currently optional for those users that will use it (maybe something else too), and I think that could be a good idea here too.

Again, I haven't looked through the entire pyspark conversation and changes yet.

[zjpoh]

I think that makes a lot of sense. I'll look into how rdkit is set as optional and apply that to pyspark. I'm open to making that optional for development version too.

[hectormz]

Sounds good, feel free to discuss any thoughts, or we'll check it out in the PR

[hectormz]

@zjpoh , on your next pyspark related PR, do you want to add spark_functions to pytest.ini following the pattern of other modules?

Running pytest gives this warning:

Unknown pytest.mark.spark_functions - is this a typo?  You can register custom marks to avoid this warning