Implement Clip in the Pyspark backend

[emilyreff7]

Proposed Change

Implement support to be able to execute clip in the Pyspark backend. For example:

t.col.clip(lower=0, upper=10).execute()

Tests

Moved existing pandas and dask clip tests into test_numeric.py to call on pyspark as well.

[datapythonista]

Happy with this implementation, but in case you haven't considered, you can implement an upper/lower bound in a (probably) simpler way using min and max. Like:

def clip_lower(value, lower):
    return max(value, lower)

And equivalent for upper with min. I think your implementation here would be way simpler using this approach (and not sure if the translated expression could be faster too.

[icexelloss]

@datapythonista Maybe I misunderstood your suggestion, but the object that we work with here are pyspark columns and sth like

max(spark_column, 0.0)

won't really work here.

[icexelloss]

Or do you mean to implement like this?

def column_max(col1, col2):
    return when(col1 < col2, col2).otherwise(col1)
    
def column_min(col1, col2):
    return when(col1 < col2, col1).otherwise(col2)
    
def clip(col, lower, upper):
    return column_max(column_min(col, F.lit(upper)), F.lit(lower))

[datapythonista]

Didn't use pyspark for a while and I don't remember, but isn't it a pyspark function to get the min or max between a column and a literal? Or if it's not, we should have it in Ibis, we could write this using them.

In any case, not important, if this is simpler, let's just go with this approach.

[emilyreff7]

I don't believe there is any native pyspark function that will directly get us this min/max comparison functionality. Probably easiest to leave it with the when/otherwise paradigm for now.

[datapythonista]

You can now merge master, and the CI should be fixed.

[jreback]

@emilyreff7 can you also add a release note

[emilyreff7]

@emilyreff7 can you also add a release note

Done!

[icexelloss]

LGTM

[jreback]

thanks @emilyreff7