Simplify has_parallel_type()

[madsbk]

This PR simplify the implementation of has_parallel_type() and make it easy to extend by down stream projects like rapidsai/dask-cuda#451

• Tests added / passed
• Passes black dask / flake8 dask

[quasiben]

I think this is ok. @mrocklin do you think this will adversely affect GeoPandas users ? My guess is no -- GeoPandas is the only the library (aside from cuDF) I am familiar with which extends dask dataframes

[rjzamora]

I definitley like your solution better. Do you think there is any need for a get_parallel_type test, or do you think the functionality is covered elsewhere?

[madsbk]

It should be covered in all the has_parallel_type() and new_dd_object() tests

[rjzamora]

...and make it easy to extend by down stream projects...

I like the simplification here, but can you explain how it helps down-stream projects? (Sorry if this is a silly question)

[madsbk]

I think this is ok. @mrocklin do you think this will adversely affect GeoPandas users ? My guess is no -- GeoPandas is the only the library (aside from cuDF) I am familiar with which extends dask dataframes

The semantic of get_parallel_type and has_parallel_type is exactly the same. Extending it using get_parallel_type.register() is also exactly the same.
The only potential breakage is the removal of parallel_types() but AFAIK this is a help function used by has_parallel_type exclusively. At least, searching through github I haven't find any projects using it (GeoPandas doesn't use it).
Having said that, it is easy enough to re-implement if someone is using it.

[madsbk]

I like the simplification here, but can you explain how it helps down-stream projects? (Sorry if this is a silly question)

My fault should have been more clear.
The current implementation makes it hard (if not impossible) for a wrapper class like the proxy object in Dask-CUDA to implement a dispatch handle of get_parallel_type(). The problem is that by registering the dispatch handle, parallel_types() will always include it as a parallel_type even if the instance being wrapped isn't a parallel type.
This is also why the current implementation of parallel_types() has to exclude get_parallel_type_object explicitly.

PS: The device memory spilling in Dask-CUDA makes use of this PR here: rapidsai/dask-cuda@8b977b7

[madsbk]

@mrocklin @jrbourbeau, if any of you have time to review this, it would be great. It is currently blocking rapidsai/dask-cuda#451

[jrbourbeau]

Thanks @madsbk! Apologies for the delayed review

[madsbk]

Thanks @madsbk! Apologies for the delayed review

No worries :)