#394: Use Union[List[Column], List[str]] for Select

[jhereth]

Passing a List[str] to select raises a mypy warning, similar for List[Column]. We change the type from List[Union[Column, str]] to Union[List[Column], List[str]].

Fixes #394 .

[zero323]

Thanks for you work @qudade.

Unfortunately, this is not going to work. In fact Spark is very specific about types here, and the only support collection is list.

Sequence would type check for all kind of different types, like

>>> isinstance(range(10), Sequence)                                                                                                                                                      
True
>>> isinstance((), Sequence)                                                                                                                                                             
True

which are clearly not supported on runtime.

If you want to address this particular scenario you should use Union[List[Column], List[str]], i.e.

 def select(self, __cols: Union[List[Column], List[str]]) -> DataFrame: ...

though it should probably cover other cases with similar signature.

[zero323]

That looks much better. Would you mind adding data tests for this?

[jhereth]

@zero323 sure, I'll try to add some tests next week.

Is there some information how to run the tests? (simply calling pytest didn't seem to do the trick)

[zero323]

@zero323 sure, I'll try to add some tests next week.

Thanks.

Is there some information how to run the tests? (simply calling pytest didn't seem to do the trick)

You'll have to add annotations directory to your MYPYPATH and set MYPY_TEST_PREFIX, i.e.

pyspark-stubs/.travis.yml

Line 17 in c3e949c

- MYPY_TEST_PREFIX=$PWD MYPYPATH=$PWD/third_party/3 pytest

[jhereth]

@zero323 Tests have been added. Code was rebased vs master and squashed.

Tests are passing (except for #397)

[zero323]

Merged to master, thank you.