ENH: #9847, adding a "same" and "expand" param to StringMethods.split()

[sreejata]

closes #9847

[TomAugspurger]

I haven't read through all the issues, but has expanding an Index to a MultiIndex (if needed) been proposed? Analogous to Series.str.* -> DataFrame

[jreback]

@TomAugspurger ohh, that would be nice, but let's make another issue about that.

[jorisvandenbossche]

Can you move this comment to the return_type explanation?

[jorisvandenbossche]

To explain: the first sentence is used in summary tables on the API pages, so this is not what we want to show there.

[shoyer]

Did the 'index' option ever make it into a release? I'm pretty sure it didn't, in which case we don't need to preserve it.

[jorisvandenbossche]

@TomAugspurger yes, I also thought about that! Looks like a very good idea to me, but indeed lets discuss in a separate issue (eg I think it should be in the main namespace, not in str)

[jorisvandenbossche]

Indeed, 'index' should not be mentioned.

[jorisvandenbossche]

I am still slightly in favor of having a expand=True/False instead of return_type='same'/'expand'.

Keeping return_type='series'/'frame' as backcompat has to happen in both cases, and it is only there for one release.

[shoyer]

Yes, expand=True/False seems like a slightly cleaner API than string options.

[mortada]

this is a minor point but it might be good idea to use a dict to organize the mapping from old to new arg names, at the beginning of the function, something like:

arg_map = {'series': 'same', 'index': 'same', 'frame': 'expand'}
if return_type in arg_map:
    warnings.warn("return_type='%s' is deprecated, please use '%s' instead" % (return_type, arg_map[return_type]), FutureWarning)
    return_type = arg_map[return_type]

this way the rest of the code doesn't have to contain references to the old names, making it easier to remove the deprecation stuff later.

[sreejata]

Should this be updated to use expand=True/False or keep as same/expand?

[jreback]

@sreejata can you update for the comments above?

[mortada]

I'm also slightly in favor of expand=True/False. It seems cleaner and more intuitive especially since we'd like to do a similar expansion for Index.str -> MultiIndex in v0.17.0

[jreback]

ok, let's go with expand=True/False then. This should be complmetely back-compat (use the deprecate decorator)

@sreejata can you update

[sinhrks]

I've implemented the expand wrapper in #9773. #9773 work as below

Invoked by	expand=False	expand=True (default)
Series	Series	DataFrame
Index	Index (of tuples)	MultiIndex

I think the points are

split (this PR):

• Can follow the above behavior simply replacing return_type to expand.
• Whether to change the defualt to True on this occasion.

extract (#9985)

• Expected Series and Index to work as the same.
• Adding extract option breaks current Series behavior which implicitly decide returning type.

get_dummies (#9985)

• Expected Series and Index to work as the same.
• Adding extract option breaks current Series behavior which always return DataFrame.

[jreback]

@sinhrks

this needs to handle the backward compat issue (by deprecating the existing return_type=series/frame). The index is new in 0.16.1, so no need to handle that.

[jreback]

@sinhrks can you split out the return_type changes (and incorporate the deprecation warning), from #9773 (as that is for partition, yes?). Then can close this PR.

[sinhrks]

@jreback #9773 the logic is implemented in _wrap_result_expand, and it can be used in str.split as an alternative of _wrap_result. I think changes required in this PR are:

• adding deprecate_kwargs to str_split
• remove return_type logic from str_split (https://github.com/pydata/pandas/pull/9870/files#diff-4627140a96de4a6390883e22e983f1b2R675)
• Change str.split to use _wrap_result_expand rather than _wrap_result.

[jreback]

@sinhrks are you able to put in the deprecation of the existing return_type logic? (in favor or expand)?

[sinhrks]

@sreejata Can you update above?

@jreback If this needs in v0.16.1, I'll work on it.

[jreback]

ideally this could be fixed in next day or 2

[jreback]

replaced by #10085