get_dummies docs

[hayd]

Should put this in the online docs and elaborate / add an example to the docstring.

[hayd]

will include with pr for #4446

[hayd]

I'm tempted to keep this one open (it's the api rst, but not in reshaping). Should expand on it really, perhaps will consult wes' book (where get_dummies is mentioned).

[jreback]

@hayd for 0.13?

[hayd]

@jreback When is 0.13?

[cpcloud]

i'll spam the dev ML and ask

[jreback]

I think that the sql refactor should drive the timeline
how that coming?

[hayd]

@jreback If that is the main thing we're holding back 0.13 perhaps we should push sql back to 0.14... I think it would be sensible to have sql merged into master/tested for a while and better test coverage before releasing new sql anyways... atm @jtratner is hacking apart my refactor...

[jreback]

@hayd This all depends on you and @jtratner deciding best option. I would say:

• leave existing code, introduce new code in new module as experimental
• make new code default and move existing to legacy
• use a fallback with new code engine='legacy', to call existing
• use just new code
• wait till 0.13.1

I don't think 0.13 will be ready for release candidate for say 2 weeks minimum, but could easily do 1 month or even more. I think its worthwhile to include an improved version of SQL, and am +1 on using SQLAlchemy

[jtratner]

@hayd can you just clarify how you expect to get an engine for SQLAlchemy? Once you get that to work, can just pass it to the class it should be fine.

At least personally, I think there are so many moving parts here that we may not get it together for 0.13 (in particular, the code to generate a connection from passed in parameters is really complicated and doesn't need to be). I propose we put this in as an experimental feature in a new module with a very small API (i.e., two methods, write_frame and read_sql and that's it), we can call it sql_experimental. Then when we eventually move the new sql code out of experimental, we can leave the aliases there.

To simplify for 0.13 ONLY, I propose that this experimental module:

1. Not support any driver but SQLAlchemy. (flavor = 'sqla')
2. Not support direct usage of the class.
3. Not support using tquery or uquery directly.

Then, for 0.14, we can replace the existing code in io/sql, move the legacy code to sql_legacy, and leave the legacy code in place for the tquery and uquery globals.

That way we can put something together and kick the tires. As soon as 0.13 is released, we can settle on a public API for classes and how that all should work out.

[jreback]

@hayd docs on this?

[jreback]

@hayd ping?

[hayd]

Opening wes' book on get_dummies page...

[jreback]

@hayd doc?

[jreback]

@hayd docs?

[hayd]

Sorry for needing so much pinging! I've shameless yoinked one example from wes' book and appended to reshape docs. I wanted to add a final example... but maybe I shouldn't.

Wes discusses doing manual get_dummies with delimited strings (of movie categories), by creating an empty DataFrame and filling it. I was going to suggest the following instead, but maybe this discussion is (for now) more of a cookbook example:

In [1]: s = pd.Series(['ab', 'bc', 'c', 'abc']).apply(list)

In [2]: s
Out[2]: 
0       [a, b]
1       [b, c]
2          [c]
3    [a, b, c]
dtype: object

In [3]: s.apply(lambda x: pd.get_dummies(x).sum()).fillna(0)
Out[3]: 
   a  b  c
0  1  1  0
1  0  1  1
2  0  0  1
3  1  1  1

Maybe this would even be more useful as a Series/str method (or is it already?).

[jreback]

there is an issue IIRC, but can't find it right now....about implemented that behavior directly
as there is a fair amount of overhead doing it via apply

[jreback]

here it is....put it on for 0.14....not hard to do this (and could be cythonized) ... #3695

[jreback]

maybe could be something ilke

s.get_dummies() ?

which could just call pd.get_dummies(split=True) or something

[hayd]

I forgot your neat lambda x: Series(1, x) trick! :)

I suppose kinda weird get_dummies isn't already a Series method as it is.

Maybe should take a delimiter, or are thinking split to work in the (shudders) case of a Series of lists?

[jreback]

thanks you sir!