docs(python): Rewrite pivot and melt docstrings

[Wainberg]

pivot and merge are among the more complicated dataframe manipulation functions, so I thought I'd improve the documentation on them.

[Wainberg]

@stinodego love to get your thoughts on this one. I guess this can be the first "chunk" of the documentation rewrite strategy we were envisioning.

[Wainberg]

@stinodego I wonder if another core developer might prefer to take a look at this and future documentation PRs?

[MarcoGorelli]

i'll take a look, but it would be good to get #14048 in first

[Wainberg]

Thanks so much @MarcoGorelli! Feel free to edit my edits to reflect your new pivot behavior - you should have commit privileges on this branch.

[stinodego]

There is good information in here, but it's much too verbose for a docstring. Docstrings should be a one-line summary, maybe two or three sentences of extended summary, and then some examples with a one-sentence explanation.

We actually have a melt and pivot section in the user guide, and there currently is not much information in there. The information would be a welcome addition there.

So I would propose to update the user guide sections and keep only the most essential information the docstring. I can have another look then.

[Wainberg]

@stinodego Good idea! I moved most of the info to the user guide. I also updated my changes to account for @MarcoGorelli's fixes in #14048. Note that my changes to the user guide will need to be updated for 1.0 to account for #14491.

[Wainberg]

@MarcoGorelli looks good? Once you're happy with it, I'll update to account for #14477 and fix the merge conflict.

[MarcoGorelli]

Overall, I think this is an improvement - in particular, the examples you provide for pivot make for easier teaching material than the current ones. Nice one @Wainberg 🙌 ! If you fetch and merge upstream/main then I'll check all the details

[Wainberg]

@MarcoGorelli thanks so much! ready for you to take a look :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.00%. Comparing base (baacf3d) to head (1ad84bf).
Report is 29 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13693      +/-   ##
==========================================
+ Coverage   80.94%   81.00%   +0.05%     
==========================================
  Files        1327     1331       +4     
  Lines      172081   172786     +705     
  Branches     2453     2456       +3     
==========================================
+ Hits       139290   139964     +674     
- Misses      32320    32354      +34     
+ Partials      471      468       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MarcoGorelli]

I've only really got 3 minor comments, it's clear you've spent quite some time looking into pivot and melt. LMK if you don't have bandwidth to address them

Thanks for doing this, it'll help users!

[MarcoGorelli]

I think this is only true if there's a single index column? Which is fine, you explain what happens with multiple index columns below, I just think it needs caveating here

[MarcoGorelli]

Maybe "Notes" rather than "Warnings"?

[MarcoGorelli]

does group_by render correctly here? you may need to use

:meth:`DataFrame.group_by`

?

[Wainberg]

Yes,

:func:`group_by`

works - see how https://github.com/pola-rs/polars/blob/py-0.20.13/py-polars/polars/dataframe/frame.py#L7164 renders correctly at https://docs.pola.rs/py-polars/html/reference/dataframe/api/polars.DataFrame.fill_nan.html.

[Wainberg]

Thanks so much @MarcoGorelli! I incorporated your suggestions.

I'm glad you appreciate the amount of work that went into this. I put a similar amount of care into #13402; it would be great if at least some of these changes could be incorporated at some point.

[MarcoGorelli]

I've made some little fixes, as there were some things which still needed sorting out

In particular, the last example from pivot didn't render

With the regards to the other PR - may I offer some unsolicited advice?

• as you can see, reviews generally take a lot of effort. It's easy to miss things. That's why you're being asked to only focus on a single topic per PR, reviewing 74 docs changes in one go is practically impossible
• I'd really advise against tagging so many people in one message...that would only really be OK if you were reporting an absolutely critical issue which simply couldn't wait...
• I'd suggest merging conflicts before asking for reviews, else content can get out of date (and another review will be needed after you've merged conflicts anyway)

Having said that, I really appreciate the work you've done here!

[Wainberg]

@MarcoGorelli thanks for the edits!

Re: the other PR, it would probably be best at this point if the core developers take that PR as a general guideline for things that could be improved in the docs, and manually incorporate whichever changes they like on their own timeframe.