Closed
Description
Feature Type
Adding new functionality to pandas
Changing existing functionality in pandas
Removing existing functionality in pandas
Problem Description
df.join() does not retain the attrs of the dataset. Given that the attrs manual states that "many operations that create new datasets will retain attrs", this seems like an omission.
Feature Description
Join is different from concat because there is a clear dataframe that the operation is on. Therefore, it would seem natural if df.join() would retain the attrs of the initial dataframe.
Alternative Solutions
It would also be possible to make the attrs dependent on "how" but this would only be natural for "left" and "right".
Additional Context
No response
Activity
timhoffm commentedon Nov 18, 2024
This should be handled the same way as
concat
: Propagate only if all inputs have the same attrs.concat
is currently a hard-coded special case,pandas/pandas/core/generic.py
Line 6056 in 7fe270c
but we may want to delegate the attrs combination back to the operation instead, i.e.
Note that
other
is the "combination object" for there calls, i.e._MergeOperation
,_Concatenator
etc, which will have to grow the logic for combining attrs.Alternatively, one could leave the combination logic in
__finalize__
but provide a uniform interface on all "combination objects" to give their inputs. Currently, thats non-uniform_Concatenator.objs
, but_MergeOperation.left
/_MergeOperation.right
.rhshadrach commentedon Nov 18, 2024
Thanks for the report! @timhoffm - can you post a reproducible example.
BUG: Copy attrs on pd.merge()
timhoffm commentedon Nov 19, 2024
#60357 should fix this. I've choosen the somewhat smaller refactoring and not pushed the combination logic back into the "combination objects". In fact #59141 removed
_Concatenator
in favor of simple functions. Therefore, I've now choosen the common API to be "provides the inputs viainput_objs
parameter".Note that
join()
is implemented viaconcat()
ormerge()
depending on the case. I've only added explicit tests for these fundamental operations, not forjoin()
, but could add that if desired.BUG: Copy attrs on pd.merge()
BUG: Copy attrs on pd.merge()
BUG: Copy attrs on pd.merge()
BUG: Copy attrs on pd.merge()
rommeswi commentedon Nov 20, 2024
I disagree because concat is a symmetric operation operating on a list of dfs but join is an operation that joins other data to a specific dataframe. The properties of the initial dataframe should then be maintained unless pandas is instructed otherwise. It's as if you start dropping rows from your dataframe after a join just because the added data has some NaNs.
timhoffm commentedon Nov 20, 2024
I recommend to be defensive on all operations that have multiple DataFrames as input. The inputs may have contradicting Information in their attrs and joining them can lead to misinterpretation. Say,
df1.attrs['date'] = "1.1.2024"; df2.attrs['date'] = "1.1.2025"
then, having one of the dates put on the join is not fair.Instead, I propose to implement a
set_attrs()
helper as proposed in #52166 (comment) to enable easy explicit handling of attrs by the user.rommeswi commentedon Nov 20, 2024
Something like set_attrs() saves half a line of boilerplate code. But the main cost is cognitive: users constantly need to chase after the attrs to make sure they are copied. If the manual says that most operations copy them, I would expect any operation to copy those attrs that look like a reasonable default. Otherwise I expect the manual to say that attrs are usually dropped unless pandas is absolutely sure that the user wants to maintain the attrs.
I would even argue that on a concat the most sensible thing to do would be to merge the attrs by field and throw a warning in case of conflicting fields.
timhoffm commentedon Nov 20, 2024
We can possibly improve the docs.
Concerning the behavior, there are two aspects stemming from the experimental state of the feature
This has lead to the suggestion to deprecate
attrs
(#52166). I've argued against the deprecation because I believe attrs are valuable, but the topic is still an experimental and incomplete feature. That's also the reason for the vague "most operations copy" wording.My personal take (as a non-pandas developer but advocate for attrs) on the intended behavior is to be very defensive and minimal:
DataFrame -> DataFrame
should copy the attrs[DataFrame, DataFrame, ...] -> DataFrame
should copy the attrs only if all inputs are identical. - In the face of ambiguity, refuse the temptation to guess.rhshadrach commentedon Nov 21, 2024
I'm supportive of @timhoffm's proposal. Users can also override this behavior via subclassing.
Perhaps that's not helpful in all cases, but hopefully is in many.
BUG: Copy attrs on pd.merge() (#60357)