GOTCHA: You can't use dot notation to add a column to a DataFrame

[AllenDowney]

I ran into a nasty gotcha I have not seen mentioned in the documentation. In general, you can refer to DataFrame columns like a dictionary key or like an object attribute. But the behavior is inconsistent. In particular, you can modify an existing column using dot notation, but you can't add a new column.

The following example demonstrates.

d = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),
     'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
df = pd.DataFrame(d)

# this adds a column to df
df['three'] = df.two + 1

# this successfully modifies an existing column
df.one += 1

# this seems to work but does not add a column to df
df.four = df.two + 2

print df.three
print df.four
print df

Is this behavior unavoidable?

[jreback]

see docs here: http://pandas-docs.github.io/pandas-docs-travis/indexing.html#attribute-access
I guess an additional warning to that effect could be added.

This whole attribute access bizness if a bit of a hybrid, e.g. it WILL work if the column already exists, but not for a completely new column.

The reason is that it is VERY error prone; e.g. very easy to overwite an existing method (though this could raise). Furthermore people often want to simply assign an attribute e.g. df.name = 'foo' or whatever, so very tricky to figure out what is needed (e.g. is that an additional column?)

IMHO the assignment should be completely banned and only allow read-only access, but I may be in a minority view.

dupe of this issue: #6160

[jreback]

actually....going to reopen this issue; I think what you are doing should raise, e.g. assigning a pandas object to a non-existant column

[AllenDowney]

Thanks for considering the issue. The documentation you linked says that
"Setting is allowed as well", but does not distinguish modifying an
existing column from creating a new one (unless I am not reading carefully
enough).

On Mon, May 19, 2014 at 11:52 AM, jreback notifications@github.com wrote:

Reopened #7175 #7175.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7175#event-122520592
.

[jreback]

yes I think that is an oversite. would you like to do a PR to put in a warning about the setting part? (e.g. that can only set an existing column), as this won't be fixed till next release at the earliest (docs can put in now).

[AllenDowney]

Sure, but I don't know where the source lives for the docs.

On Mon, May 19, 2014 at 12:07 PM, jreback notifications@github.com wrote:

yes I think that is an oversite. would you like to do a PR to put in a
warning about the setting part? (e.g. that can only set an existing
column), as this won't be fixed till next release at the earliest (docs can
put in now).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7175#issuecomment-43523616
.

[jreback]

https://github.com/pydata/pandas/blob/master/doc/source/indexing.rst

[AllenDowney]

PR submitted

On Tue, May 20, 2014 at 2:25 PM, jreback notifications@github.com wrote:

pandas/docs/source/indexing.rst

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7175#issuecomment-43664480
.

[onesandzeroes]

@jreback Just came across this on StackOverflow with this question. This seems like pretty weird/surprising behaviour to me, is there any good reason why this shouldn't raise or at least warn? I might try changing this to a raise and see how many tests it breaks.

[AllenDowney]

I would argue for a warning rather than an exception. It is not illegal to
attach a new attribute to a DataFrame (although it is probably not great
style).

Allen

On Mon, Sep 22, 2014 at 3:09 AM, onesandzeroes notifications@github.com
wrote:

@jreback https://github.com/jreback Just came across this on
StackOverflow with this question
http://stackoverflow.com/questions/25968118/how-to-delete-an-item-from-a-pandas-dataframe.
This seems like pretty weird/surprising behaviour to me, is there any good
reason why this shouldn't raise or at least warn? I might try changing
this to a raise and see how many tests it breaks.

—
Reply to this email directly or view it on GitHub
#7175 (comment).

[jreback]

I think could deal with this in several ways:

• add an option (mode.attribute_assignment).that by default warns (but could alsoe None or raise) to control this: that way everyone/noone is happy!
• invoke the option if a method is attempting to be overwritten (and if you actually DO want to monkey-patch you would then simply do this in a context manager to remove the warning), so that doing df.max = 'foo' will warn
• still it remains ambiguous whether df.Age = 'foo' is meant for df['Age'] = 'foo' or really an attribute assignement. I think you could allow the former, then need a way to do the latter
  (like maybe df.set_attribute('Age','foo')

[AllenDowney]

I think there are three cases to consider for

df.foo = bar

1. If foo is the name of an existing column

Current behavior: replace the contents of the column with bar
Desired behavior: same

2. If foo is the name of a DataFrame method

Current behavior: clobbers the existing method
Desired behavior: warning

3. If foo doesn't exist

Current behavior: adds an attribute to the DataFrame (not a column)
Desired behavior: warning

Does that sound right?

If so, then I think the warning message should give instructions for
suppressing the warning. Now there are two options for suppressing the
warning:

1. Provide an option. Pro: you can suppress once and never worry again.
   Con: you can suppress once and forget about it.

2. Require the programmer to catch the warning. Pro: You have to actively
   suppress every time. Con: You have to actively suppress every time.

Jeff, you suggested that there could be a way to set an attribute named foo
even if there is a column named foo. I assume you were kidding ;)

Allen

On Mon, Sep 22, 2014 at 9:33 AM, jreback notifications@github.com wrote:

I think could deal with this in several ways:

• add an option (mode.attribute_assignment).that by default warns (but
  could alsoe None or raise) to control this: that way everyone/noone is
  happy!
• invoke the option if a method is attempting to be overwritten (and
  if you actually DO want to monkey-patch you would then simply do this in a
  context manager to remove the warning), so that doing df.max = 'foo'
  will warn
• still it remains ambiguous whether df.Age = 'foo' is meant for df['Age']
  = 'foo' or really an attribute assignement. I think you could allow
  the former, then need a way to do the latter (like maybe
  df.set_attribute('Age','foo')

—
Reply to this email directly or view it on GitHub
#7175 (comment).

[jreback]

@AllenDowney good summary. pull-request?

I had meant that df.a = 'foo' could translate to df['a'] = 'foo', but then you would have to have a method df.set_attribute('a2','foo') that could then set an actual attribute. But on 2nd thought this is really a big change. So let's go with your summary.

I think an option for the warning is the best as it will happend EVERY time (unless you turn of the option entirely, and can be worked around by using pd.option_context to set the context

[jtratner]

+1 for warning. Also if you make the warning message generic (i.e., don't
include the attribute name), you'll only see the warning once per script,
because I believe the default is to suppress warnings that are generated
multiple times.

On Mon, Sep 22, 2014 at 7:28 AM, jreback notifications@github.com wrote:

@AllenDowney https://github.com/AllenDowney good summary. pull-request?

I had meant that df.a = 'foo' could translate to df['a'] = 'foo', but
then you would have to have a method df.set_attribute('a2','foo') that
could then set an actual attribute. But on 2nd thought this is really a big
change. So let's go with your summary.

I think an option for the warning is the best as it will happend EVERY
time (unless you turn of the option entirely, and can be worked around by
using pd.option_context to set the context

—
Reply to this email directly or view it on GitHub
#7175 (comment).

[jreback]

ahh, but I think we DO want this warning to keep happening!

[Zenadix]

Yes. This should trigger a warning.

[ghost]

+1 for a warning (pandas user here which just run into this issue).

[sebastienplat]

Has the warning been implemented ? The milestone says 0.18.2, but on 0.19.0 it doesn't seem to appear.

[jreback]

no, it says Next Major Release, and this issue is still open. pull-requests to implement are welcome.

[sebastienplat]

Ah ok. I would gladly give it a try, but I am not sure where to start looking :/

[jreback]

right here

easy enough to see if one is setting using a pandas object but the column is not defined, then could show a warning

[sebastienplat]

Thanks @jreback ! I will have a look and see what I can do.