Bug: SettingWithCopyWarning in some cases cites wrong or no lines when showing where an error occurred.

[Khris777]

Changing the topic with some editing:

I stumbled over SettingWithCopyWarning outputting wrong lines when referencing where an error occurred.

Example:

import pandas as pd
L1 = ['A','A','A','A','B','B','B','B']
L2 = ['a','b','c','d','a','b','c','d']
L3 = [1,2,3,4,5,6,7,8]
df0 = pd.DataFrame({"L1":L1,"L2":L2,"L3":L3})

def doStuff1(df0):
    df0 = df0[["L2"]]
    df0.loc['L2'] = 'z'
    return df0

def doStuff2(df0):
    df0 = df0[["L1","L2"]]
    df0.loc[df0.L1 == 'A','L2'] = 'x'
    return df0

df1 = doStuff1(df0)
df2 = doStuff2(df0)

This code throws three SettingWithCopyWarnings. If I have this code in a file without anything else, this is my output:

E:/Temp/test.py:16: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df0.loc['L2'] = 'z'
C:\Users\My.Name\AppData\Local\Continuum\Anaconda2\envs\Python3\lib\site-packages\pandas\core\indexing.py:179: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)
E:/Temp/test.py:21: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df0.loc[df0.L1 == 'A','L2'] = 'x'

I'm getting three warnings, first and third do cite the correct lines.

But if I have the same code in another file where it is between lots of other code that is commented, then I'm getting this:

E:/Entwicklung/SVN/trunk/Sources/Misc/temp.py:70: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df0.loc[df0.L1 == 'A','L2'] = 'x'
C:\Users\My.Name\AppData\Local\Continuum\Anaconda2\envs\Python3\lib\site-packages\pandas\core\indexing.py:179: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)
E:/Entwicklung/SVN/trunk/Sources/Misc/temp.py:75: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  #print(df0.groupby("L2")["L1"].apply(lambda x: '_'.join(x)))

Only by being in the other file the lines that are cited are wrong. The first warning cites the line where the third error occurs, the third warning shows a commented line that appears after the code.

I can not properly reproduce this behaviour either. The behaviour and output seem to depend on if the code is executed the first time in a console or not.

As an example, I can execute the exact same code in two different consoles with different histories, and one does output the same as the first output, while the other outputs this:

E:/Temp/test.py:70: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
C:\Users\My.Name\AppData\Local\Continuum\Anaconda2\envs\Python3\lib\site-packages\pandas\core\indexing.py:179: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)
E:/Temp/test.py:75: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

Here it does reference the correct lines but does not cite them at all.

Python 3.6.1
Pandas 0.20.3

[gfyoung]

@Khris777 : Hmmm...this is very strange. It's also difficult to check given the nature of Python's interpreter to silence warnings, especially after they have been raised. I also can't tell why it might be warning with references to commented lines OR no references at all.

Given the difficulty of reproduction, I wonder whether we might be able to tackle this, but let's give it a shot. Can you post to this issue, two Python files (or two code-blocks, whichever is easier to read):

1. The original, uncommented code
2. The commented code

In addition, if you could provide step-by-step instructions for reproducing the warnings (all of them), that would be great!

[Khris777]

If I execute the code with all the stuff commented out that yesterday threw the wrong citations (example 2) in a new console today it behaves correctly. It doesn't seem to be reproduceable like that.

But one other thing I forgot yesterday:
In the second case where it cited that unrelated commented line it always did cite that specific line, even when I added other commented lines before that one. Maybe that's another hint.

I also forgot to say that I'm using IPython 6.1.0. Running this code in the standard Python console by hand throws only two warnings and no reports on where the errors occur (IPython feature?), and only on the first run, not on successive runs.

And only if I run it manually, if I execute this code as a file in the standard console, the console becomes unresponsive and stops working.

[gfyoung]

@Khris777 : Without code and steps to reproduce, it's a little difficult to follow everything that you're seeing on your end. See my comment above

[Khris777]

The uncommented code is in the OP post, I don't want to post the one with the comments because it's from a file I just use to test various things, so it's over 450 lines of clutter and different kind of methods and tests that I do not want to post publicly.

And like I said, I can't even reproduce the problem I had yesterday today with the same file.

[gfyoung]

Okay, very well then. I will close this for now, but if the problem rears its head again, feel free to ping this conversation again. Thanks!

[Khris777]

Okay, I will. Maybe I'll find more clues myself.