ERR: DataFrame can get into unprintable state

[chris-b1]

I know my replication case looks very strange, I had a typo assigning the wrong thing to index=, but the way the error is reported out of this made it very confusing to debug, possibly can do something better:

Code Sample, a copy-pastable example if possible

dct = {'a': pd.DataFrame({'a':[1,2,3]}), 'b': [1, 2, 3], 'c': 5}

df = pd.DataFrame({'a': range(3)}, index=dct.values())

df  # calling repr
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\core\formatters.py in __call__(self, obj)
    309             method = get_real_method(obj, self.print_method)
    310             if method is not None:
--> 311                 return method()
    312             return None
    313         else:

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\frame.py in _repr_html_(self)
    608 
    609             return self.to_html(max_rows=max_rows, max_cols=max_cols,
--> 610                                 show_dimensions=show_dimensions, notebook=True)
    611         else:
    612             return None

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\frame.py in to_html(self, buf, columns, col_space, header, index, na_rep, formatters, float_format, sparsify, index_names, justify, bold_rows, classes, escape, max_rows, max_cols, show_dimensions, notebook, decimal, border)
   1606                                            decimal=decimal)
   1607         # TODO: a generic formatter wld b in DataFrameFormatter
-> 1608         formatter.to_html(classes=classes, notebook=notebook, border=border)
   1609 
   1610         if buf is None:

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\format.py in to_html(self, classes, notebook, border)
    698                                       border=border)
    699         if hasattr(self.buf, 'write'):
--> 700             html_renderer.write_result(self.buf)
    701         elif isinstance(self.buf, compat.string_types):
    702             with open(self.buf, 'w') as f:

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\format.py in write_result(self, buf)
   1022         indent += self.indent_delta
   1023         indent = self._write_header(indent)
-> 1024         indent = self._write_body(indent)
   1025 
   1026         self.write('</table>', indent)

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\format.py in _write_body(self, indent)
   1181                 self._write_hierarchical_rows(fmt_values, indent)
   1182             else:
-> 1183                 self._write_regular_rows(fmt_values, indent)
   1184         else:
   1185             for i in range(len(self.frame)):

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\format.py in _write_regular_rows(self, fmt_values, indent)
   1203             index_values = self.fmt.tr_frame.index.map(fmt)
   1204         else:
-> 1205             index_values = self.fmt.tr_frame.index.format()
   1206 
   1207         row = []

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\indexes\base.py in format(self, name, formatter, **kwargs)
   1596             return header + list(self.map(formatter))
   1597 
-> 1598         return self._format_with_header(header, **kwargs)
   1599 
   1600     def _format_with_header(self, header, na_rep='NaN', **kwargs):

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\indexes\base.py in _format_with_header(self, header, na_rep, **kwargs)
   1610         if is_object_dtype(values.dtype):
   1611             result = [pprint_thing(x, escape_chars=('\t', '\r', '\n'))
-> 1612                       for x in values]
   1613 
   1614             # could have nans

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\indexes\base.py in <listcomp>(.0)
   1610         if is_object_dtype(values.dtype):
   1611             result = [pprint_thing(x, escape_chars=('\t', '\r', '\n'))
-> 1612                       for x in values]
   1613 
   1614             # could have nans

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\printing.py in pprint_thing(thing, _nest_lvl, escape_chars, default_escapes, quote_strings, max_seq_items)
    218         result = _pprint_seq(thing, _nest_lvl, escape_chars=escape_chars,
    219                              quote_strings=quote_strings,
--> 220                              max_seq_items=max_seq_items)
    221     elif isinstance(thing, compat.string_types) and quote_strings:
    222         if compat.PY3:

C:\Users\chris.bartak\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\formats\printing.py in _pprint_seq(seq, _nest_lvl, max_seq_items, **kwds)
    115     for i in range(min(nitems, len(seq))):  # handle sets, no slicing
    116         r.append(pprint_thing(
--> 117             next(s), _nest_lvl + 1, max_seq_items=max_seq_items, **kwds))
    118     body = ", ".join(r)
    119 

StopIteration: 

pandas 0.19.1

[has2k1]

Same problem (the stack traces match)

good_df = pd.DataFrame({'three': [3]})
bad_df = pd.DataFrame({'three': [3, 3]})

df1 = pd.DataFrame({
    'x': [1, 2, 3],
    'y': [1, '2', [good_df]]
})

df2 = pd.DataFrame({
    'x': [1, 2, 3],
    'y': [1, '2', [bad_df]]
})

# df1 can be printed
# df2 raises a StopIteration exception when printed

[rickymal]

I have the same error

[jobh]

I also have this problem. Basically, it means that Series or DataFrames with embedded dict-of-DataFrames cannot be printed — they are either printed with wrong value (case 1 below) or raises an error (case 2 below). I think priority should be raised. My reproducer:

In [1]: df=pd.DataFrame([3]); pd.Series([{'key':df}])
Out[1]: 
0    {'key': [0]}
dtype: object

In [2]: df=pd.DataFrame([3,3]); pd.Series([{'key':df}])
Out[2]: ---------------------------------------------------------------------------
[...]
StopIteration: 

[jobh]

Note: list-of-DataFrames have the same problem as dict-of-DataFrames, which links this to the original report.

In [4]: df=pd.DataFrame([3]); pd.Series([[df]])
Out[4]: 
0    [[0]]
dtype: object

In [5]: df=pd.DataFrame([3,3]); pd.Series([[df]])
Out[5]: ---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)

[jobh]

And finally (sorry for the noise) — I see that the prio-low label was already removed. So that's already good :)

[jreback]

anyone one in the community can submit a PR - this is how issues are solved

there is no priority was there are no resources to allocate - pandas is all volunteer

[jobh]

I understand that @jreback, it was just that I (mistakenly) thought it was tagged as "Prio-low" which felt wrong.

edit: to clarify, felt wrong because it would discourage volunteers from working on it, not wrong because someone should be forced to prioritize it.

[cliffckerr]

Just hit this bug myself with code like this:

import pandas as pd

dict1 = {'a': [1,2]}
dict2 = {'b': pd.DataFrame(dict1)}
dict3 = {'c': [dict2]}

df2 = pd.DataFrame(dict3)
print(df2)

It's come up on StackOverflow a few times, e.g. here and here.