Skip to content

max_rows seems to have max value of 60 #3541

Closed
hayd opened this Issue May 7, 2013 · 18 comments

6 participants

@hayd
Python for Data member
hayd commented May 7, 2013

Max_rows seems to have a maximum value of 60 (in 0.11 and dev, this works as expected in 0.10.1):

In [1]: %paste
from pandas import *
n = 100
foo = DataFrame(index=range(n))
foo['floats'] = np.random.randn(n)

set_option('display.max_rows', 101)

## -- End pasted text --

In [2]: foo.head(100)
Out[2]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 100 entries, 0 to 99
Columns: 1 entries, floats to floats
dtypes: float64(1)

In [3]: foo.head(61)
Out[3]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 61 entries, 0 to 60
Data columns (total 1 columns):
floats    61  non-null values
dtypes: float64(1)

In [4]: foo.head(60)
Out[4]:
      floats
0  -0.631717
1   0.231468
2  -0.465971
3  -1.427025
...
In [5]: %paste
set_option('display.max_rows', 10)

## -- End pasted text --

In [6]: foo.head(10)
Out[6]:
     floats
0 -0.194574
1 -0.130100
2  0.547474
3  0.669904
4  0.776946
5 -2.384084
6 -0.011147
7 -0.272660
8 -0.051326
9  1.757020

In [7]: foo.head(11)
Out[7]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 11 entries, 0 to 10
Data columns (total 1 columns):
floats    11  non-null values
dtypes: float64(1)
In [8]: version.version # same true for 0.11
Out[8]: '0.12.0.dev-d925966'

Essentially from http://stackoverflow.com/questions/16424493/pandas-setting-no-of-max-rows.

Related: #3395, #3426.

@lodagro
Python for Data member
lodagro commented May 8, 2013

There are two limits that apply display.heigth and display.max_rows.

@cpcloud
Python for Data member
cpcloud commented May 10, 2013

@hayd do u want max_rows to override height?

@y-p
y-p commented May 10, 2013

The display logic has bells and it has whistles and admittedly it's not very simple anymore.

Basically, max_rows determines whether we need to somehow reformat the output
to look good on the screen, and height dictates the dimensions for which we are formatting.

I understand it's not as intuitive as it one might hope, but aside from better docs, I don't think
this can be altered.

marked as usage+docs, and we should add something to the documentation explaining this.

@hayd
Python for Data member
hayd commented May 10, 2013

@cpcloud I think there is a case for that, but provided it is clear in the docs (i.e. is mentioned explicitly in the max_rows part of the set_options docstring) I can live with it. :)

@hayd
Python for Data member
hayd commented May 20, 2013

http://stackoverflow.com/questions/16652651/pandas-df-head-error

I think it should overwrite the display.height...

@hayd
Python for Data member
hayd commented May 20, 2013

maybe there should be a UserWarning saying if max_rows is less than display_height?

@y-p
y-p commented May 20, 2013

Pretty sure it's the default value of display.width that's at play in that SO question.

@hayd, could you please test out #3657, and ask the SO question owner to do the same?
I'd like to know this solves his problem, and that he knows we care.

@y-p
y-p commented May 20, 2013

pushed 88d892a, can we close this?

@hayd
Python for Data member
hayd commented May 20, 2013

@y-p thanks for putting these together, the SO OP said he't test in the morning and seemed happy :)

I'm still a but sulky from the example at the top of this thread, but happy to close this is everyone else is ok with it.

@y-p
y-p commented May 20, 2013

Oh no! a sulky pandas user.

You have a fair point. There are now two options controlling roughly
the same thing, height was supposed to control the formatting of expand_repr
only. Need to rework the display logic again. sigh

@hayd
Python for Data member
hayd commented May 20, 2013

Sorry! ...I am concerned about the kittens, honest.

@y-p
y-p commented May 21, 2013

If #3663 sees gets enough testing, I'll merge it for 0.11.1.

@agconti
agconti commented May 21, 2013

Hi guys,

I'm the SO user from the above post. I believe I tested y-p's update correctly, I did so by forking y-p's pandas repo and then physically overwriting the appropriate files with his commits, but it has not solved my problem. I can link my IPython notebook if it helps. Please let me know I did not test out the fix correctly. Thanks for looking into the issue for me!

@y-p
y-p commented May 21, 2013

You shouldn't need to manually overwrite the files.
try:

git clone http://github.com/y-p/pandas
git co GH3541 # checkout the PR branch
sudo python setup.py install
ipython notebook

then, in ipnb, try running:

from pandas.util.testing import makeCustomDataframe as mkdf
mkdf(10,20) # generates a dataframe with 10 rows, and 20 cols

you should see HTML output, probbaly wider then the browser window.

IIUC, that's what you were after.

If you want to roll back, reinstall pandas in whatever way you prefer.
You can just checkout a version into a new branch and install that:

git checkout v0.11.0 -b v0.11.0 # or whatever version you prefer
sudo python setup.py install
@agconti
agconti commented May 22, 2013

Hi y-p,

Thank you for the walk-through on getting the pull request. I ran into a hitch though: I tried your steps several times but ran into this error when running the final code in an ipnb.

Code:

from pandas.util.testing import makeCustomDataframe as mkdf
mkdf(10,20) # generates a dataframe with 10 rows, and 20 cols

Error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-a622f6364bf3> in <module>()
----> 1 from pandas.util.testing import makeCustomDataframe as mkdf
      2 mkdf(10,20) # generates a dataframe with 10 rows, and 20 cols

C:\Users\Andrew\Documents\Kaggle\pandas_git\pandas\pandas\__init__.py in <module>()
      4 
      5 try:
----> 6     from . import hashtable, tslib, lib
      7 except Exception:  # pragma: no cover
      8     import sys

ImportError: cannot import name hashtable

cannot import name hashtable

I also received this error while running setup.py :

running build_ext
skipping 'pandas\index.c' Cython extension (up-to-date)
building 'pandas.index' extension
error: Unable to find vcvarsall.bat

At any rate, I was not able to verify the html output. Any tips on how I might successfully test your fix?

Thanks for your help and your time.

@y-p
y-p commented May 22, 2013

Oh, you're on windows. that does complicate matters, It can be tricky to setup a
windows build env. I understand now why You tried to simply overwrite the python files.
If you're using a 0.11.0 binray though, the results are unpredictable, since the
python may be out of sync with the compiled portions.

At least one other user besides me reported that the fix work, I plan to merge it for 0.11.1
which should be released soon.

As an interim solution, I think you can work around the problem by setting display.width to a larger value.

@agconti
agconti commented May 22, 2013

Thanks y-p. I'll try playing with display.width until 0.11.1 comes out; thanks for the help in the meantime.

@wesm
Python for Data member
wesm commented May 25, 2013

Merged in #3663 so closing this. thanks guys

@wesm wesm closed this May 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.