Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Series.rank(pct=True).max() != 1 for a large series of floats #18271

Closed
proinsias opened this issue Nov 13, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@proinsias
Copy link
Contributor

commented Nov 13, 2017

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

rs = np.random.RandomState(seed=0, )

len_df = 20000000
df = pd.DataFrame(data=rs.rand(len_df), columns=['abc'], ).sort_values('abc')

df['Rank'] = df['abc'].rank()
df['Rank_Pct']= df['abc'].rank(pct=True)
df['Rank_Pct_Manual'] = df['Rank']/len_df

df.describe()

Output:

	abc	        Rank	        Rank_Pct        Rank_Pct_Manual
count	2.000000e+07	2.000000e+07	2.000000e+07	2.000000e+07
mean	4.999223e-01	1.000000e+07	5.960465e-01	5.000000e-01
std	2.886891e-01	5.773503e+06	3.441276e-01	2.886751e-01
min	1.036192e-08	1.000000e+00	5.960464e-08	5.000000e-08
25%	2.498756e-01	5.000001e+06	2.980233e-01	2.500000e-01
50%	4.999781e-01	1.000000e+07	5.960465e-01	5.000000e-01
75%	7.499111e-01	1.500000e+07	8.940697e-01	7.500000e-01
max	1.000000e+00	2.000000e+07	1.192093e+00	1.000000e+00

Problem description

I have a set of 20 million floats, and I am trying to follow this StackOverflow example. This code discusses calculating the percentile ranking of a column using either the pct=True option for the rank() function, or by manually dividing the output of rank(pct=True) by the length of the Series.

I noticed that the former values have a maximum that is not 1, while the latter have the expected maximum of 1.

I have tried this with the latest (0.21.0) version of pandas, and can replicate it with an array of random floats.

It seems to be related to the number of rows being greater than 2^23 – you can see this by comparing the output when len_df=16770000 and len_df=16780000.

I believe the responsible code is pandas/_libs/algos.pyx/rank_1d_float64().

I'm working on a PR now.

Expected Output

I would expect the values of Rank_Pct and Rank_Pct_Manual to be the same, and that the maximum of both should be 1.

Output of pd.show_versions()

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.3.0
Cython: None
numpy: 1.13.3
scipy: 0.19.1
pyarrow: 0.7.1
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: 1.1.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

proinsias added a commit to proinsias/pandas that referenced this issue Nov 14, 2017

BUG: Use float64 for row counter in rank() GH18271
As reported in pandas-dev#18271, for a large `Series` `s` of `float`s,
`Series.rank(pct=True).max()` may not be `<=1` as expected.

This is due to the use of a `float` for the row counter. This updates
the counter to use a `float64`.

@proinsias proinsias referenced this issue Nov 14, 2017

Closed

BUG: Use float64 for row counter in rank() #18274

4 of 4 tasks complete

proinsias added a commit to proinsias/pandas that referenced this issue Nov 14, 2017

BUG: Use float64 for row counter in rank() GH18271
As reported in pandas-dev#18271, for a large `Series` `s` of `float`s,
`Series.rank(pct=True).max()` may not be `<=1` as expected.

This is due to the use of a `float` for the row counter. This updates
the counter to use a `float64`.

@jreback jreback added the Numeric label Nov 14, 2017

proinsias added a commit to proinsias/pandas that referenced this issue Nov 14, 2017

BUG: Use float64 for row counter in rank() GH18271
As reported in pandas-dev#18271, for a large `Series` `s` of `float`s,
`Series.rank(pct=True).max()` may not be `<=1` as expected.

This is due to the use of a `float` for the row counter. This updates
the counter to use a `float64`.
@jreback

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2017

xref to #15630.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.