Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API/BUG: Handling Dtype Coercions in Series/Index #15832

Closed
gfyoung opened this issue Mar 29, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@gfyoung
Copy link
Member

commented Mar 29, 2017

Off master (bd169d):

# Case I: Overflow on int64
>>> Index([np.iinfo(np.uint64).max-1],dtype='int64')
...
OverflowError: Python int too large to convert to C long

# Case II: Coercion to uint64
>>> Index([-1], dtype='uint64')
UInt64Index([18446744073709551615], dtype='uint64')

# Case III: Ignoring coercion to int
>>> Index([1, 2, 3.5], dtype=int)
Float64Index([1.0, 2.0, 3.5], dtype='float64')

So we got some coercions that fail but others that work. Although all of these issues involve Index, the first two are also applicable to Series (in the last issue, it does coerce all elements to integer).

How should we handle failed coercions? Is the second case even a failure? Should iron out this.

xref #12758
xref #15187

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2017

I think our rules are pretty well established (FYI, we should actually document these).

If dtype=None (on Series/Index). then we find a best-fit dtype.
If dtype=..... Then we coerce to that dtype if its safe, otherwise should raise (a useful message)

Case I: Overflow on int64
Index([np.iinfo(np.uint64).max-1],dtype='int64')
...
OverflowError: Python int too large to convert to C long

I think this is ok (let's test it). Maybe a more user friendly message though.

Case II: Coercion to uint64
Index([-1], dtype='uint64')
UInt64Index([18446744073709551615], dtype='uint64')

This should raise

Case III: Ignoring coercion to int
Index([1, 2, 3.5], dtype=int)
Float64Index([1.0, 2.0, 3.5], dtype='float64')

This should raise

@ucals

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2017

Now that we were able to close 1 novice and 2 intermediate bugs, I'd like to try an advanced bug - this one :)

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2017

sure!

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 8, 2018

MAINT: More useful error msg on Index overflow
Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 8, 2018

MAINT: More useful error msg on Index overflow
Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 11, 2018

MAINT: More useful error msg on Index overflow
Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

jreback added a commit that referenced this issue Jun 12, 2018

MAINT: More friendly error msg on Index overflow (#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses gh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses gh-15823.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 13, 2018

API/BUG: Raise when int-dtype coercions fail
Related to the Index and Series constructors.

Closes pandas-devgh-15832.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 13, 2018

API/BUG: Raise when int-dtype coercions fail
Related to the Index and Series constructors.

Closes pandas-devgh-15832.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 13, 2018

API/BUG: Raise when int-dtype coercions fail
Related to the Index and Series constructors.

Closes pandas-devgh-15832.

@jreback jreback modified the milestones: Next Major Release, 0.24.0 Jun 13, 2018

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 13, 2018

API/BUG: Raise when int-dtype coercions fail
Related to the Index and Series constructors.

Closes pandas-devgh-15832.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 15, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 15, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 15, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

david-liu-brattle-1 added a commit to david-liu-brattle-1/pandas that referenced this issue Jun 18, 2018

MAINT: More friendly error msg on Index overflow (pandas-dev#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses pandas-devgh-15823.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 18, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 19, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

gfyoung added a commit to forking-repos/pandas that referenced this issue Jun 19, 2018

API/BUG: Raise when int-dtype coercions fail
* Related to the Index and Series constructors.

Closes pandas-devgh-15832.

* Add integer dtype fixtures to conftest.py

Can used for subsequent refactoring.

jreback added a commit that referenced this issue Jun 20, 2018

alimcmaster1 added a commit to alimcmaster1/pandas that referenced this issue Jun 28, 2018

MAINT: More friendly error msg on Index overflow (pandas-dev#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses pandas-devgh-15823.

alimcmaster1 added a commit to alimcmaster1/pandas that referenced this issue Jun 28, 2018

alimcmaster1 added a commit to alimcmaster1/pandas that referenced this issue Jun 28, 2018

MAINT: More friendly error msg on Index overflow (pandas-dev#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses pandas-devgh-15823.

alimcmaster1 added a commit to alimcmaster1/pandas that referenced this issue Jun 28, 2018

jorisvandenbossche added a commit that referenced this issue Jun 29, 2018

MAINT: More friendly error msg on Index overflow (#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses gh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses gh-15823.

(cherry picked from commit defdb34)

jorisvandenbossche added a commit that referenced this issue Jul 2, 2018

MAINT: More friendly error msg on Index overflow (#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses gh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses gh-15823.

(cherry picked from commit defdb34)

Sup3rGeo added a commit to Sup3rGeo/pandas that referenced this issue Oct 1, 2018

MAINT: More friendly error msg on Index overflow (pandas-dev#21377)
* MAINT: More useful error msg on Index overflow

Display a more friendly error message when there
is an OverflowError during Index construction.

Partially addresses pandas-devgh-15832.

* DOC: Clarify how Index.__new__ handles dtype

Partially addresses pandas-devgh-15823.

Sup3rGeo added a commit to Sup3rGeo/pandas that referenced this issue Oct 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.