lib.maybe_convert_objects will fail on uint64 values that exceed int64 max

[wesm]

xref #11440 for addtl tests

Observed in the wild. cc @blais

[jtratner]

@jreback @wesm objects that would pass this wouldn't be compatible with anything else but integers that are all > 0 right? Float, float128, complex, and longdouble all lose precision.

[jtratner]

I'm wrong, float128 (which I think is the same as longdouble) can work...

[cpcloud]

float128 doesn't have be long double... long double could be 64 bits... it's an implementation detail but it ends up being what you expect most of the time...

[cpcloud]

same applies to int64 etc ... e.g., long is 32 bits on 32 bit arch and 64 on 64-bit arch

[jtratner]

@cpcloud so what's the right dtype to contain something that's uint64 and greater than what int64 can handle? - this SO answer claims float128 is 'a mess'. http://stackoverflow.com/questions/9062562/what-is-the-internal-precision-of-numpy-float128

[jtratner]

i.e., just allow uint64_t size and go from there? and then disallow with anything that's not an actual integer > 0?

[cpcloud]

I'm not sure why this is happening ... uint64 should hold values up to 2 * INT_MAX... i think probably allowing uint64 is the way 2 go...not sure i follow the second question.

[jtratner]

@cpcloud in convert_objects, if you can't fit everything into the same container, then it doesn't work. This is why uint64 doesn't work:

        elif util.is_integer_object(val):
            seen_int = 1
            floats[i] = <float64_t> val
            complexes[i] = <double complex> val
            if not seen_null:
                try:
                    ints[i] = val
                except OverflowError:
                    seen_object = 1
                    break

[jtratner]

it's not hard to set this up, I just wanted to clarify I had the right idea...going to fix it now.

[jtratner]

@cpcloud what I mean by the second question is what should be returned from this:

import sys
arr = np.array([-5, sys.maxint + 5, 3], dtype=object)
lib.maybe_convert_objects(arr)

It should be object right? Otherwise the -5 becomes gobbdledygook.

[jtratner]

Well, this is mostly useless anyways, because BlockManager converts uint64 to object internally in form_block:

        elif issubclass(v.dtype.type, np.integer):
            if v.dtype == np.uint64:
                # HACK #2355 definite overflow
                if (v > 2 ** 63 - 1).any():
                    object_items.append((i, k, v))
                    continue
            int_items.append((i, k, v))

So need a unsigned int type or something in block manager

[jtratner]

Anyways, working version of lib.maybe_convert_objects here: https://github.com/jtratner/pandas/tree/GH4471_fix_uint64_maybe_convert_objects

[pwaller]

I keep hitting this while importing a dataset which has uint64's in it. Is there anything I can do to help it along, given that someone already made a patch but it didn't get in?

[jreback]

where's the patch?

[pwaller]

@jreback see @jtratner's comment above. Is the patch unsuitable or is it just that it wasn't shepherded into master?

[jreback]

that's 2 years old - if someone wants to cherry pick and present then can look

[DrRibosome]

also hitting this bug - just wondering if the fix is in progress, or if interest is simply too low

[jreback]

well need someone motivated to push a fix