ValueError: threshold must be numeric and non-NAN, try sys.maxsize for untruncated representation

[gowthamcr]

Issue is observed while runing Candle P1B3 benchmark

File "p1b3_baseline_keras2.py", line 26, in
import p1b3 as benchmark
File "/Pilot1/P1B3/p1b3.py", line 37, in
np.set_printoptions(threshold=np.nan)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/arrayprint.py", line 246, in set_printoptions
floatmode, legacy)
File "/usr/local/lib/python3.5/dist-packages/numpy/core/arrayprint.py", line 93, in _make_options_dict
raise ValueError("threshold must be numeric and non-NAN, try "
ValueError: threshold must be numeric and non-NAN, try sys.maxsize for untruncated representation

[seberg]

Hmmm, the largest value that works for me is (and not sys.maxsize which the issue suggests?):

In [30]: np.set_printoptions(2**31-1)                                                                                   

In [31]: np.arange(10).astype(np.float64)                                                                               
Out[31]: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

in any case, that benchmark should probably just use a reasonable value for the threshold. The question is whether we should nevertheless go back to a warning.

[eric-wieser]

threshold=np.nan was never intended to be supported, and is a bug in your code.

This was introduced in #12353

[eric-wieser]

@seberg: can you show the output when you pass sys.maxsize?

[seberg]

Sorry, sure, may look at it myself (on the other hand, I do not know printing super well), this is on 1.16.1, sys.maxsize is 2**63 - 1, maybe an int/long/ssize_t confusion somewhere:

In [1]: np.set_printoptions(2**31)                                                                                      
In [2]: np.arange(10).astype(np.float64)

OverflowError: signed integer is greater than maximum

Full Error below:

In [1]: np.set_printoptions(2**31)                                                                                      
In [2]: np.arange(10).astype(np.float64)                                                                                
Out[2]: ---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
/usr/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

/usr/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    400                         if cls is not object \
    401                                 and callable(cls.__dict__.get('__repr__')):
--> 402                             return _repr_pprint(obj, self, cycle)
    403 
    404             return _default_pprint(obj, self, cycle)

/usr/lib/python3.7/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    695     """A pprint that just redirects to the normal repr function."""
    696     # Find newlines and replace them with p.break_()
--> 697     output = repr(obj)
    698     for idx,output_line in enumerate(output.splitlines()):
    699         if idx:

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in _array_repr_implementation(arr, max_line_width, precision, suppress_small, array2string)
   1417     elif arr.size > 0 or arr.shape == (0,):
   1418         lst = array2string(arr, max_line_width, precision, suppress_small,
-> 1419                            ', ', prefix, suffix=suffix)
   1420     else:  # show zero-length shape unless it is (0,)
   1421         lst = "[], shape=%s" % (repr(arr.shape),)

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in array2string(a, max_line_width, precision, suppress_small, separator, prefix, style, formatter, threshold, edgeitems, sign, floatmode, suffix, **kwarg)
    688         return "[]"
    689 
--> 690     return _array2string(a, options, separator, prefix)
    691 
    692 

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in wrapper(self, *args, **kwargs)
    468             repr_running.add(key)
    469             try:
--> 470                 return f(self, *args, **kwargs)
    471             finally:
    472                 repr_running.discard(key)

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in _array2string(a, options, separator, prefix)
    494 
    495     # find the right formatting function for the array
--> 496     format_function = _get_format_function(data, **options)
    497 
    498     # skip over "["

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in _get_format_function(data, **options)
    427             return formatdict['longfloat']()
    428         else:
--> 429             return formatdict['float']()
    430     elif issubclass(dtypeobj, _nt.complexfloating):
    431         if issubclass(dtypeobj, _nt.clongfloat):

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in <lambda>()
    366         'int': lambda: IntegerFormat(data),
    367         'float': lambda:
--> 368             FloatingFormat(data, prec, fmode, supp, sign, legacy=legacy),
    369         'longfloat': lambda:
    370             FloatingFormat(data, prec, fmode, supp, sign, legacy=legacy),

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in __init__(self, data, precision, floatmode, suppress_small, sign, **kwarg)
    857         self.large_exponent = False
    858 
--> 859         self.fillFormat(data)
    860 
    861     def fillFormat(self, data):

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in fillFormat(self, data)
    914                                        sign=self.sign == '+')
    915                     for x in finite_vals)
--> 916             int_part, frac_part = zip(*(s.split('.') for s in strs))
    917             if self._legacy == '1.13':
    918                 self.pad_left = 1 + max(len(s.lstrip('-+')) for s in int_part)

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in <genexpr>(.0)
    914                                        sign=self.sign == '+')
    915                     for x in finite_vals)
--> 916             int_part, frac_part = zip(*(s.split('.') for s in strs))
    917             if self._legacy == '1.13':
    918                 self.pad_left = 1 + max(len(s.lstrip('-+')) for s in int_part)

/usr/lib/python3.7/site-packages/numpy/core/arrayprint.py in <genexpr>(.0)
    913                                        unique=unique, trim=trim,
    914                                        sign=self.sign == '+')
--> 915                     for x in finite_vals)
    916             int_part, frac_part = zip(*(s.split('.') for s in strs))
    917             if self._legacy == '1.13':

OverflowError: signed integer is greater than maximum

[gowthamcr]

with numpy version 1.15.4
python3
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
np.nan
nan
np.set_printoptions(threshold=np.nan)
import sys
sys.maxsize
9223372036854775807

maxsize value is similar with latest numpy also

[seberg]

Well, just checked the dragon4 code is limited to a precision of int32, the parsing uses C int. Maybe we should just ensure those limitations in set_printoptions...

[eric-wieser]

Threshold and precision are independent. Threshold is about omitting elements, not about omitting digits.

[seberg]

@eric-wieser, ooopst, somehow kept on that line. I guess that had nothing to do with the initial report and is much less reasonable to set to a large value in any case.

[eric-wieser]

@gowthamcr: Ignoring @seberg's confusion, this is intended behavior, so I'm closing this issue.

You should be using np.set_printoptions(threshold=sys.maxsize), the code you have was never how threshold was intended to be used.

[zbtong6799]

my English not so good.
Please import sys
then replace “np.set_printoptions(threshold=np.nan)“ with ”np.set_printoptions(threshold=sys.maxsize)”

[bharath5673]

replace “np.set_printoptions(threshold=np.nan)“ with ”np.set_printoptions(threshold=sys.maxsize)

it worked for me!

[kivok12]

@bharath5673 which version of python did you use? cause I have the same problem

[eric-wieser]

Locking this, there's enough discussion at https://stackoverflow.com/questions/1987694/how-to-print-the-full-numpy-array-without-truncation, we don't need more in our bug tracker