TypeError: unsupported format string passed to numpy.ndarray.__format__

[hugovk]

Reproducing code example:

import numpy as np
x, y = np.array([-969100.0]), np.array([-4457000.0])

# This works
"%.4g, %.4g" % (x, y)

# This errors
"{:.4g}, {:.4g}".format(x, y)

Error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to numpy.ndarray.__format__

NumPy/Python version information:

1.15.4 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)]

OS: macOS High Sierra and Linux (Travis CI)

This happened when upgrading a codebase using the pyupgrade tool. %g and {:g} work the same for builtin types. Is this a NumPy bug?

[asottile]

I applied a small patch to cpython to try and get it to print what that format string is -- it appears to be garbage (?)

diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index ed2d40064a..eb1d85e295 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -4453,8 +4453,8 @@ object_format(PyObject *self, PyObject *args)
        should reject format specifications */
     if (PyUnicode_GET_LENGTH(format_spec) > 0) {
         PyErr_Format(PyExc_TypeError,
-                     "unsupported format string passed to %.200s.__format__",
-                     self->ob_type->tp_name);
+                     "unsupported format string passed to %.200s.__format__: '%s'",
+                     self->ob_type->tp_name, PyUnicode_AsUTF8String(format_spec));
         return NULL;
     }
     self_as_str = PyObject_Str(self);

$ python -c 'print("{:e}".format(__import__("numpy").array([1.])))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: unsupported format string passed to numpy.ndarray.__format__: '�'

hexdumped:

$ python -c 'print("{:e}".format(__import__("numpy").array([1.])))' |& hd -c
00000000  54 72 61 63 65 62 61 63  6b 20 28 6d 6f 73 74 20  |Traceback (most |
0000000   T   r   a   c   e   b   a   c   k       (   m   o   s   t    
00000010  72 65 63 65 6e 74 20 63  61 6c 6c 20 6c 61 73 74  |recent call last|
0000010   r   e   c   e   n   t       c   a   l   l       l   a   s   t
00000020  29 3a 0a 20 20 46 69 6c  65 20 22 3c 73 74 72 69  |):.  File "<stri|
0000020   )   :  \n           F   i   l   e       "   <   s   t   r   i
00000030  6e 67 3e 22 2c 20 6c 69  6e 65 20 31 2c 20 69 6e  |ng>", line 1, in|
0000030   n   g   >   "   ,       l   i   n   e       1   ,       i   n
00000040  20 3c 6d 6f 64 75 6c 65  3e 0a 54 79 70 65 45 72  | <module>.TypeEr|
0000040       <   m   o   d   u   l   e   >  \n   T   y   p   e   E   r
00000050  72 6f 72 3a 20 75 6e 73  75 70 70 6f 72 74 65 64  |ror: unsupported|
0000050   r   o   r   :       u   n   s   u   p   p   o   r   t   e   d
00000060  20 66 6f 72 6d 61 74 20  73 74 72 69 6e 67 20 70  | format string p|
0000060       f   o   r   m   a   t       s   t   r   i   n   g       p
00000070  61 73 73 65 64 20 74 6f  20 6e 75 6d 70 79 2e 6e  |assed to numpy.n|
0000070   a   s   s   e   d       t   o       n   u   m   p   y   .   n
00000080  64 61 72 72 61 79 2e 5f  5f 66 6f 72 6d 61 74 5f  |darray.__format_|
0000080   d   a   r   r   a   y   .   _   _   f   o   r   m   a   t   _
00000090  5f 3a 20 27 02 27 0a                              |_: '.'.|
0000090   _   :       ' 002   '  \n                                    
0000097

[asottile]

oh 🤦‍♂️, my printing is wrong -- that's the PyBytes object -- I meant to format its contents

[asottile]

The format string is being passed through properly it seems.

I think the fix would happen here -- it would need similar C code to the code for __float__ here

[asottile]

looks related to #5543

[billtubbs]

I stumbled on this issue when printing a pandas data frame with a poorly formed index.

Hope it can be solved as this could be a common mistake and the traceback message does not provide much clue as to what went wrong (other than it was in NumPy not Pandas).

Steps to reproduce:

index = pd.Index(np.array([[0], [1]]))  # Mistake! - array is wrong shape
df = pd.DataFrame({'A': [0, 1], 'B': [1.1, 1.2]}, index=index)
print(df)  # raises TypeError: unsupported format string passed to numpy.ndarray.__format__

[seberg]

@billtubbs I think you should probably move that to pandas, I am not sure d (which is an integer) can be a valid format string for numpy arrays, so the issue is that Pandas accept something that they should not be accepting.

[billtubbs]

Ah. Okay thanks. Then I will do that.

[billtubbs]

Actually, I just noticed that this error is now being caught in the latest version of Pandas (1.0.0) when the index is created. Sorry for unnecessarily raising it here.

[asottile]

@charris I believe the original issue is still a thing ?

[ndevenish]

I just hit this through pyupgrade also. The fact that %.4f-formatting works but f-strings/.format doesn't, seems like a genuine bug.

[cod3licious]

@charris Yes, I also think this this issue should be reopened; the behaviour seems really inconsistent:

print("hello %.2f" % np.array([1.38362]))     # works
print("hello %.2f" % np.array([[1.38362]]))   # works
print(f"hello {np.array([1.38362]):.2f}")     # TypeError

It would be really nice to have the last think working as well!

[eric-wieser]

I think this issue is closed because it is a duplicate of #5543

There is no reason to expect that % and f-string formatting be consistent. One uses inflexible rules built into python itself, while the other is a customization point that we can actually hook into - here's an example where they differ in plain python:

>>> from decimal import Decimal
>>> d = Decimal("1.234")
>>> f"{d:E}"
'1.234E+0'
>>> "%E" % d
'1.234000E+00'