Using `reprlib.repr` fails for very large numbers

[MarcellPerger1]

Bug report

Bug description:

The bug

reprlib.Repr.repr_int fails if the integer exceeds the string conversion limit (on line 184)

cpython/Lib/reprlib.py

Lines 183 to 189 in 6eb6c5d

	def repr_int(self, x, level):
	s = builtins.repr(x) # XXX Hope this isn't too slow...
	if len(s) > self.maxlong:
	i = max(0, (self.maxlong-3)//2)
	j = max(0, self.maxlong-3-i)
	s = s[:i] + self.fillvalue + s[len(s)-j:]
	return s

This impacts other parts of the standard library which use reprlib.Repr. A quick code search reveals that this affects/could affect:

• pydoc
• bdb
• idlelib & IDLE (debugger-related things)
• asyncio

See #135487 (comment) for detailed rationale for why IMHO this is a bug.

pydoc example

Given the following code (in a file called temp11.py),

a = 1 << 19000

running pydoc on it like python -m pydoc '.\temp11.py' gives the following error:

Stack trace

Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "C:\Python313\Lib\pydoc.py", line 2846, in 
    cli()
    ~~~^^
  File "C:\Python313\Lib\pydoc.py", line 2807, in cli
    help.help(arg, is_cli=True)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 2062, in help
    else: doc(request, 'Help on %s:', output=self._output, is_cli=is_cli)
          ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 1773, in doc
    pager(render_doc(thing, title, forceload), f'Help on {what!s}')
          ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 1758, in render_doc
    return title % desc + '\n\n' + renderer.document(object, name)
                                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 543, in document
    if inspect.ismodule(object): return self.docmodule(*args)
                                        ~~~~~~~~~~~~~~^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 1375, in docmodule
    contents.append(self.docother(value, key, name, maxlen=70))
                    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 1642, in docother
    repr = self.repr(object)
  File "C:\Python313\Lib\reprlib.py", line 71, in repr
    return self.repr1(x, self.maxlevel)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\pydoc.py", line 1234, in repr1
    return getattr(self, methodname)(x, level)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Python313\Lib\reprlib.py", line 184, in repr_int
    s = builtins.repr(x) # XXX Hope this isn't too slow...
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

Possible fix

The underlying issue is that the size of the integer isn't checked before calling builtin repr.
A way to fix this may be to display it as hexadecimal over a certain threshold (then truncate the middle like usual). (IMHO it's not worth it to add a special method to compute the first and last n digits of a number without calculating the middle ones)

CPython versions tested on:

3.13

Operating systems tested on:

Windows

Linked PRs

• gh-135487: fix reprlib.Repr.repr_int when given very large integers #135506

[picnixz]

I wouldn't treat it as a bug since the fix would to increase the limit:

use sys.set_int_max_str_digits() to increase the limit

There are many other places where whenever we display some integer value, we could hit that limit.
Nevertheless, we can catch the ValueError. We can easily compute the first digit and the last few digits (we need a naive log10 for the first digit and last few can be done with moduli). We could also have a sentinel value saying "too large value" (like, <integer exceeding XXX digits>).

cc @serhiy-storchaka

[MarcellPerger1]

I wouldn't treat it as a bug since the fix would to increase the limit

pydoc

IHMO the pydoc CLI still shouldn't raise an exception just because there's a very large integer in the file provided - personally, I would expect pydoc to 'just work'. Additionally, I don't think the pydoc HTML viewer should refuse to show me the rest of the contents if there's a large integer constant in it.

help() function

Also, as help() is based on reprlib, calling help on a module containing a large int will give the user a large error message, leaving them to wonder what they've done wrong - giving an error message when help is called on a module is not very beginner-friendly.

IDLE / idlelib

The IDLE debugger seems to not work properly when it steps into a module with a large integer as a global (and it's set to display globals). It seems to stop evaluating anything in that module.
For example, if temp11.py contains

a = 1 << 16000
b = 6  # Doesn't show this in Globals


def f():
    print('f() called')  # Doesn't print this

It doesn't print f() called, doesn't show any of the global variables defined after it, and seems to stop execution of the function:

This is caused by idlelib incorrectly assuming that reprlib doesn't raise an error (

cpython/Lib/idlelib/debugger.py

Line 569 in 6eb6c5d

svalue = self.repr.repr(value) # repr(value)

). IHMO this is a bug (I don't think weird behavior of the debugger when a value in scope is too large is 'intended behaviour')

Conclusion

Lastly, reprlib seems to try to not raise any exceptions (see the except Exception in repr_instance) and I think it would be more logical to keep it this way.
I often assume that reprlib.repr() will always work and won't raise an exception. The standard library code also seems to support this assumption - there doesn't seem to be an except around reprlib.repr() anywhere in the standard library. Therefore, it is reasonable to think that a lot of code out there also makes this assumption (even though it's not currently correct) - surely it would be a good idea to make this assumption true again.
Special cases aren't special enough to break the rules - I believe this also applies to reprlib.repr() not raising exceptions.

(P.S. sorry for the wall of text)

[picnixz]

The idlelib issue is, I believe, a real issue though. But I don't know if it's a fixable issue and how easy it can be (wrapping self.repr.repr() seems a possible way to do it). cc @terryjreedy

To solve all issues in one go, I think that we should just display a sentinel value or the first and last digits with some ... inbetween (without knowing the exact number of digits that were omitted). We can compute a naive log10. So, I think we can render it as <integer exceeding XXX digits>.

The downside is that it won't be parsable. In general a repr can be parsed back by eval (at least for some types, including built-ins), but not all types follow that convention. Here we're dealing with integers so it's a bit tricky... So... I'd rather not introduce more possible outputs and leave it that way in reprlib.

Now, if we were to add except ValueError, it also means that we're also hiding other ValueErrors that could be important =/

[serhiy-storchaka]

math.log10() is fast for arbitrary integers.

[terryjreedy]

The bigint ValueError is a nuisance for IDLE, including in Shell even without the debugger. I though of def bigint_rept(n): return f"int(10**{log10(n)})". It returns a legal expression, but given that bigintrep(1<<16000) is int(10**4816.479930623699), eval() overflows. For IDLE, I am considering f10**round(log10(n)) # Approximately. and will open a separate issue.

[AndPuQing]

In the specific context of builtins.repr(huge_integer), a ValueError is almost always raised for one reason: the integer exceeds the system’s int_max_str_digits limit. See the relevant code here Objects/longobject.c and Objects/object.c.

In such cases, rendering a placeholder marker seems like a reasonable choice. For example, reprlib.repr([0] * 1000) produces '[0, 0, 0, 0, 0, 0, ...]', which is also not fully parsable.

At the same time, I think it's valuable to include information about the size of the large integer — for example:

<[sign] integer with ~{N} digits (exceeds limit of {L})>

# Estimate total digits efficiently
num_digits = int(abs(x).bit_length() * math.log10(2)) + 1

Knowing the approximate number of digits helps developers decide what to do next.
For instance:

• “Oh, it’s only 5,000 digits — maybe I can temporarily raise the system limit to inspect it.”
• vs. “Whoa, 5 million digits — my algorithm is definitely broken; I need to debug the logic, not the display.”

[picnixz]

Striclty speaking, the ... is parsable, though we wouldn't get the original value. But I see the point, in which case I'll open a PoC PR using numbits / math log10

[picnixz]

Note that we're actually doing this in repr_instance

        try:
            s = builtins.repr(x)
            # Bugs in x.__repr__() can cause arbitrary
            # exceptions -- then make up something
        except Exception:
            return '<%s instance at %#x>' % (x.__class__.__name__, id(x))

I'm therefore convinced that we can use a custom repr for large ints.