-
-
Notifications
You must be signed in to change notification settings - Fork 31.6k
Stack overflow in itertools.chain.from_iterable. #74128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
itertools.chain.from_iterable (somewhat ironically) uses recursion to resolve the next iterator, which means it can run out of the C stack when there's a long run of empty iterables. This is most obvious when building with low optimisation modes, or with Py_DEBUG enabled: Python 3.7.0a0 (heads/master:c431854a09, Mar 29 2017, 10:03:50)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> next(itertools.chain.from_iterable(() for unused in range(10000000)))
Segmentation fault (core dumped) |
This looks fine. Feel free to apply and to backport this to earlier versions. I would have guessed that the C compiler would have automatically removed the tail recursion, but your experience would indicate otherwise. |
I think it probably does, unless optimisation is turned off: I'm unable to reproduce except in debug builds of Python. |
FWIW, we ran into this in real-world cases (Youtube, I think), when we switched from using a pre-built Python interpreter to one built from source using the same optimisation and debug levels as we use for all other C/C++ code. Even so, the accompanying test really does fail in pydebug mode ;-P I'll backport to 3.6, 3.5 and 2.7. |
Possible workaround: use chain.from_iterable(filter(None, iterables)) instead of chain.from_iterable(iterables). But this works only when iterables are collections, not iterators. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: