IterationLimit, higher RecursionLimit

[poke1024]

No description provided.

[sn6uv]

We need to be careful with sys.setrecursionlimit:

The highest possible limit is platform-dependent. A user may need to set the limit higher when they have a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.

Also, how is performance when when the recursion depth gets significantly higher?

[poke1024]

Performance shouldn't be a problem, since this just sets a maximum stack size limit. The platform-dependent part is tricky though. One has to check that sys.setrecursionlimit does not overflow the OS thread stack size, which is hard to do without estimating the Python stack frame size. Will think about this.

Ideally, WL recursions shouldn't be related to Python recursions, but that's what they are, currently, and it's not simple to change. We won't be able to support MMA's Infinite recursions in this way, but we should at least try to allow a large number.

The rationale is obviously that, to my large surprise, MMA does not seem to support any tail recursion optimizations at all (each Module counts as one recursion), and so getting into the 1000s of recursions is really simple with a real calculation involving a couple of functions with modules calling each other (I ran into this with a simple example, so I opened this PR).

[poke1024]

There's now MATHICS_MAX_RECURSION_DEPTH (an environment variable), which overrides the hard coded maximum of 512 recursions (more specifically, it specifies the maximum value $RecursionLimit may be set to). If not set, things work just as before (max recursion limit of 512, no special code run). If it's set, however, the thread stack size is now adjusted in run_with_timeout_and_stack to accomodate for as much recursions as needed.

This way, you can now boost $RecursionLimit to anything provided your OS and your memory allow for it.

The calculation of the exact needed stack size in bytes is black magic (esp. the python_stack_frame_size now hard coded to 512); it can depend on the Python implementation as well as on the machine architecture, and it should be a maximum covering all sensible environments. 512 works on OS X for all tests I did, and it seems reasonable to me (512 bytes for one Python stack call seems pretty large actually, but 256 is too small on OS X). I found no docs on the exact numbers needed here (but then I didn't search really thoroughly and esp. not in Python's C source, which would be a start).