List, tuple constants with more than about 350 entries lead to "RuntimeError: pystack exhausted"

[vahanm01]

Hello,

With Raspberry Pi Pico: installed firmware using UF2 is Adafruit CircuitPython 6.2.0-beta.4 on 2021-03-18; Raspberry Pi Pico with rp2040.

In Thonny, when trying to create a ULAB array that is 1024 length, I receive "RuntimeError: pystack exhausted".

import ulab test_array=ulab.array([0, 0.00000943077, 0.0000377227, ... 0.00000943077, 0])

I did not want to include the large array here, but its 1024 length of float64...basically a hanning window array.
I also tried just values 1-1024 int and it gave the same error.

I am unsure how to adjust the pystack value with a UF2 setup and Windows 10. I cannot find any core files.
I apologize if this is a limitation of Circuitpython and/or RP2040, but I can't imagine an array of that length is pretty much a bottleneck...at least I hope not.

A workaround I found was to save the hanning values into a text file, import them into a list, and then convert them to a ulab array. Also if I create a loop at 1024 and populated a list by appending to it, there are no issues. It seems that having the raw values in a script for the interpreter to determine causes the pystack issue.

Thank you.

[deshipu]

Perhaps this will help? https://circuitpython.readthedocs.io/en/6.1.x/shared-bindings/supervisor/index.html#supervisor.set_next_stack_limit

[jepler]

Can you verify that you've correctly shown the source code you are executing? The code you showed gives a different error, because you cannot assign to 'ulab.array':

>>> import ulab
>>> test_array=ulab.array=([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object cannot assign attribute 'array'

To create an array filled with 1024 zeros of a given type, you should instead write

>>> test_array = ulab.zeros(1024, dtype=ulab.float)
>>> test_array
array([0.0, 0.0, 0.0, ..., 0.0, 0.0, 0.0], dtype=float32)

These lines worked for me on a board with the RP2040 microcontroller:

Adafruit CircuitPython 6.2.0-beta.4 on 2021-03-18; Adafruit Feather RP2040 with rp2040
>>> import ulab
>>> test_array = ulab.zeros(1024, dtype=ulab.float)
>>> test_array
array([0.0, 0.0, 0.0, ..., 0.0, 0.0, 0.0], dtype=float32)

Please note that float64 is not supported in any CircuitPython board.

[tannewt]

pystack exhausted isn't impacted by set_next_stack_limit. The stack limit is the C stack (which used to include python data too.) It is usually hit by deep python recursion. Generally, we may want to increase the size a bit.

[vahanm01]

@jepler my apologies.

Yes when creating an empty array it gives that error. I updated my post to show some sample values I am loading into the array. I basically created a numpy hanning window of 1024 values on my personal machine, then copied it over to Thonny connecting to RP2040; I am trying to perform FFT using ULAB (can't quite figure out if there is a way to get frequencies yet but the transforms work). I need to save these static values for use in my calculations and cannot simply create a random array. Unless if there is a numpy like function to retrieve hanning values... np.hanning(1024)

Any direct array creation that is about 200 plus records gives me the pystack error regardless if it is an integer or float. I essentially created a text file of these values, read them in line by line attaching to a new array; that works perfectly fine. I think that is sufficient for me but if it is of any value, it may be a good idea to add a fix for this if users want to create large arrays directly.

import ulab test_array=ulab.array=([0, 0.00000943077, 0.0000377227, ... 0.00000943077, 0])

[jepler]

When creating a literal sequence, the amount of pystack used is proportional to the size of the sequence.

>>> eval(str(list(range(367))))
[0, 1, ¸, 365, 366]
>>> eval(str(list(range(368))))
RuntimeError: pystack exhausted

To allow a list of 1024 numbers to appear in a literal list or tuple value, even if there are no other function calls, imports, etc., involved, it appears that pystack would need to be enlarged by about 2700 bytes (to somewhat over 4200 bytes total) compared to its current value of 1536.

Placing the data in a file compiled with mpy_cross doesn't seem to help.

Some suggestions, based on converting the data to float32 in binary form (e.g., with array_as_bytes = bytes(memoryview(np.hanning(1024).astype(np.float32))):

• Place the data in a file, and read it similar to

  with open("hanning.dat", "rb") as f:
      test_array = ulab.frombuffer(f.read(4*1024))

• Place the data inline in a binary string, and use test_array = ulab.frombuffer(b'\x00\x00\x00\x00\xd88\x1e7w8\x1e8\…') -- however, such large strings may confuse common text editors.

@tannewt Is "needs the ability to specify an array of 1024 float values inline, in a module directly imported from code.py or the repl" a concrete enough story, and one we want to support? If so, it points to increasing the pystack to somewhere around 4200 bytes. If not, hopefully a different technique like one of the ones I suggest above will work for our original reporter.

[jepler]

[I have updated the title and changed the text of the initial comment where it said 'ulab.array=([...])' but intended 'ulab.array([...])'. Another comment makes a similar error, which I did not correct]

[Neradoc]

I encountered a question with a situation like that in #help-with-circuitpython, and assuming (probably wrongly) that the cause was that the literal array was created on the stack before being put in a variable on the heap (or something like that), I suggested cutting it into parts of 300 items length (approx).

long_list = [0, 1, 2 ... 1000]
# pystack exhausted

long_list = [0, ... 300,]
long_list += [301, ... 600]
long_list += [601, ... 900]
long_list += [901, ... 1000]
# works

long_list = [0, ... 300,] + [301, ... 600] + [601, ... 900] + [901, ... 1000]
# works too

[tannewt]

@tannewt Is "needs the ability to specify an array of 1024 float values inline, in a module directly imported from code.py or the repl" a concrete enough story, and one we want to support? If so, it points to increasing the pystack to somewhere around 4200 bytes. If not, hopefully a different technique like one of the ones I suggest above will work for our original reporter.

I'm totally open to increasing the pystack size. We just need to remember that it's a tradeoff with heap size.

The non-pystack version of code will allocate large python stack frames onto the heap instead of the stack. Perhaps that's really what we need, allocate these structures on the heap instead of the stack.

[vahanm01]

Thank you all for the support. I think the suggestions are great and even just importing from a file is sufficient for me. Now, I cannot speak for the average user and if there is enough use case to create large arrays of static values, then it may be worth it without compromising a lot of performance.

Thanks again for looking into this, I leave the rest to you great developers!