RecursionError: maximum recursion depth exceeded during pulumi up

[piotr-bzdyl-vertexinc]

What happened?

pulumi up failed with:

error: Program failed with an unhandled exception:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/pulumi/3.43.1/x64/pulumi-language-python-exec", line 179, in <module>
    loop.run_until_complete(coro)
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/runner/work/taxamo-infrastructure-github/taxamo-infrastructure-github/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 126, in run_in_stack
    await run_pulumi_func(lambda: Stack(func))
  File "/home/runner/work/taxamo-infrastructure-github/taxamo-infrastructure-github/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
    await wait_for_rpcs()
  File "/home/runner/work/taxamo-infrastructure-github/taxamo-infrastructure-github/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 73, in wait_for_rpcs
    await RPC_MANAGER.rpcs.pop()
RecursionError: maximum recursion depth exceeded

Steps to reproduce

I'm not sure - it happens sometimes in a program managing >2k resources (most of them managed by Github and AWS providers). When I tried to check the actual number of the resources @ https://app.pulumi.com, it was really slow to render so maybe it's related to slowness/issues of https://app.pulumi.com.

Expected Behavior

No RecursionError.

Actual Behavior

RecursionError occurs.

Output of pulumi about

pulumi up is run on Github Actions: pulumi cli version 3.43.1.

From my local machine:

$ pulumi about
CLI          
Version      3.51.1
Go Version   go1.19.5
Go Compiler  gc

Plugins
NAME       VERSION
aws        5.17.0
github     5.0.0
python     unknown
terraform  5.6.7
tls        4.6.1

...

Dependencies:
NAME              VERSION
deepdiff          6.2.1
pip               22.2.2
pulumi-aws        5.17.0
pulumi-github     5.0.0
pulumi-terraform  5.6.7
pulumi-tls        4.6.1
pydantic          1.10.2
setuptools        63.4.3

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[dixler]

Hi. Thanks for filing this. I'll try reproducing this in python.

I'd suggest taking a look at the PULUMI_SKIP_CHECKPOINTS environment variable. It can improve performance of large stacks at the cost of not checkpointing on every operation that is performed. It should go much faster, but if the update fails, fixing the state may be more complicated.
blog

[Frassle]

Given this crashed in Python code I'd expect PULUMI_SKIP_CHECKPOINTS to have no effect on solving it.

I think the stacktrace in this case is mis-leading. RPC_MANAGER.rpcs is a list of awaitables (generally asyncio futures), and awaiting an awaitable that has failed re-throws the exception but with the stacktrace of the await statement, not the original.

We should probably see if that can be tweaked so that we can get stacktraces of where the recursion limit was actually broken.

[abhinav]

We should probably see if that can be tweaked so that we can get stacktraces of where the recursion limit was actually broken.

Trying to get a better stack trace here in #11911, but looking at the related code,
it seems that the original stack trace was also logged somewhere by RPC_MANAGER.
Access to that would help debug this issue.
If the logs aren't accessible, then we'll probably need #11911 to debug this.

[dixler]

Hi, @piotr-bzdyl-vertexinc we're cutting a release v3.52.1. If you upgrade your pulumi program's python SDK to that version when it's released, it should contain a better stack trace that you can post here.

[justinvp]

Hi @piotr-bzdyl-vertexinc, since we haven't heard back, I'm going to close this issue. If you are able to upgrade your program's Pulumi Python SDK and are still seeing an error, please open a new issue with the improved stack trace (or reopen this issue). Thanks!