RuntimeError: maximum recursion depth exceeded while calling a Python object #37646
Comments
|
@himadrisingh001 just a couple questions:
In the meantime I have a setup with your configs and i'll wait to see if I see the same behavior. |
We see around 270+ times in a day. Somehow its not printing the timestamp. We dont see high cpu usage too often though since the udpate.
High cpu usage might be unrelated but salt-minion was taking up the most of the cpu. |
|
Ran it with debug logging |
|
@himadrisingh001 still am having a hard time replicating this. But can you clarify a couple things as well:
|
|
|
Thanks for clarifying Do you have any scheduled jobs that are running? |
|
yes we do have one |
|
ping @rallytime or @cachedout i'm pretty stumped on what could be causing this. do either of you have any ideas on this one? I can't seem to replicate it and not sure of any further follow up questions to ensure we get @himadrisingh001 up and running. |
|
@Ch3LL @himadrisingh001 I think I have a reproducer for this issue. I'm not seeing high CPU load, but I have a very similar RuntimeError about recursion depth exceeded in the same section of Try this state: Perform a highstate:
schedule.present:
- function: state.highstate
- job_kwargs:
test: True
- when: 1:55pmbut modify the
I'm not sure if this is related, but if you just modify the ----------
ID: Perform a highstate
Function: schedule.present
Result: True
Comment: Modifying job Perform a highstate in schedule
Started: 14:14:48.576346
Duration: 25.37 ms
Changes:
----------
diff:
---
+++
@@ -1,7 +1,7 @@
enabled:True
function:state.highstate
jid_include:True
-kwargs:{'test': True}
+kwargs:OrderedDict([('test', True)])
maxrunning:1
name:Perform a scheduled highstate
-when:2:19pm
+when:2:18pmso it looks like something funny is happening with dicts vs Here is my version report: Let me know if I can provide any more info! |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue. |
|
I'm able to see this consistently on all versions of salt from 2016.11 to latest RC if the message to be serialized contains a circular reference. Other encoders like json.dumps are able to detect this and raise an exception that accurately describes the problem, msgpack crashes with the error described above. |
|
@Ch3LL or @sagetherage this needs to be re-opened, please. |
|
Thank you for updating this issue. It is no longer marked as stale. |
Ch3LL
added
severity-high
Ch3LL
added
severity-critical
|
@thusoy do you have some specific states that can reproduce this? I'm trying with this: I'm not getting the max recursion depth, but I am seeing pretty repeatedly |
|
Try to run a state like this: #!py
def run():
array = []
array.append(array)
return {
'test-circular-reference': {
'file.managed': [
{'name': '/tmp/foo'},
{'source': 'salt://foo.txt'},
{'template': 'jinja'},
{'context': {
'array': array,
}},
]
}
} |
|
@thusoy yep, that'll do it. I'll see what I can figure out here 👍 |
|
@waynew The fix from #51322 seems weird to me, I don't think anyone would want the value to be silently converted to a string, an explicit exception raised describing the recursion is probably preferable. The other thing I noticed while reading the fix is that the recursion detection seems flawed, any repeated object in the value would be detected as a recursion. Ie repeated = 'repeated'
data = [repeated, repeated]Would also be detected as recursion if the |
|
Note that this isn't likely to happen while using yaml states, but more complicated states written in python could easily trip on this. |
|
I'm not sure that there's an reason necessarily to prohibit recursion - and we're never actually losing information, unless something is depending on a self-referential cycle to be present. It's possible that a recursive structure indicates a deeper failure, but that's not clear to me right now. You are right that we're not handling the case where we have multiple of the same immutable object in a list. It may cause a problem due to an implementation detail, like this: I'll go ahead and put in a PR to fix that, but without some evidence to the contrary, I'm going to leave the recursive str-ifying in there. |
Description of Issue/Question
Running salt-minion on the master runs into following recursion and eats up the cpu, making master unresponsive.
Setup
Steps to Reproduce Issue
Its randomly happening so dont have exact steps to reproduce this. But its not uncommon.
Versions Report
The text was updated successfully, but these errors were encountered: