Salt-master does not process batch calls correctly after receiving too much data #55184
Comments
|
@keagan-thinkst Thank you for reporting this issue. Seems to be a duplicate to #55180 |
frogunder
added
Duplicate
|
This bug is actually a duplicate of #54884 and is related to some outrageous defaults set by the version of msgpack being used. This bug is heavily discussed in #53230, so there's more info there too. @keagan-thinkst, try fiddling around with the msgpack version as per these other issues. The msgpack module had some absurd constraints imposed in order to attempt to prevent denial-of-service attacks. (Not trying to stop on your toes or anything, frogunder) |
|
Hi @arizvisa Thanks for your suggestion, however this does not seem to fix my problem. I pinned the msgpack version on my salt stack to 0.5.6 and still get the same behaviour where the salt master does not process the batch calls correctly after I run my module function. The module function returns a large dictionary back from the minions. |
|
Ok then.. Well in that case, the only other option is literally to modify the calls to |
|
Thank you very much @arizvisa !!! I set I also set my msgpack version back to Is this possibly going to be a configurable value to set in an upcoming version of salt so that the fix for the problem is easier to do than hardcoding/patching salt files? |
|
It's up to the maintainers really. Maybe @frogunder would know better. I have a number of PRs that are waiting to get merged by their team, so I'm hesitant to create fixes for these other bugs until they get around to merging those. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue. |
|
As prior mentioned, this issue is a duplicate of #54884 and should be closed as a result. |
|
Thank you for updating this issue. It is no longer marked as stale. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue. |
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes saltstack#53230, fixes saltstack#54884, fixes saltstack#55180, fixes saltstack#55184, fixes saltstack#57026, fixes saltstack#57921
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes saltstack#53230, fixes saltstack#54884, fixes saltstack#55180, fixes saltstack#55184, fixes saltstack#57026, fixes saltstack#57921
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes saltstack#53230, fixes saltstack#54884, fixes saltstack#55180, fixes saltstack#55184, fixes saltstack#57026, fixes saltstack#57921
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes #53230, fixes #54884, fixes #55180, fixes #55184, fixes #57026, fixes #57921
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes #53230, fixes #54884, fixes #55180, fixes #55184, fixes #57026, fixes #57921
Description of Issue
Whenever I run a salt module function against my minions that returns too much data, I get the following error in the salt master's logs:
[salt.transport.ipc:195 ][ERROR ][28934] Exception occurred while handling stream: 11844764 exceeds max_str_len(1048576)Once this error has been seen, then any call from the master to a batch of minions fails on a random set of minions that seem to not reply back to the master. I test it with a simple
test.pingwith the-ldebugflag set.Each time I run
test.pingagainst * a different set of minions fail.From the masters side the debug lines state that the master is not receiving any info from the minion, it retries the query once, then after that fails again it times out and then the call on the master finishes with output stating that the call failed against a random set of minions.
When I look into the salt logs of any of the minions that failed, I see that the minion is receiving all the correct data from the master, processing the request for the ping correctly and then connecting to the master with its response to the call. The master seemingly does not receive this communication from the minion and therefore the master retries. Again in the minions logs I see it receiving the retry attempt from the master, the minion processes the request and sends it onto the master successfully. But the master does not receive or process that data correctly and therefore the master sees the call as failing against the minion.
So to sum up, the minions are performing as expected but the master is not processing the info sent to it from a random set of minions correctly.
The master will continue to fail batch salt calls like this until I restart the salt master service. Then it runs again correctly until a module call returns too much data such as the log line posted above.
Steps to Reproduce Issue
Run a module function that causes the minion to return too much data for the master, ie causing the following error in the master logs:
[salt.transport.ipc:195 ][ERROR ][28934] Exception occurred while handling stream: 11844764 exceeds max_str_len(1048576)Then performing any salt call from master against a large batch such as
test.pingagainst *.Versions Report
Master:
Minions are the same except do not have the following installed:
The text was updated successfully, but these errors were encountered: