fix msgpack max_buffer_size being to low in ver >= 0.6 and < 1.0#58283
Conversation
ac6136a to
b11c9df
Compare
|
Need some insight into my 1 failing test in ci/pre-commit. This seems to be the exit code that is failing, but I'm not sure what I'm supposed to do to correct it. Nothing in my code had six or any Py2 related things. Also, the Black test failing is perplexing, as I ran this code through Black and it should have fixed it all in that commit. |
|
https://jenkinsci.saltstack.com/job/pr-pre-commit/job/PR-58283/4/console If you look at the output, you'll see what black is complaining about. |
|
@s0undt3ch Thank you for help on the pre-commit checks. It should be fixed now. I missed that little section in the contributing docs on this. |
|
Off the bat, 500+ commits? |
@s0undt3ch Yep. I did what I was trying not to do, and I even ran a rebase, but must have not done it properly. This is my first time doing this, and the contributing doc did not give me enough info to make me comfortable to doing this. Here's what I did (below), where do I go from here with this mess? Just close the request and resubmit? git fetch upstream It would not allow me to do a ff merge as i had changes from my fix, so I saw "git pull --rebase origin master" in the contributing doc, and that did not look right on for my fix branch I was on. |
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes saltstack#53230, fixes saltstack#54884, fixes saltstack#55180, fixes saltstack#55184, fixes saltstack#57026, fixes saltstack#57921
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their developers changed the max length of a string that could be unpacked from 2GiB to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB will throw an error. For example a file state which tries to send a file larger than 1MiB will get the following error: Unable to manage file: 1048688 exceeds max_str_len(1048576) This error could send the master into a tailspin where subsequent file operations take a long time or just time out. This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB which is the msgpack 1.0 default now. It will allow people to use msgpack versions >.6 and < 1.0 without this crazy low limit. It should be discussed if we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger than 100MB. This code does a version check, and will only set the keyword on msgpack > = 0.6 and < 1.0. The test checks that we have a buffer size of at least 100MiB in case the Python msgpack devs decide to lower this value again, and break things. We want enforce a decent minimum. fixes saltstack#53230, fixes saltstack#54884, fixes saltstack#55180, fixes saltstack#55184, fixes saltstack#57026, fixes saltstack#57921
e7d6f6c to
463d5d8
Compare
salt 3000 introduced a dependency on Python msgpack >= 0.6. In msgpack 0.6 their
developers changed the max length of a string that could be unpacked from 2GiB
to 1MiB. They then proceeded to change it to 100MiB in msgpack 1.0. Now when
using msgpack >=0.6 and < 1.0, any message that is unpacked larger than 1MiB
will throw an error. For example a file state which tries to send a file larger
than 1MiB will get the following error:
Unable to manage file: 1048688 exceeds max_str_len(1048576)
This error could send the master into a tailspin where subsequent file
operations take a long time or just time out.
This patch sets the keyword argument max_buffer_size for the Unpacker to 100MiB
which is the msgpack 1.0 default now. It will allow people to use msgpack
versions >.6 and < 1.0 without this crazy low limit. It should be discussed if
we go back to the 2GiB size from < 0.6 days (my vote), or just anything larger
than 100MB. This code does a version check, and will only set the keyword on
msgpack > = 0.6 and < 1.0.
The test checks that we have a buffer size of at least 100MiB in case the Python
msgpack devs decide to lower this value again, and break things. We want enforce
a decent minimum.
fixes #53230, fixes #54884,
fixes #55180, fixes #55184,
fixes #57026, fixes #57921
What does this PR do?
Fixes error "...exceeds max_str_len(1048576)" when using msgpack ver >= 0.6 and < 1.0.
What issues does this PR fix or reference?
fixes #53230, fixes #54884,
fixes #55180, fixes #55184,
fixes #57026, fixes #57921
Previous Behavior
When trying to unpack a message larger than 1MiB from the salt bus using msgpack, you will get the error mentioned in "What does this PR do?". Then the master will usually hang for long periods or just die.
New Behavior
After patch, the limit will be bumped to 100MiB and allow much larger files and not throw errors.
Merge requirements satisfied?
Commits signed with GPG?
No