Fix two O(n^2) performance bugs in the sysctl.present state#58732
Fix two O(n^2) performance bugs in the sysctl.present state#58732Ch3LL merged 12 commits intosaltstack:masterfrom
Conversation
ghost
requested a review
from
|
Fixing the tests is blocked by #58746 . |
Ch3LL
left a comment
Ch3LL
left a comment
There was a problem hiding this comment.
Can you also add a changelog entry as documented here: https://docs.saltstack.com/en/master/topics/development/changelog.html
|
Hey @asomers thanks for the PR! I've been OOO for a bit and am just catching up, apologies for the wait! I'll be reviewing this PR as soon as I can, thanks for your patience! |
When looking up a single sysctl, just look up that single one. Don't look up every sysctl in the system.
|
On one particular server that I manage, this change reduces the time for |
|
I added the changelog entry as @Ch3LL requested. |
waynew
left a comment
waynew
left a comment
There was a problem hiding this comment.
Looks good, just a couple of minor (potential) improvements.
| self.assertDictEqual(sysctl.present(name, value), ret) | ||
| with patch.dict(sysctl.__salt__, {"sysctl.show": mock_config}): | ||
| mock_get = MagicMock(return_value=old_value) | ||
| with patch.dict(sysctl.__salt__, {"sysctl.get": mock_get}): |
There was a problem hiding this comment.
suggestion: Given the number of places where we're mocking stuff like sysctl.get, we should probably import the function and use create_autospec to create these, if we can.
suggestion: deeply nested with statements can be avoided by rewriting like so:
mock_get = create_autospec(sysctl.get, return_value=old_value)
patch_opts = patch.dict(sysctl.__opts__, {"test": True})
patch_salt = patch.dict(sysctl.__salt__, {"sysctl.show": mock_config, "sysctl.get", mock_get})
with patch_opts, patch_salt:
self.assertDictEqual(...)
This could be done throughout the tests in this PR
There was a problem hiding this comment.
I can't use create_autospec, because there is no sysctl.get object. And the sysctl.__salt__ dict is empty until I put a mock inside.
There was a problem hiding this comment.
🤔 if there's no sysctl.get, what is called with __salt__['sysctl.get']?
There was a problem hiding this comment.
The mock object that gets put there by patch.dict
There was a problem hiding this comment.
Right, but normally it exists /somewhere/, right? Or the code wouldn't work when calling via the command line.
That's the function to autospec
There was a problem hiding this comment.
I'm sure that it normally exists somewhere. But I can't figure out where. Does it have something to do with __virtualname__ and __virtual__()? Are modules like these not loaded during the state unit tests?
There was a problem hiding this comment.
Ah, that's where the confusion is 🙃
Normally __salt__ will contain a 'module.function' for everything in salt/modules/, but def __virtual__(): does allow you to change the name. This is useful for cases where you need different behavior based on operating system or even underlying library.
In /this/ case, there are several files in modules/*_sysctl.py that could apply. Fortunately they all have the same API, so any one of them could apply. I'm not sure if there's a better way for create_autospec, though 🤔
There was a problem hiding this comment.
I guessed it must be something like that. But in this case, there's literally nothing in sysctl.__salt__:
(Pdb) p sysctl.__salt__
{}
There was a problem hiding this comment.
Yeah - that's because within the (unit) test suites we avoid loading all the things 🙃
salt/states/sysctl.py
Outdated
| elif name in configured and name in current: | ||
| if str(value).split() == __salt__["sysctl.get"](name).split(): | ||
| elif name in configured and current != "": | ||
| if str(value).split() == current.split(): |
There was a problem hiding this comment.
question: As I look at this if/elif chain would it be possible to rewrite this a bit clearer, perhaps by shuffling the order/indenting things?
There was a problem hiding this comment.
Ok. I made the chain a little deeper but less long now.
|
The test failures look unrelated. |
|
@waynew there's still one test failure, but it looks unrelated to this PR. What is the next step? |
waynew
left a comment
waynew
left a comment
There was a problem hiding this comment.
@asomers good question. Hopefully this will be fixed in the main branch - pinged the core team to ask. I'm approving it but we may need to rebase/merge master to get that test green (or re-run the suite if it's an infrastructure issue)
|
Thanks Wayne. Just ping me if I need to do anything. |
|
Should be good, just need to get the tests green. A lot of cleanup on pytest has occurred recently so i'll merge to master and should be good to go after that. |
6 participants
What does this PR do?
Fixes two O(n^2) performance bugs in the sysctl.present state
Previous Behavior
Two bugs:
test=True, thesysctl.presentstate would call the sysctl module to get the values of every_single_sysctl. Sometimes there are tens of thousands of those. And it would do that again, for every_singlesysctl.presentstate.sysctl.showfunction would split the output of thesysctlcommand line-by-line. For multiline values, it would reassemble them by appending to a string, which isO(n^2).New Behavior
sysctl.presentstate will callsysctl.getinstead ofsysctl.show, only looking up a single value.sysctl.showconcatenates strings usingstr.join, which is much faster.Merge requirements satisfied?
Commits signed with GPG?
Yes
Tests run?
yes