Skip to content

/salt

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matcher in loader, take 4 #48809

Merged
merged 44 commits into from Sep 26, 2018

Conversation

Reviewers

@terminalmage terminalmage

@isbm isbm

@gtmanfred gtmanfred

@rallytime rallytime

@cachedout cachedout

Assignees
No one assigned
Labels
Fluorine
Projects
None yet
Milestone
No milestone
6 participants
@cro @gtmanfred @rallytime @cachedout @terminalmage @isbm
@cro
Copy link
Member

commented Jul 27, 2018

What does this PR do?

Creates a new directory salt/matchers and migrates the matcher functions into separate .py files in this directory. These functions are loaded at minion startup, but can be reloaded with the new saltutil.refresh_matchers function.

Tests written?

Matcher tests apply here.

Commits signed with GPG?

Yes

@cro cro requested a review from saltstack/team-core as a code owner Jul 27, 2018

@salt-jenkins salt-jenkins requested a review from saltstack/team-suse Jul 27, 2018

@cro

This comment has been minimized.

Sign in to view
Copy link
Member Author

commented Jul 27, 2018

Previous incarnations:

#48624
#48784
#48785
@gtmanfred

gtmanfred reviewed Jul 28, 2018
View changes

tests/integration/netapi/rest_tornado/test_app.py Outdated
@@ -203,8 +203,8 @@ def test_multi_local_async_post(self):
self.assertEqual(len(ret), 2)
self.assertIn('jid', ret[0])
self.assertIn('jid', ret[1])
self.assertEqual(ret[0]['minions'], sorted(['minion', 'sub_minion', 'localhost']))
self.assertEqual(ret[1]['minions'], sorted(['minion', 'sub_minion', 'localhost']))

This comment has been minimized.

Sign in to view
Copy link
@gtmanfred

gtmanfred Jul 28, 2018

Contributor

Why are these getting removed?

This was added for testing the enable_ssh_minions feature added in 2018.3.

It does need the ssh daemon started in runtests.py with python -m tests.runtests --ssh, but those minions should still be returning from the tornado tests.

This comment has been minimized.

Sign in to view
Copy link
@cro

cro Jul 28, 2018

Author Member

OK, this is what I needed to know, thanks.
@gtmanfred

gtmanfred suggested changes Jul 28, 2018
View changes

Copy link
Contributor

left a comment

After talking to Nicole, those tests are not the flaky ones that she marked in 2017.7, so they should still be passing.

These should be returning localhost, because it should be matching the minions for the enable_ssh_minions targeting. in salt.utils.minions

https://github.com/saltstack/salt/blob/develop/salt/utils/minions.py#L715
@isbm

isbm approved these changes Jul 30, 2018
View changes

Copy link
Contributor

left a comment

@cro 💥 very cool! 💥
@gtmanfred

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Jul 31, 2018

@rallytime looks like these tests are failing on the head of develop too? can you confirm?

@cro revert the test changes, and I will fix them in a different PR, and approve this one.
@gtmanfred

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Jul 31, 2018

Trying the tests inside of test-kitchen to see if I can verify them in our test setup.

@cro cro force-pushed the cro:matcher_in_loader2 branch from ad947c1 to 627f9f2 Jul 31, 2018

@rallytime rallytime added the Fluorine label Jul 31, 2018

@gtmanfred

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Aug 1, 2018

The tests look better, and do not appear to be failing on the tornado tests anymore, rerunning the tests that did not post unittest results.
@rallytime

rallytime approved these changes Aug 1, 2018
View changes

@rallytime rallytime requested a review from cachedout Aug 2, 2018

@gtmanfred

gtmanfred approved these changes Aug 2, 2018
View changes

@rallytime rallytime added the Tests Passed label Aug 2, 2018

@cachedout

cachedout reviewed Aug 3, 2018
View changes

Copy link
Collaborator

left a comment •
edited

My big concern here is that these functions all take self and then get converted on-the-fly to class-based methods from the context of the minion.

I presume that this was to avoid changing the interface to the Matchers from the minion's perspective but I'm not sure this is the right decision. IMHO, we should modify the minion-side logic so that it just calls functions. This is the pattern in every other pluggable system in Salt and I am very hesitant about diverging from that.
doc/topics/matchers/index.rst Outdated

Matchers are modules that provide Salt's targeting abilities. As of the
Flourine release, matchers can be dynamically loaded. Currently new matchers
cannot be created, but existing matchers may have their functionality altered or

This comment has been minimized.

Sign in to view
Copy link
@cachedout

cachedout Aug 3, 2018

Collaborator

Is this just because the CLI would need to be modified as well? I feel like this needs more explanation.

@rallytime rallytime changed the base branch from develop to fluorine Aug 6, 2018

@rallytime

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Aug 6, 2018

@cro I have changed the base branch of this PR from develop to fluorine for inclusion in the Fluorine feature release.

Also, if you could make the changes requested above ASAP, that would be great so we can get this merged in. Thanks!
@cachedout

This comment has been minimized.

Sign in to view
Copy link
Collaborator

commented Aug 10, 2018

Bump @cro
@cro

This comment has been minimized.

Sign in to view
Copy link
Member Author

commented Aug 14, 2018 •
edited

So if I understand correctly @cachedout, you would like to see this behave more like executors, which are loaded specifically with a call to salt.loader.executors(), and then referenced with the standard LazyLoader pattern, e.g.

matchers = salt.loader.matchers(opts)
matchers['list_match.match'](...)

@cro cro force-pushed the cro:matcher_in_loader2 branch 2 times, most recently from 6561a34 to d62b87d Aug 14, 2018

@cro

This comment has been minimized.

Sign in to view
Copy link
Member Author

commented Aug 14, 2018

K, d62b87d there removes the Matcher class and goes instead with the bare loader.
@cro

This comment has been minimized.

Sign in to view
Copy link
Member Author

commented Aug 14, 2018

Tests may not pass fully yet, I have a mock or two I have to figure out what to do with.
@cachedout

This comment has been minimized.

Sign in to view
Copy link
Collaborator

commented Aug 15, 2018

@cro Correct. I know that's a bit of a pain but the consistency will be a win in the long run. Thanks for doing this!
@cachedout

This comment has been minimized.

Sign in to view
Copy link
Collaborator

commented Aug 15, 2018

@cro Also a syntax error snuck in: https://jenkinsci.saltstack.com/job/pr-lint/job/PR-48809/13/warnings52Result/new/

@cachedout cachedout removed the Tests Passed label Aug 15, 2018

@cachedout

cachedout requested changes Aug 18, 2018
View changes

salt/modules/match.py Outdated
try:
return matcher.pcre_match(tgt)
return matcher['pcre_match.match'](tgt)

This comment has been minimized.

Sign in to view
Copy link
@cachedout

cachedout Aug 18, 2018

Collaborator

Should be plural, I think.
@cachedout

This comment has been minimized.

Sign in to view
Copy link
Collaborator

commented Aug 20, 2018

@cro More fixing needed: https://jenkinsci.saltstack.com/job/pr-kitchen-centos7-py2/job/PR-48809/15/console
@rallytime

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Aug 24, 2018

@cro Any update here? We need these tests fixed before we can get this in.

cro and others added some commits Sep 18, 2018

@cro
Lint
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
8941245
@cro
Back out list matcher optimization, somehow breaks nodegroup matching.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
3cd0848
@cro
Backed out the change too far :-(
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
4c1f215
@cro
Pass __opts__ in modules/match.py since we no longer have self.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
5ca3b9c
@terminalmage @cro
Optimize list matcher by doing string membership checks, 
This is faster than splitting into a list and then iterating over it.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
1593b58
@terminalmage @cro
Optimize subnet membership check 
By using a generator comprehension, we avoid pre-allocating an entire
list of objects.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
a3c5d24
@terminalmage @cro
Optimize subdict matching 
This reverses the order in which matches are attempted so that we start
with more deeply-nested matches first. The rationale for this is that it
is much more likely that when one uses `foo:bar:baz:hello:world`, they
are looking for a deeply-nested dictionary containing a key `hello`
which maps to `world`, than a key named `foo` matching the string
`bar:baz:hello:world`. The one case in which this change would be less
performant would be in the case where the value being checked is a
UNIX-style PATH (which uses colons as separators).
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
4373a0b
@terminalmage @cro
Also support passing address as a tuple 
*Just* in case something else that invokes this starts using tuples at
some point, it'll save us a traceback.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
Loading status checks…
631eab3

@cro cro force-pushed the cro:matcher_in_loader2 branch from 670f087 to 631eab3 Sep 21, 2018

cro added some commits Sep 21, 2018

@cro
Try list matcher optimization again.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
Loading status checks…
05b216a
@cro
Imported wrong `six`.
Verified
This commit was signed with a verified signature.
@cro cro C. R. Oldham
GPG key ID: 336DDBE4E260935A Learn about signing commits
Loading status checks…
72b2c21
@rallytime

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Sep 24, 2018

@cro Getting closer, but there are still some tests that need to be fixed here:

https://jenkinsci.saltstack.com/job/pr-kitchen-centos7-py3/job/PR-48809/24/

All of those test failures are related to the changes in this PR.
@terminalmage
Make sequence optimization more efficient
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
b58fa9d
@terminalmage

terminalmage reviewed Sep 24, 2018
View changes

salt/matchers/list_match.py Outdated
or tgt.startswith(__opts__['id'] + ',') \
or tgt.endswith(',' + __opts__['id'])
if isinstance(tgt, collections.Sequence) and not isinstance(tgt, six.string_types):
result = bool(__opts__['id'] in tgt)

This comment has been minimized.

Sign in to view
Copy link
@terminalmage

terminalmage Sep 24, 2018

Contributor

__opts__['id'] in tgt already returns a bool, so wrapping it in bool() is redundant.

Also, we can skip the type checking altogether by just relying on exception handling to narrow down what type the target is. I've opened cro#16 with one possible way we could do this.

cro and others added some commits Sep 24, 2018

@cro
Merge pull request #16 from terminalmage/matcher-optimizations 
Make sequence optimization more efficient
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
95a9ae8
@terminalmage
Add target to the warning log message
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
Loading status checks…
93dd1ad
@terminalmage
Fix netapi tests 
localhost should not appear in minion list unless there is an exchanged
key for it, which there is not
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
112d2ec
@terminalmage
Add a dunder init for matchers
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
d627af8
@terminalmage
Fix pillar unit tests to work properly with loader-based matchers
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
Loading status checks…
b829729
@terminalmage
lint
Verified
This commit was signed with a verified signature.
@terminalmage terminalmage Erik Johnson
GPG key ID: 5E5583C437808F3F Learn about signing commits
Loading status checks…
0125185
@terminalmage

This comment has been minimized.

Sign in to view
Copy link
Contributor

commented Sep 25, 2018

AWS is having some issues, but tests did pass on the one platform where they finished without SSH connection issues. This should be good to merge once we can get the test suite to run to completion.

@rallytime rallytime closed this Sep 25, 2018

@rallytime rallytime reopened this Sep 25, 2018

@brejoc brejoc referenced this pull request Sep 26, 2018

Merged

No longer passes missing in list compound engine #49435

@gtmanfred gtmanfred closed this Sep 26, 2018

@gtmanfred gtmanfred reopened this Sep 26, 2018

@rallytime rallytime merged commit 1db2d2e into saltstack:fluorine Sep 26, 2018
4 of 11 checks passed

4 of 11 checks passed

jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has failed
Details
jenkins/pr/py2-windows-2016 The py2-windows-2016 job has failed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has failed
Details
jenkins/pr/py3-windows-2016 The py3-windows-2016 job has failed
Details
jenkins/pr/"py2"-"centos-7" running "py2"-"centos-7"...
Details
jenkins/pr/docs Testing docs...
Details
jenkins/pr/py2-centos-7 running py2-centos-7...
Details
WIP ready for review
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
jenkins/pr/lint The lint job has passed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has passed
Details

@kfdm kfdm referenced this pull request Apr 18, 2019

Closed

Crash when mixing ext_pillar and pillar:match in /srv/pillar/top.sls #52597

@max-arnold max-arnold referenced this pull request Apr 24, 2019

Open

Compound targeting by nodegroup broken in 2019.2.0 #52678

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.