Matcher in loader, take 4 #48809

cro · 2018-07-27T22:07:08Z

What does this PR do?

Creates a new directory salt/matchers and migrates the matcher functions into separate .py files in this directory. These functions are loaded at minion startup, but can be reloaded with the new saltutil.refresh_matchers function.

Tests written?

Matcher tests apply here.

Commits signed with GPG?

Yes

cro · 2018-07-27T22:08:16Z

Previous incarnations:

#48624
#48784
#48785

gtmanfred · 2018-07-28T00:10:05Z

tests/integration/netapi/rest_tornado/test_app.py

@@ -203,8 +203,8 @@ def test_multi_local_async_post(self):
        self.assertEqual(len(ret), 2)
        self.assertIn('jid', ret[0])
        self.assertIn('jid', ret[1])
-        self.assertEqual(ret[0]['minions'], sorted(['minion', 'sub_minion', 'localhost']))
-        self.assertEqual(ret[1]['minions'], sorted(['minion', 'sub_minion', 'localhost']))


Why are these getting removed?

This was added for testing the enable_ssh_minions feature added in 2018.3.

It does need the ssh daemon started in runtests.py with python -m tests.runtests --ssh, but those minions should still be returning from the tornado tests.

OK, this is what I needed to know, thanks.

gtmanfred

After talking to Nicole, those tests are not the flaky ones that she marked in 2017.7, so they should still be passing.

These should be returning localhost, because it should be matching the minions for the enable_ssh_minions targeting. in salt.utils.minions

https://github.com/saltstack/salt/blob/develop/salt/utils/minions.py#L715

isbm

@cro 💥 very cool! 💥

gtmanfred · 2018-07-31T17:05:35Z

@rallytime looks like these tests are failing on the head of develop too? can you confirm?

@cro revert the test changes, and I will fix them in a different PR, and approve this one.

gtmanfred · 2018-07-31T17:08:50Z

Trying the tests inside of test-kitchen to see if I can verify them in our test setup.

gtmanfred · 2018-08-01T13:19:50Z

The tests look better, and do not appear to be failing on the tornado tests anymore, rerunning the tests that did not post unittest results.

cachedout

My big concern here is that these functions all take self and then get converted on-the-fly to class-based methods from the context of the minion.

I presume that this was to avoid changing the interface to the Matchers from the minion's perspective but I'm not sure this is the right decision. IMHO, we should modify the minion-side logic so that it just calls functions. This is the pattern in every other pluggable system in Salt and I am very hesitant about diverging from that.

cachedout · 2018-08-03T14:19:03Z

doc/topics/matchers/index.rst

+
+Matchers are modules that provide Salt's targeting abilities.  As of the
+Flourine release, matchers can be dynamically loaded.  Currently new matchers
+cannot be created, but existing matchers may have their functionality altered or


Is this just because the CLI would need to be modified as well? I feel like this needs more explanation.

rallytime · 2018-08-06T20:28:57Z

@cro I have changed the base branch of this PR from develop to fluorine for inclusion in the Fluorine feature release.

Also, if you could make the changes requested above ASAP, that would be great so we can get this merged in. Thanks!

cachedout · 2018-08-10T21:24:17Z

Bump @cro

cro · 2018-08-14T19:56:13Z

So if I understand correctly @cachedout, you would like to see this behave more like executors, which are loaded specifically with a call to salt.loader.executors(), and then referenced with the standard LazyLoader pattern, e.g.

matchers = salt.loader.matchers(opts)
matchers['list_match.match'](...)

cro · 2018-08-14T22:35:42Z

K, d62b87d there removes the Matcher class and goes instead with the bare loader.

cro · 2018-08-14T22:36:10Z

Tests may not pass fully yet, I have a mock or two I have to figure out what to do with.

cachedout · 2018-08-15T04:14:57Z

@cro Correct. I know that's a bit of a pain but the consistency will be a win in the long run. Thanks for doing this!

cachedout · 2018-08-15T04:15:34Z

@cro Also a syntax error snuck in: https://jenkinsci.saltstack.com/job/pr-lint/job/PR-48809/13/warnings52Result/new/

cachedout · 2018-08-18T12:21:28Z

salt/modules/match.py

    try:
-        return matcher.pcre_match(tgt)
+        return matcher['pcre_match.match'](tgt)


Should be plural, I think.

cachedout · 2018-08-20T14:18:25Z

@cro More fixing needed: https://jenkinsci.saltstack.com/job/pr-kitchen-centos7-py2/job/PR-48809/15/console

rallytime · 2018-08-24T13:08:23Z

@cro Any update here? We need these tests fixed before we can get this in.

This is faster than splitting into a list and then iterating over it.

By using a generator comprehension, we avoid pre-allocating an entire list of objects.

This reverses the order in which matches are attempted so that we start with more deeply-nested matches first. The rationale for this is that it is much more likely that when one uses `foo:bar:baz:hello:world`, they are looking for a deeply-nested dictionary containing a key `hello` which maps to `world`, than a key named `foo` matching the string `bar:baz:hello:world`. The one case in which this change would be less performant would be in the case where the value being checked is a UNIX-style PATH (which uses colons as separators).

*Just* in case something else that invokes this starts using tuples at some point, it'll save us a traceback.

rallytime · 2018-09-24T13:48:33Z

@cro Getting closer, but there are still some tests that need to be fixed here:

https://jenkinsci.saltstack.com/job/pr-kitchen-centos7-py3/job/PR-48809/24/

All of those test failures are related to the changes in this PR.

terminalmage · 2018-09-24T17:48:08Z

salt/matchers/list_match.py

-            or tgt.startswith(__opts__['id'] + ',') \
-            or tgt.endswith(',' + __opts__['id'])
+        if isinstance(tgt, collections.Sequence) and not isinstance(tgt, six.string_types):
+            result = bool(__opts__['id'] in tgt)


__opts__['id'] in tgt already returns a bool, so wrapping it in bool() is redundant.

Also, we can skip the type checking altogether by just relying on exception handling to narrow down what type the target is. I've opened cro#16 with one possible way we could do this.

Make sequence optimization more efficient

localhost should not appear in minion list unless there is an exchanged key for it, which there is not

terminalmage · 2018-09-25T18:16:32Z

AWS is having some issues, but tests did pass on the one platform where they finished without SSH connection issues. This should be good to merge once we can get the test suite to run to completion.

cro requested a review from a team as a code owner July 27, 2018 22:07

ghost self-requested a review July 27, 2018 22:07

gtmanfred reviewed Jul 28, 2018

View reviewed changes

gtmanfred suggested changes Jul 28, 2018

View reviewed changes

isbm approved these changes Jul 30, 2018

View reviewed changes

cro force-pushed the matcher_in_loader2 branch from ad947c1 to 627f9f2 Compare July 31, 2018 19:01

rallytime added the ZRELEASED - Fluorine reitred label label Jul 31, 2018

rallytime approved these changes Aug 1, 2018

View reviewed changes

rallytime requested a review from cachedout August 2, 2018 18:07

gtmanfred approved these changes Aug 2, 2018

View reviewed changes

rallytime added the Tests-Passed label Aug 2, 2018

cachedout reviewed Aug 3, 2018

View reviewed changes

rallytime changed the base branch from develop to fluorine August 6, 2018 20:28

cro force-pushed the matcher_in_loader2 branch 2 times, most recently from 6561a34 to d62b87d Compare August 14, 2018 22:33

cachedout removed the Tests-Passed label Aug 15, 2018

cachedout suggested changes Aug 18, 2018

View reviewed changes

cro and others added 8 commits September 21, 2018 09:36

Lint

8941245

Back out list matcher optimization, somehow breaks nodegroup matching.

3cd0848

Backed out the change too far :-(

4c1f215

Pass __opts__ in modules/match.py since we no longer have self.

5ca3b9c

Optimize list matcher by doing string membership checks,

1593b58

This is faster than splitting into a list and then iterating over it.

Optimize subnet membership check

a3c5d24

By using a generator comprehension, we avoid pre-allocating an entire list of objects.

Also support passing address as a tuple

631eab3

*Just* in case something else that invokes this starts using tuples at some point, it'll save us a traceback.

cro force-pushed the matcher_in_loader2 branch from 670f087 to 631eab3 Compare September 21, 2018 15:36

cro added 2 commits September 21, 2018 15:44

Try list matcher optimization again.

05b216a

Imported wrong six.

72b2c21

Make sequence optimization more efficient

b58fa9d

terminalmage reviewed Sep 24, 2018

View reviewed changes

cro and others added 6 commits September 24, 2018 11:50

Merge pull request #16 from terminalmage/matcher-optimizations

95a9ae8

Make sequence optimization more efficient

Add target to the warning log message

93dd1ad

Fix netapi tests

112d2ec

localhost should not appear in minion list unless there is an exchanged key for it, which there is not

Add a dunder init for matchers

d627af8

Fix pillar unit tests to work properly with loader-based matchers

b829729

lint

0125185

rallytime closed this Sep 25, 2018

rallytime reopened this Sep 25, 2018

brejoc mentioned this pull request Sep 26, 2018

No longer passes missing in list compound engine #49435

Merged

gtmanfred closed this Sep 26, 2018

gtmanfred reopened this Sep 26, 2018

rallytime merged commit 1db2d2e into saltstack:fluorine Sep 26, 2018

kfdm mentioned this pull request Apr 18, 2019

Crash when mixing ext_pillar and pillar:match in /srv/pillar/top.sls #52597

Closed

max-arnold mentioned this pull request Apr 24, 2019

Compound targeting by nodegroup broken in 2019.2.0 #52678

Closed

funbaker mentioned this pull request Dec 16, 2019

Wrong minion_id in matcher methods. #55656

Open

Matcher in loader, take 4 #48809

Matcher in loader, take 4 #48809

Uh oh!

Conversation

cro commented Jul 27, 2018

What does this PR do?

Tests written?

Commits signed with GPG?

Uh oh!

cro commented Jul 27, 2018

Uh oh!

gtmanfred Jul 28, 2018

Choose a reason for hiding this comment

Uh oh!

cro Jul 28, 2018

Choose a reason for hiding this comment

Uh oh!

gtmanfred left a comment

Choose a reason for hiding this comment

Uh oh!

isbm left a comment

Choose a reason for hiding this comment

Uh oh!

gtmanfred commented Jul 31, 2018

Uh oh!

gtmanfred commented Jul 31, 2018

Uh oh!

gtmanfred commented Aug 1, 2018

Uh oh!

cachedout left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachedout Aug 3, 2018

Choose a reason for hiding this comment

Uh oh!

rallytime commented Aug 6, 2018

Uh oh!

cachedout commented Aug 10, 2018

Uh oh!

cro commented Aug 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cro commented Aug 14, 2018

Uh oh!

cro commented Aug 14, 2018

Uh oh!

cachedout commented Aug 15, 2018

Uh oh!

cachedout commented Aug 15, 2018

Uh oh!

cachedout Aug 18, 2018

Choose a reason for hiding this comment

Uh oh!

cachedout commented Aug 20, 2018

Uh oh!

rallytime commented Aug 24, 2018

Uh oh!

rallytime commented Sep 24, 2018

Uh oh!

terminalmage Sep 24, 2018

Choose a reason for hiding this comment

Uh oh!

terminalmage commented Sep 25, 2018

Uh oh!

Uh oh!

cachedout left a comment •

edited

Loading

cro commented Aug 14, 2018 •

edited

Loading