Support loading entrypoints by passing a module instead of a function

[s0undt3ch]

What does this PR do?

Support loading entrypoints by passing a module instead of a function.

The old way of defining salt loader entrypoints was, for example, on setup.cfg:

[options.entry_points]
salt.loader=
  runner_dirs = thirpartypackage.loader:func_to_get_list_of_dirs
  module_dirs = thirpartypackage.loader:func_to_get_list_of_dirs

This old way is still supported in the rare ocasion where you actually
want to pass multiple paths to the loader and to not break backwards
compatibility.

The new way of defining the salt loader entrypoints is, for example, on setup.cfg:

[options.entry_points]
salt.loader=
  <this-name-does-not-matter = thirpartypackage
  <this-name-does-not-matter = thirpartypackage:callable

With this new approach, when the target is a package, Salt looks for the
ext_type being loaded as submodule of the package and if it exists, it
loads from there.
If the target is a callable, then, that callable should return a
dictionary where the keys are the several salt loader names. For
salt execution modules it would be modules, for salt state modules it would
be states, etc. The same directory layout you see in salt.
The values for those keys needs to be an iterable(not a string which is
also an iterable), ie, a list, a generator, etc, whose values are the
paths from where to load the salt loader modules.

What issues does this PR fix or reference?

Fixes #58939

[dwoz]

[options.entry_points]
salt.loader=
  <this-name-does-not-matter = thirpartypackage
  <this-name-does-not-matter = thirpartypackage:callable

I think this-name-does-not-matter is part of what we want to show up in the versions report, in which case it would matter. The reason being a pypi package might contain more than one 'extension'.

[options.entrypoints]
salt.loader=
   myplugin1 = myplugin1module
   myplugin2 = myplugin2module

At least, I believe this is what was intended for they keys in the configs for entrypoints and can see some value from it.

Then in the versions report we could show something like

third-party-extension[myplugin1]
third-party-extension[myplugin2]

or simply

myplugin1
myplugin2

Not sure what's really better there..

I could see that in most cases there would be a single plugin and the name would match the package name.

[s0undt3ch]

But then you have the same extension, with the same version, showing up multiple times on the versions report, just because they provide more than one entry point.... I don't see any value added there....

third-party-extension[myplugin1,myplugin2,etc...]

Would be less verbose, but I still don't see added value there....

[s0undt3ch]

re-run all

addressed

[dwoz]

But then you have the same extension, with the same version, showing up multiple times on the versions report, just because they provide more than one entry point.... I don't see any value added there....

third-party-extension[myplugin1,myplugin2,etc...]

Would be less verbose, but I still don't see added value there....

It's not about the extension, it's about the plugin. There can be a case where the package providing the plugin has a whole lot of functionality that has nothing to do with Salt. For instance, boto could provide a salt plugin that they maintain which ships with boto3.

[s0undt3ch]

It's not about the extension, it's about the plugin. There can be a case where the package providing the plugin has a whole lot of functionality that has nothing to do with Salt. For instance, boto could provide a salt plugin that they maintain which ships with boto3.
And?

We only report salt related extensions...

I'm not seeing the issue

[s0undt3ch]

For instance, boto could provide a salt plugin that they maintain which ships with boto3.

And we report that boto3 version X.Y.Z provide extensions for salt...

[waynew]

I'll look into this closer tomorrow morning, but in the meantime there is the changelog comment

[waynew]

Hey! Sorry this took a bit longer than planned to get back to you. I think this looks excellent!

There are a couple of places where we could add a bit more testing (the negative cases). But otherwise this seems 👍 to me.

I cloned this locally and added my own "extension module" and tried the 3 different styles of loading - list, dict, and ext_types sub module + /modules/ dir and manually verified that it works.

Oh, one more question: should we also test the other types of dirs? I see that this test includes modules, but don't we have other potential dirs? from my debugging code:

ext_dirs? auth_dirs
ext_dirs? engines_dirs
ext_dirs? fileserver_dirs
ext_dirs? grains_dirs
ext_dirs? log_handlers_dirs
ext_dirs? matchers_dirs
ext_dirs? module_dirs
ext_dirs? returner_dirs
ext_dirs? runner_dirs
ext_dirs? serializers_dirs
ext_dirs? tokens_dirs
ext_dirs? top_dirs
ext_dirs? utils_dirs
ext_dirs? wheel_dirs