Loader docs

[AstraLuma]

What does this PR do?

Add documentation about module loading and in general attempts to improve documentation about writing modules for salt.

What issues does this PR fix or reference?

#50594

Notes

I already know that there's changes for the 2018.3 branch. I'm not sure how to handle that?

• Add token modules
• File server modules can now be loaded from the file server
• Search system was removed
• Documentation for Executors was added.

For flourine:

• matcher modules added https://github.com/saltstack/salt/blob/develop/doc/topics/matchers/index.rst

[AstraLuma]

I am looking for some feedback about the restructuring I'm doing.

I am also thinking about writing some docs about specific module interfaces, for the module types that don't have it yet.

[cachedout]

This needs a newline below it or the documentation will not build.

[AstraLuma]

Thanks! Handled.

[AstraLuma]

@cachedout Did you have any feedback about the "big picture" changes I'm doing?

[cachedout]

@astronouth7303 I think that overall these changes make a lot of sense. 👍

[cachedout]

@astronouth7303 Is this ready to go? If so, please remove WIP from the PR subject line.

[AstraLuma]

I want to look at ref/file_server/dynamic_modules.rst (since a lot of its content overlaps with what I've been writing). There's also a few other sections that need at least a cursory summary.

I'm also hoping to write full interface references for each of the systems, but I guess that can be saved for future PRs.

[cachedout]

@astronouth7303 Sounds good. Just let us know when it's all ready to go. Thanks!

[AstraLuma]

Since I don't feel like writing detailed interface documentation for 20-some modules just yet, this is ready for a more intense review.

[max-arnold]

@cachedout Should the docs discourage using direct module imports? There are tons of them:

grep -r 'salt\.modules' salt/ | grep import

For example, there is an unconditional import of vsphere.py which is quite large and slow:

% grep -r 'salt\.modules.\vsphere' salt | grep import
salt/grains/esxi.py:import salt.modules.vsphere

% ls -la salt/modules/vsphere.py
-rw-r--r--  1 user  staff  373420 Oct 29 20:12 salt/modules/vsphere.py

What is the proper way to cross-reference loadable modules?

[max-arnold]

Also it would be useful to document all the disable_* loader options. So far only the disable_modules and disable_returners are documented: https://docs.saltstack.com/en/latest/ref/configuration/minion.html#minion-execution-module-management

In theory, disable_* should work:

1. both in master and minion config
2. for all module types https://github.com/saltstack/salt/blob/develop/salt/loader.py#L1180 (briefly mentioned in the PR Fix disable_<tag-name> config option #42567)

[cachedout]

@cachedout Should the docs discourage using direct module imports? There are tons of them:

grep -r 'salt\.modules' salt/ | grep import

For example, there is an unconditional import of vsphere.py which is quite large and slow:

% grep -r 'salt\.modules.\vsphere' salt | grep import
salt/grains/esxi.py:import salt.modules.vsphere

% ls -la salt/modules/vsphere.py
-rw-r--r--  1 user  staff  373420 Oct 29 20:12 salt/modules/vsphere.py

What is the proper way to cross-reference loadable modules?

IIRC, there are a few very specialized cases in this turns out to be necessary, but as a general practice, cross-calling modules should be done using the __salt__[module_name.module_func] form. I agree that this would be a good idea to put this in the docs.

[AstraLuma]

In my experience, it's actually pretty hard to do for any modules that aren't shipped with salt. But it's been noted.

[AstraLuma]

Added both a note about importing and a quick list of loader-related settings.

[max-arnold]

Undocumented settings like disable_grains, disable_beacons etc, can have significant performance impact: #48773 (comment)

[AstraLuma]

I'm cutting this PR off at conducting discovery for undocumented settings and documenting them. I really wanted to focus on documenting the salt.load entry point and what's the name for all of the systems in various contexts (since some vary by pluralization, etc). I also ended up doing some dunder completionism since I was hip-deep in loader.py anyways.

I also don't think setting docs should live in this corner of the documentation. They should live with the settings reference and be referred to.

[AstraLuma]

And for some reason, the link to the whitelist_modules setting isn't working locally.

EDIT: Figured it out, the link ID is apparently enable_whitelist_modules for some reason.

[AstraLuma]

@cachedout Can I get your updated review?