Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loader docs #50633

Merged
merged 29 commits into from Dec 10, 2018

Conversation

Projects
None yet
3 participants
@astronouth7303
Copy link
Contributor

commented Nov 26, 2018

What does this PR do?

Add documentation about module loading and in general attempts to improve documentation about writing modules for salt.

What issues does this PR fix or reference?

#50594

Notes

I already know that there's changes for the 2018.3 branch. I'm not sure how to handle that?

  • Add token modules
  • File server modules can now be loaded from the file server
  • Search system was removed
  • Documentation for Executors was added.

For flourine:

astronouth7303 added some commits Nov 26, 2018

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Nov 26, 2018

I am looking for some feedback about the restructuring I'm doing.

I am also thinking about writing some docs about specific module interfaces, for the module types that don't have it yet.

astronouth7303 added some commits Nov 26, 2018


This is done via setuptools entry points:

.. code-block:: python

This comment has been minimized.

Copy link
@cachedout

cachedout Nov 26, 2018

Collaborator

This needs a newline below it or the documentation will not build.

This comment has been minimized.

Copy link
@astronouth7303

astronouth7303 Nov 26, 2018

Author Contributor

Thanks! Handled.

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Nov 26, 2018

@cachedout Did you have any feedback about the "big picture" changes I'm doing?

astronouth7303 added some commits Nov 26, 2018

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Nov 27, 2018

@astronouth7303 I think that overall these changes make a lot of sense. 👍

astronouth7303 added some commits Nov 28, 2018

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Nov 29, 2018

@astronouth7303 Is this ready to go? If so, please remove WIP from the PR subject line.

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Nov 30, 2018

I want to look at ref/file_server/dynamic_modules.rst (since a lot of its content overlaps with what I've been writing). There's also a few other sections that need at least a cursory summary.

I'm also hoping to write full interface references for each of the systems, but I guess that can be saved for future PRs.

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Nov 30, 2018

@astronouth7303 Sounds good. Just let us know when it's all ready to go. Thanks!

astronouth7303 added some commits Nov 30, 2018

@astronouth7303 astronouth7303 changed the title [WIP] Loader docs Loader docs Dec 1, 2018

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2018

Since I don't feel like writing detailed interface documentation for 20-some modules just yet, this is ready for a more intense review.

astronouth7303 added some commits Dec 1, 2018

@max-arnold

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

@cachedout Should the docs discourage using direct module imports? There are tons of them:

grep -r 'salt\.modules' salt/ | grep import

For example, there is an unconditional import of vsphere.py which is quite large and slow:

% grep -r 'salt\.modules.\vsphere' salt | grep import
salt/grains/esxi.py:import salt.modules.vsphere

% ls -la salt/modules/vsphere.py
-rw-r--r--  1 user  staff  373420 Oct 29 20:12 salt/modules/vsphere.py

What is the proper way to cross-reference loadable modules?

@max-arnold

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

Also it would be useful to document all the disable_* loader options. So far only the disable_modules and disable_returners are documented: https://docs.saltstack.com/en/latest/ref/configuration/minion.html#minion-execution-module-management

In theory, disable_* should work:

  1. both in master and minion config
  2. for all module types https://github.com/saltstack/salt/blob/develop/salt/loader.py#L1180 (briefly mentioned in the PR #42567)
@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Dec 3, 2018

@cachedout Should the docs discourage using direct module imports? There are tons of them:

grep -r 'salt\.modules' salt/ | grep import

For example, there is an unconditional import of vsphere.py which is quite large and slow:

% grep -r 'salt\.modules.\vsphere' salt | grep import
salt/grains/esxi.py:import salt.modules.vsphere

% ls -la salt/modules/vsphere.py
-rw-r--r--  1 user  staff  373420 Oct 29 20:12 salt/modules/vsphere.py

What is the proper way to cross-reference loadable modules?

IIRC, there are a few very specialized cases in this turns out to be necessary, but as a general practice, cross-calling modules should be done using the __salt__[module_name.module_func] form. I agree that this would be a good idea to put this in the docs.

astronouth7303 added some commits Dec 4, 2018

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 4, 2018

In my experience, it's actually pretty hard to do for any modules that aren't shipped with salt. But it's been noted.

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 4, 2018

Added both a note about importing and a quick list of loader-related settings.

@max-arnold

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2018

Undocumented settings like disable_grains, disable_beacons etc, can have significant performance impact: #48773 (comment)

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 4, 2018

I'm cutting this PR off at conducting discovery for undocumented settings and documenting them. I really wanted to focus on documenting the salt.load entry point and what's the name for all of the systems in various contexts (since some vary by pluralization, etc). I also ended up doing some dunder completionism since I was hip-deep in loader.py anyways.

I also don't think setting docs should live in this corner of the documentation. They should live with the settings reference and be referred to.

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 4, 2018

And for some reason, the link to the whitelist_modules setting isn't working locally.

EDIT: Figured it out, the link ID is apparently enable_whitelist_modules for some reason.

@astronouth7303

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2018

@cachedout Can I get your updated review?

@cachedout cachedout merged commit e4e9563 into saltstack:2017.7 Dec 10, 2018

6 of 10 checks passed

jenkins/pr/py2-centos-7 The py2-centos-7 job has failed
Details
jenkins/pr/py2-windows-2016 The py2-windows-2016 job has failed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has failed
Details
jenkins/pr/py3-windows-2016 The py3-windows-2016 job has failed
Details
WIP Ready for review
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
jenkins/pr/docs The docs job has passed
Details
jenkins/pr/lint Python lint test has passed
Details
jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has passed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has passed
Details

@max-arnold max-arnold referenced this pull request May 1, 2019

Open

Conformance tests #52775

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.