Introduce a template system for extending SaltStack open

[tonybaloney]

What does this PR do?

Extending salt by adding new modules, states, adding tests, is quite tricky because there's not really a good example to start from. It's a job of copying and pasting from another module, the salt team don't get much control over this and there isn't much consistency.

What I've done is:

• Introduce a new command salt-extend, which asks the user a series of questions about what they're doing
• Added a new module called salt.utils.extend which wraps the cookiecutter package to generate a new project and merges it into the salt source code

Command line usage for salt-extend is this (anything not sent on command line will be prompted)

usage: salt-extend [-h] [--extension EXTENSION]
                   [--salt-directory SALT_DIRECTORY] [--name NAME]
                   [--description DESCRIPTION] [--no-merge]

Quickly boilerplate an extension to SaltStack

optional arguments:
  -h, --help            show this help message and exit
  --extension EXTENSION, -e EXTENSION
                        Extension type, e.g. 'module', 'state'.
  --salt-directory SALT_DIRECTORY, -o SALT_DIRECTORY
                        Directory where your salt installation is kept
                        (defaults to .).
  --name NAME, -n NAME  Module name.
  --description DESCRIPTION, -d DESCRIPTION
                        Short description of what the module does.
  --no-merge            Don't merge the module into the salt directory, keep
                        in a temp location

If the user (like a core saltstack developer) does work often, they can save their details in the ~/.cookiecutterrc file https://cookiecutter.readthedocs.io/en/latest/advanced/user_config.html

I've submitted 4 templates as an example, based on how I typically extend Salt functions.

• An execution module (module)
• A state module (state)
• An execution unit test module
• A state unit test module

You can use this to provide more templates to any of the SaltStack ecosystem, those are the only 4 that I'm familiar with. The templates use Jinja2 so you can do all sorts of crazy stuff like

Please review Salt's Contributing Guide for best practices.

Comments please, I'm no way tied to the design of this or having templates in the project root for example, so I've tagged all the top contributors for feedback :-)

CC @techhat @cachedout @rallytime @thatch45 @s0undt3ch @terminalmage @nmadhok

[tonybaloney]

weirdly the linter is raising an issue with the logging import about a Py3 compat issue. I develop on py3 and it works fine!

[tonybaloney]

And the other one is dumb "Unnecessary parens after u'print' keyword", well yeah but that's for Python 2, I wouldn't want to write this purely for Py2

[s0undt3ch]

from __future__ import print_function

That will take care of the print function lint issue.

[s0undt3ch]

The other lint issue is not about logging, if you look at the console output on Jenkins it will really tell you want its missing for py3 compat.

from salt.ext.six.moves import zip

[thatch45]

@tonybaloney I really like this concept, I am a little worried about adding another dep in cookiecutter, so I will need to chew on this for a bit.
@cachedout @techhat @rallytime @terminalmage @s0undt3ch what do you all think?

[thatch45]

@tonybaloney we REALLY like this, but as we discuss it internally, we are leaning towards saying that this should be a standalone project instead of being part of Salt itself. We are still debating this but I thought I would ask for your feedback on the subject.

[s0undt3ch]

The __virtualname__ does not need to be the module name although it often is, perhaps add support for a configurable virtualname parameter defaulting to module_name?

[tonybaloney]

slightly long-winded answer, but I added a dictionary of questions in the template, the default value is also templated-using jinja. d99b21c

[s0undt3ch]

And that's about it for now. I hope I'm not sounding too picky.

[tonybaloney]

@s0undt3ch not at all, I'll work through those today. Salt practically has its own standard library so it's best I ask the experts. I looked at salt.version and couldn't figure it out so that all makes sense

[tonybaloney]

OK! I'm done.

Docs, tests, templates (I ended up adding user-defined questions to the template context as well).

Pls review @thatch45 @cachedout

[tonybaloney]

linter is green, the test failures don't have anything to do with these changes *famous last words

[cachedout]

salt.utils.fopen

[tonybaloney]

@cachedout I dropped the namespace on the import https://github.com/saltstack/salt/pull/34921/files#diff-585fafce47a7c44a935dd239a8b93ff0R26

[cachedout]

Ah! FWIW, our style guide suggests using fully-qualified names. (Mostly because I am easily confused.) ;]

[thatch45]

I think we are likely good to merge here....

[cachedout]

It might be nice to use the salt serializer classes here. We do some work to try to massage/optimize YAML. See salt.serializers.yaml for example.

[tonybaloney]

ok done 39159cc

[tonybaloney]

so its just waiting on a response around a jinja class and then I think we're finally good to merge?

[cachedout]

Merged! Thanks so much @tonybaloney. This is really excellent. Many people will benefit from your hard work here!

[tonybaloney]

sweeeeet! Thanks everyone, I actually learnt a lot of new stuff doing this PR.

[thatch45]

This gets the Awesome label too. Thanks for putting up with our scrutiny, it is not usually this bad! We will also need to make sure that this is covered in the release notes for Carbon