Adding an execution module providing helpers for the NAPALM formulas

[mirceaulinic]

What does this PR do?

Some time ago I started preparing some formulas for network configuration management with respect to the OpenConfig standards. The repositories are created, part of the formulas are available, e.g, https://github.com/saltstack-formulas/napalm-ntp-formula

Generally, all these formulas (unfortunately) only generate a text blob that is eventually loaded on the device. Therefore the formulas will 90% be some Jinja template with a considerable size. To shrink this size, and make it more human readable, I thought about having a module, that transforms blocks such as:

{%- if subif_cfg.config.get('description') %}
  description {{ subif_cfg.config.description }};
{%- endif %}

Into a more simplified (when reading, at least):

{{ salt.napalm_formula.render_field(subif_cfg.config, 'description') }}

With the same rendering result:

description "Some description";

Which does all these repetitive checks such as above, which would appear tens of times in a template. Besides, it also add the field name.

This module is generic enough, and it will be used by all the future NAPALM Formulas, e.g., https://github.com/saltstack-formulas/napalm-bgp-formula, https://github.com/saltstack-formulas/napalm-interfaces-formula etc.

Together with this, adding some more little helpers for various manipulation of the OpenConfig-like Python objects.

[rallytime]

@mirceaulinic Lint :)

https://jenkinsci.saltstack.com/job/pr-lint/job/PR-48879/1/warnings52Result/

[mirceaulinic]

Ooops! It should be fixed now, sorry!

[terminalmage]

Since this code is going into develop, there is no need to do this try/except, since the function lives in salt.utils.data now. Not only that, there are three problems with this approach:

1. Importing a function into the module's global scope will add it to the __salt__ dunder (i.e. __salt__['napalm_formula.traverse_dict_and_list'])
2. This will also add this function to the docs, which will be confusing.
3. Since we have compatibility functions in salt.utils for all the funcs that were moved for 2018.3, the first import will succeed and this will result in deprecation warnings.

Instead, you should simply import salt.utils.data and invoke the function with its full name.

[gtmanfred]

When possible, it would be great to also use __utils__['data.traverse_dict_list']() because then it will allow people to include that data.py file in their salt://_utils/ directory if they are unable to upgrade to the newer version of salt.

[terminalmage]

Doing it @gtmanfred's way will also prevent you from needing to import the module at all.

[mirceaulinic]

Thanks guys for reviewing this and your thoughts. As Daniel mentioned, I made these imports having backwards potability in mind; using __utils__ sounds like a better idea, thought it would require backporting loads of util modules.

Re 2: Indeed, I thought about that too, but it seems like Sphinx is smart enough not to pick these functions (no idea why) and they don't appear in the documentation (tested with both Salt and our internal documentation).

Anyway, I will change to use __utils__ as it seems a better tradeoff.

[mirceaulinic]

Updated, though I just wanted to point this out: porting utils.data means it also requires porting utils.dictupdate, utils.stringutils, and utils.yaml (as they are imported there).

This is fine, though when I need, e.g., files.fopen, or args.clean_kwargs, I would also need to port loads of others such as utils.jid, utils.path, utils.platform, utils.data... and here's the catch: salt.utils.args needs salt.utils.data - see https://github.com/saltstack/salt/blob/develop/salt/utils/args.py#L18 (which obviously won't be available). So not only that one basically needs to port almost all the modules under utils/ but also they need to be patched (i.e., replace imports like the one from utils.args from import salt.utils.data to import _utils.data plus updating all the code to use the new import. This sounds like a nightmare to me.

Looking at this vs. 1 & 3 I'm not sure which is the worst...

[mirceaulinic]

What about the following:

from salt.version import __version_info__
if __version_info__ < (2018, 3):
    from salt.utils import traverse_dict_and_list as _traverse_dict_and_list
else:
    from salt.utils.data import traverse_dict_and_list as _traverse_dict_and_list

Or why not, even simpler:

try:
    from salt.utils.data import traverse_dict_and_list as _traverse_dict_and_list
except ImportError:
    from salt.utils import traverse_dict_and_list as _traverse_dict_and_list

This way we can ensure that the function won't be exported (note that I flipped the import order to prevent deprecation warning thrown on >= Oxygen, i.e., it will firstly try to use from salt.utils.data).

Does this sound like a better alternative?

[terminalmage]

This does seem like a better alternative.

try:
    from salt.utils.data import traverse_dict_and_list as _traverse_dict_and_list
except ImportError:
    from salt.utils import traverse_dict_and_list as _traverse_dict_and_list

However, I'm not sure why this should be necessary at all in develop.

[gtmanfred]

Yes, the question just comes down to, "How friendly to we want to be for allowing people to backport these modules?"

[mirceaulinic]

Added 3e9ccdb - please let me know if this is okay for the time being.

[gtmanfred]

I definitely think importing the new method first would be the correct path. And we should add `__utils__` to the loader for utils in the next major release.

…

On Fri, Aug 3, 2018 at 2:59 AM Mircea Ulinic ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In salt/modules/napalm_formula.py <#48879 (comment)>: > +from __future__ import absolute_import, unicode_literals, print_function + +# Import python libs +import copy +import logging +import fnmatch + +# Import salt modules +import salt.utils.napalm +import salt.ext.six as six +import salt.utils.dictupdate +from salt.defaults import DEFAULT_TARGET_DELIM +try: + from salt.utils import traverse_dict_and_list +except ImportError: + from salt.utils.data import traverse_dict_and_list What about the following: from salt.version import __version_info__if __version_info__ < (2018, 3): from salt.utils import traverse_list_and_dict as _traverse_list_and_dictelse: from salt.utils.data import traverse_list_and_dict as _traverse_list_and_dict Or why not, even simpler: try: from salt.utils.data import traverse_dict_and_list as _traverse_dict_and_listexcept ImportError: from salt.utils import traverse_dict_and_list as _traverse_dict_and_list This way we can ensure that the function won't be exported (note that I flipped the import error to prevent deprecation warning thrown on >= Oxygen). Does this sound like a better alternative? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48879 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoVG3MLl7OtfRvCNaZrJpG7nnyYg2ks5uNAL5gaJpZM4VsDBr> .