extmods/utils is not in sys.path after synchronizing _utils directory

[aplanas]

Description of Issue/Question

When doing a state.hightstate, the initial synchronization works properly and all the _modules, _utils and _grains code from the formula are living in '/var/cache/salt/minion/files/base/' and ('/var/cache/salt/minion/extmods'.

The problem is that when a _module import some code that is living in _utils, this is not found the during this first highstate, as for some reason /var/cache/salt/minion/extmods/utils is not in the sys.path, as only /var/cache/salt/minion/extmods/modules is there.

The extmods/utils is present in the second highstate run, and the import succeed.

Versions Report

develop branch

[garethgreenaway]

@aplanas Thanks for the report. With the way that the Salt loader works, the utils directory cannot be overloaded. /cc @saltstack/team-core

[aplanas]

@garethgreenaway not sure that I understand the concept of overloading. What I can do (and I tested) is that creating a _utils in my formula, as a sister / brother of _modules, _grains, _states, a state.highstate (or a utils.sync_all) copy the contents into the cache, and from there to extmod.

I also checked that config/init.py will add in sys.path the /var/cache/.../extmod/utils directory, so is perfectly accessible in the minion later on.

The problem is that somewhere the code execution during the highstate, the /var/cache/.../extmod/modules is added, but not utils. I am still figuring out where the modules path is added into sys.path, to see if there is an easy fix here.

[garethgreenaway]

@aplanas Did a bit more digging into this and it appears this does work, but does not override the python path. The files which are found in _utils will be injected in __utils__ so you'll need to access them from that dictionary, eg. __utils__["module.function"](arguments).

[aplanas]

@garethgreenaway it is documented that an import will work, and it does during the second run once config/__init__.py inject the sys.path.

https://docs.saltstack.com/en/latest/topics/utils/index.html

In this case also I cannot use the __utils__ dict, as I need to access some names that are not functions.

[aplanas]

@garethgreenaway maybe you know where extmod/modules is added in the sys.path after a sync? I am not able to spot it : ))

[garethgreenaway]

@aplanas What are you trying to import from the utils module that isn't working? Classes and module functions should work.

[aplanas]

@garethgreenaway I need to import some values:

https://github.com/openSUSE/yomi/blob/master/salt/_utils/lp.py

You can see that there are some constants like EQ, LTE, etc that are expected as parameters to some methods.

Anything that is not callable is not accessible via the dunder dicts.

[max-arnold]

The following issues might be relevant: #32500 #46841 #46911 #51719

[aplanas]

@max-arnold Thanks, this will help me.

[aplanas]

Ok I have a proposal, not sure if it will be accepted:

As the extmods path is added to sys.path in two places:

• First in config/init when the minion is started, but as the minion cache is clean this do not add anything the first time
• For each _load_module the file directory is added in case that is not already in sys.path

My proposal is to update _sync in saltutil.py, so that we call insert_system_path from config/init after each sync, so this will add utils, modules, cloud, etc etc when there is some code there, as when the minion start from the beginning.

Does this looks ok?