Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extmods/utils is not in sys.path after synchronizing _utils directory #51958

Closed
aplanas opened this issue Mar 4, 2019 · 11 comments · Fixed by #52001
Closed

extmods/utils is not in sys.path after synchronizing _utils directory #51958

aplanas opened this issue Mar 4, 2019 · 11 comments · Fixed by #52001
Assignees
Labels
Pending-Discussion
Milestone

Comments

@aplanas
Copy link
Collaborator

@aplanas aplanas commented Mar 4, 2019

Description of Issue/Question

When doing a state.hightstate, the initial synchronization works properly and all the _modules, _utils and _grains code from the formula are living in '/var/cache/salt/minion/files/base/' and ('/var/cache/salt/minion/extmods'.

The problem is that when a _module import some code that is living in _utils, this is not found the during this first highstate, as for some reason /var/cache/salt/minion/extmods/utils is not in the sys.path, as only /var/cache/salt/minion/extmods/modules is there.

The extmods/utils is present in the second highstate run, and the import succeed.

Versions Report

develop branch

@garethgreenaway garethgreenaway added this to the Blocked milestone Mar 4, 2019
@garethgreenaway garethgreenaway added the Pending-Discussion label Mar 4, 2019
@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Mar 4, 2019

@aplanas Thanks for the report. With the way that the Salt loader works, the utils directory cannot be overloaded. /cc @saltstack/team-core

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 4, 2019

@garethgreenaway not sure that I understand the concept of overloading. What I can do (and I tested) is that creating a _utils in my formula, as a sister / brother of _modules, _grains, _states, a state.highstate (or a utils.sync_all) copy the contents into the cache, and from there to extmod.

I also checked that config/init.py will add in sys.path the /var/cache/.../extmod/utils directory, so is perfectly accessible in the minion later on.

The problem is that somewhere the code execution during the highstate, the /var/cache/.../extmod/modules is added, but not utils. I am still figuring out where the modules path is added into sys.path, to see if there is an easy fix here.

@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Mar 4, 2019

@aplanas Did a bit more digging into this and it appears this does work, but does not override the python path. The files which are found in _utils will be injected in __utils__ so you'll need to access them from that dictionary, eg. __utils__["module.function"](arguments).

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 4, 2019

@garethgreenaway it is documented that an import will work, and it does during the second run once config/__init__.py inject the sys.path.

https://docs.saltstack.com/en/latest/topics/utils/index.html

In this case also I cannot use the __utils__ dict, as I need to access some names that are not functions.

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 4, 2019

@garethgreenaway maybe you know where extmod/modules is added in the sys.path after a sync? I am not able to spot it : ))

@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Mar 4, 2019

@aplanas What are you trying to import from the utils module that isn't working? Classes and module functions should work.

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 4, 2019

@garethgreenaway I need to import some values:

https://github.com/openSUSE/yomi/blob/master/salt/_utils/lp.py

You can see that there are some constants like EQ, LTE, etc that are expected as parameters to some methods.

Anything that is not callable is not accessible via the dunder dicts.

@garethgreenaway garethgreenaway self-assigned this Mar 5, 2019
@max-arnold
Copy link
Contributor

@max-arnold max-arnold commented Mar 5, 2019

The following issues might be relevant: #32500 #46841 #46911 #51719

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 5, 2019

@max-arnold Thanks, this will help me.

@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 5, 2019

Ok I have a proposal, not sure if it will be accepted:

As the extmods path is added to sys.path in two places:

  • First in config/init when the minion is started, but as the minion cache is clean this do not add anything the first time
  • For each _load_module the file directory is added in case that is not already in sys.path

My proposal is to update _sync in saltutil.py, so that we call insert_system_path from config/init after each sync, so this will add utils, modules, cloud, etc etc when there is some code there, as when the minion start from the beginning.

Does this looks ok?

aplanas added a commit to aplanas/salt that referenced this issue Mar 6, 2019
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
@aplanas
Copy link
Collaborator Author

@aplanas aplanas commented Mar 6, 2019

Instead of calling the code from config, I created this simpler version: #52001

aplanas added a commit to aplanas/salt that referenced this issue Mar 12, 2019
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
aplanas added a commit to aplanas/salt that referenced this issue Mar 12, 2019
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
aplanas added a commit to aplanas/salt that referenced this issue Mar 18, 2019
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
aplanas added a commit to aplanas/salt that referenced this issue Mar 21, 2019
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
waynew pushed a commit to waynew/salt that referenced this issue Jan 13, 2020
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
waynew pushed a commit to alexey-zhukovin/salt that referenced this issue Feb 23, 2021
When the minion start clean and there are not utils directory in
the extmods cache, any import from a module to any code inside the
utils directory will fail. A second run of the highstate will fix
this issue, as the utils directories are added into sys.path inside
config/__init__.py, as part of the master and minion startup.

This commit add the utils directories during the synchronization of
the custom modules.

Fixes saltstack#51958
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Pending-Discussion
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants