2018.3.0 utils do not seem to get synced properly

[sjorge]

Description of Issue/Question

I dropped all the new zfs/zpool stuff that missed the boat for 2018.3 into the dirs to have them synced:

[.]$ find ../
../
../_states
../_states/README
../_states/zfs.py
../_states/zpool.py
../_modules
../_modules/README
../_modules/zfs.py
../_modules/zpool.py
../_runners
../_runners/README
../_grains
../_grains/README
../_grains/zfs.py
../_utils
../_utils/README
../_utils/zfs.py

They all showed succesfully synced after running salt '*' saltutil.sync_all, however the grains module fails to load properly because it cannot find the salt.utils.zfs.

I assume this might be because grains are a bit tricky and do not get a lot of the stuff loaded into it. So the sycned utils may be missing.

Setup

N/A

Steps to Reproduce Issue

Add a grains that uses something form a custom utils module.
Sync both to other minions, the new grains module will not find the newly synced utils module.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.11.2
       cherrypy: unknown
       dateutil: 2.6.0
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: 1.6.7
         Jinja2: 2.9
        libgit2: Not Installed
        libnacl: 1.5.0
       M2Crypto: 0.22
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.14 (default, Dec 30 2017, 15:38:16)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.3
           RAET: 0.6.6
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:
         locale: UTF-8
        machine: i86pc
        release: 5.11
         system: SunOS
        version: Not Installed

[sjorge]

Bit more info on how to test this, I would write a test utils and grains module but rather busy this week. Sorry.

But a simple helloworld utils that has a function hello that returns "world" that is called from a grains module that sets the hello grains to the output of the utils function, should be enough to replicate this issue.

Als the synced salt.grains.hello module wont know about salt.utils.hello yet. (even though it is synced properly)

[gtmanfred]

So, you can't overwrite utils like that. The utils in grains are imported, so they have to be in the importable path.

if you want to use the ones from extmods, you need to use __utils__['mod.func'] Which grains cannot do, because no dunders are packed in grains at all.

[sjorge]

Yeah I assumed it was something like that, a bit annoying if you’re working on something and want to push it to all your minions. (Although I could probably just use a state to write the until in the proper site-packages dir, that would be ugly though)

[gtmanfred]

Yeah, the other option would be to put the stuff you need to generate the grain only in the grains file, you would have to duplicate code, but... there isn't really a better way.

Though, eriks findings here may interest you... #46841

if he does figure out how the documented issue is supposed to work.

[sjorge]

Well in some personal projects I just build a small tree like site-packages and add it to the python path. That _might_ also work, if the path is inserted before the normal one, modules would be imported from there if available. But that would probably break so many things in salt, and the salt loader stuff really confuses me if I am honest. IIRC it was sys.path.insert(0, ‘private/site-die’), I have since switched to virtualenv for that project but I suppose it would be possible to somehow fix this. Not sure the time/effort would be worth the benefits it would bring. ~ sjorge

…

On 5 Apr 2018, at 23:24, Daniel Wallace ***@***.***> wrote: Yeah, the other option would be to put the stuff you need to generate the grain only in the grains file, you would have to duplicate code, but... there isn't really a better way. Though, eriks findings here may interest you... #46841 if he does figure out how the documented issue is supposed to work. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[gtmanfred]

Oh, alternatively you could also use this, and install the grains using a pip package, and just import them from the pip package in the grains module.

https://docs.saltstack.com/en/latest/topics/tutorials/packaging_modules.html

Then you could pip install it, and it would load grains, but could also load utils for other stuff that does use __utils__ but you could just import it directly in your grains function.

[sjorge]

Yeah that could work, but I could just as wel spin a 2018.3.0extra package at that point that ships the few extra bits I want to test on a wider set of minions.

The original idea was to sync them, have it run and quickly patch and fix bugs if I find them.

I think we can probably close this issue. The fact that grains won’t pick up utils synced over is by design (well implementation limitations).

There are a few alternatives:

• using a state to drop the new one I the python port (pro: easy, can be undone if a backup is made of the original, con: not nice to override package manager managed files)
• custom pip package (lots of work, atleast for what I wanted to use it for)
• spinning a custom salt package (same as above)

So anybodying reading this ticket in the future should find answers here.

[gtmanfred]

👍 sounds good.