Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2018.3.0 utils do not seem to get synced properly #46911

Closed
sjorge opened this issue Apr 5, 2018 · 8 comments
Closed

2018.3.0 utils do not seem to get synced properly #46911

sjorge opened this issue Apr 5, 2018 · 8 comments
Labels
Pending-Discussion
Milestone

Comments

@sjorge
Copy link
Contributor

@sjorge sjorge commented Apr 5, 2018

Description of Issue/Question

I dropped all the new zfs/zpool stuff that missed the boat for 2018.3 into the dirs to have them synced:

[.]$ find ../
../
../_states
../_states/README
../_states/zfs.py
../_states/zpool.py
../_modules
../_modules/README
../_modules/zfs.py
../_modules/zpool.py
../_runners
../_runners/README
../_grains
../_grains/README
../_grains/zfs.py
../_utils
../_utils/README
../_utils/zfs.py

They all showed succesfully synced after running salt '*' saltutil.sync_all, however the grains module fails to load properly because it cannot find the salt.utils.zfs.

I assume this might be because grains are a bit tricky and do not get a lot of the stuff loaded into it. So the sycned utils may be missing.

Setup

N/A

Steps to Reproduce Issue

Add a grains that uses something form a custom utils module.
Sync both to other minions, the new grains module will not find the newly synced utils module.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.11.2
       cherrypy: unknown
       dateutil: 2.6.0
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: 1.6.7
         Jinja2: 2.9
        libgit2: Not Installed
        libnacl: 1.5.0
       M2Crypto: 0.22
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.14 (default, Dec 30 2017, 15:38:16)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.3
           RAET: 0.6.6
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.2.2

System Versions:
           dist:
         locale: UTF-8
        machine: i86pc
        release: 5.11
         system: SunOS
        version: Not Installed
@sjorge
Copy link
Contributor Author

@sjorge sjorge commented Apr 5, 2018

Bit more info on how to test this, I would write a test utils and grains module but rather busy this week. Sorry.

But a simple helloworld utils that has a function hello that returns "world" that is called from a grains module that sets the hello grains to the output of the utils function, should be enough to replicate this issue.

Als the synced salt.grains.hello module wont know about salt.utils.hello yet. (even though it is synced properly)

@gtmanfred
Copy link
Contributor

@gtmanfred gtmanfred commented Apr 5, 2018

So, you can't overwrite utils like that. The utils in grains are imported, so they have to be in the importable path.

if you want to use the ones from extmods, you need to use __utils__['mod.func'] Which grains cannot do, because no dunders are packed in grains at all.

@sjorge
Copy link
Contributor Author

@sjorge sjorge commented Apr 5, 2018

Yeah I assumed it was something like that, a bit annoying if you’re working on something and want to push it to all your minions. (Although I could probably just use a state to write the until in the proper site-packages dir, that would be ugly though)

@gtmanfred
Copy link
Contributor

@gtmanfred gtmanfred commented Apr 5, 2018

Yeah, the other option would be to put the stuff you need to generate the grain only in the grains file, you would have to duplicate code, but... there isn't really a better way.

Though, eriks findings here may interest you... #46841

if he does figure out how the documented issue is supposed to work.

@sjorge
Copy link
Contributor Author

@sjorge sjorge commented Apr 5, 2018

@garethgreenaway garethgreenaway added this to the Blocked milestone Apr 5, 2018
@garethgreenaway garethgreenaway added the Pending-Discussion label Apr 5, 2018
@gtmanfred
Copy link
Contributor

@gtmanfred gtmanfred commented Apr 5, 2018

Oh, alternatively you could also use this, and install the grains using a pip package, and just import them from the pip package in the grains module.

https://docs.saltstack.com/en/latest/topics/tutorials/packaging_modules.html

Then you could pip install it, and it would load grains, but could also load utils for other stuff that does use __utils__ but you could just import it directly in your grains function.

@sjorge
Copy link
Contributor Author

@sjorge sjorge commented Apr 5, 2018

Yeah that could work, but I could just as wel spin a 2018.3.0extra package at that point that ships the few extra bits I want to test on a wider set of minions.

The original idea was to sync them, have it run and quickly patch and fix bugs if I find them.

I think we can probably close this issue. The fact that grains won’t pick up utils synced over is by design (well implementation limitations).

There are a few alternatives:

  • using a state to drop the new one I the python port (pro: easy, can be undone if a backup is made of the original, con: not nice to override package manager managed files)
  • custom pip package (lots of work, atleast for what I wanted to use it for)
  • spinning a custom salt package (same as above)

So anybodying reading this ticket in the future should find answers here.

@gtmanfred
Copy link
Contributor

@gtmanfred gtmanfred commented Apr 5, 2018

👍 sounds good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Pending-Discussion
Projects
None yet
Development

No branches or pull requests

3 participants