Custom grains are synced but not run

[elsmorian]

Description of Issue/Question

I'm having an odd problem which I believe started in 2016.3.1 but not sure.

I have a custom grain written, which runs fine on existing minions (2015.8.8.2), but do not on new ones(2016.3.1) - the extra grain keys and values show up in the older minion's grains, but not the new ones.
Running salt-call -l debug saltutil.sync_grains shows the file the grain is written and getting copied from the minion cache:
[INFO ] Copying '/var/cache/salt/minion/files/base/_grains/network_helpers.py' to '/var/cache/salt/minion/extmods/grains/network_helpers.py'
Running the grain in iPython on both minions works fine.

Setup

2016.3.1 master,
2015.8.8.2 slave that runs custom grains fine
2016.3.1 slave that does not run custom grains

_grains directory is in the root of the gitfs_root on the master, using gitpython as the gitfs provider

Steps to Reproduce Issue

Try and access grain value from a custom grain backed by gitfs.

Versions Report

Master:

salt --versions-report
Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-58-generic
         system: Linux
        version: Ubuntu 14.04 `trusty`

Minion that can access custom grains:

Salt Version:
           Salt: 2015.8.8.2

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-66-generic
         system: Ubuntu 14.04 trusty

Minion that can't access custom grain:

Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-86-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[Ch3LL]

@elsmorian I am not able to replicate this with the following test case on 2016.3.1:

My grains file is here

#!/usr/bin/env python

def gitgrain():
    grains = {}
    grains['testgit'] = 'works'
    return grains

Here is my master configuration file:

gitfs_provider: pygit2
gitfs_base: master
gitfs_remotes:
  - https://github.com/Ch3LL/Ch3LLScripts.git:
    - root: salt/states
fileserver_backend:
  - git 
  - roots

*Note: I also tried this with gitfs_provider: gitpython

1. Sync the grains salt-call -ldebug saltutil.sync_grains
2. attempt to get the grains:

➜  ~ sudo salt-call grains.item testgit
local:
    ----------
    testgit:
        works

1. Are you still seeing the same behavior if you use my example above?
2. Can you share your sanitized version of your master config and a small simple example of your custom grain so I can see if i can replicate it that way.

[elsmorian]

Thanks for your reply @Ch3LL your suggestions did narrow my problem down!

It looks like this and issue with my custom grain that sets grains based on other existing grains by accessing __grains__.

When I was writing the grain before it was unclear which variables you have access to in the context that custom grains are run, and I'm pretty sure the solution to use __grains__ was mentioned in the salt IRC channel - but looks like this var may no longer available or not pre-populated with grain information that is loaded from the /etc/salt/grains file as of 2016.3 where it was before in 2015.8.8?

Included is the custom grain, its designed so I can use {{ external_interface }} in my templates and it will know which actual interface that is based on who provided the machine, which datacenter its in etc.

#!/usr/bin/env python
"""Network helper grain to easily access interfaces."""
from salt.modules import network

PROVIDER_NETWORK_INTERFACES = {
    'online.net': {
        'internal': ['em2'],
        'external': ['em1', 'p1p1'],
        'vpn': ['tun0'],
    },
    'rackspace': {
        'internal': ['eth2'],
        'external': ['eth0'],
        'vpn': ['eth0'],
    },
}

def generate_node_interface_names():
    """Generate salt grains for which IPs are internal and external."""
    node_grains = {}

    try:
        provider_name = __grains__['provider']
    except KeyError:
        dlnode_grains = {'node_grain_error': 'provider not set!'}
        return node_grains

    interface_info = PROVIDER_NETWORK_INTERFACES[provider_name]
    host_interfaces = network.interfaces().keys()

    for interface_type, interface_names in interface_info.iteritems():
        interface_str = '{}_interface'.format(interface_type)
        ip_str = '{}_ip'.format(interface_type)

        for interface_name in interface_names:
            if interface_name in host_interfaces:
                node_grains.update({
                    interface_str: interface_name,
                    ip_str: network.interface_ip(interface_name)
                })
                break

    return node_grains

The provider grain is set at machine build time and stored in /etc/salt/grains.

Is it poor form to use something like below within a custom grain?

import salt.config
import salt.loader
__opts__ = salt.config.minion_config('/etc/salt/minion')
__grains__ = salt.loader.grains(__opts__)

[elsmorian]

OK, well, I have answered my own question - doing the above is indeed poor form, as it causes an infinite loop!!

Is there a way for custom grains to access the static grains set on the minion in the minion config file, or am I going about this all the wrong way?

[elsmorian]

Wow, yeah ok, turns out if you try and do the above, and access the grain system from within a custom grain, you get a loop which stops all salt commands working- even the ones that invalidate the cache - I had to manually rm the cache dir on every minion the bad grain had been pushed to...!

[dalwar23]

I have created a custom grain which declares __opts__ and __grains__ like mentioned above by elsmorian. Now salt \* grains.ls doesn't show my custom grains. But salt \* saltutil.refresh_module returns True and salt \* saltutil.sync_grains returns the name of the custom grain. Is there any solution to this? How can avoid this situation? Help!

[closepin]

I am seeing something pretty similar to this on 2017.7.2 -
We have a custom grain made to identify names of interfaces on Windows machines. Every day we get at least one failure trying to access these custom grains in rendering logic or API queries from our Python code interacting with Salt.

A sync_grains or a sync_all fixes the issue, but that seems like an unnecessary step to ensure our custom grains stay in place when we have not changed this particular custom grain in nearly 2 years. We had erroneously assumed this was due to a version issue; but we recently migrated all minions to 2017.7.2 and the issue remains.

[Ch3LL]

ping @DmitryKuzmenko do you know if this is suppose to be possible to use __grains__ in a custom grain module?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.