Custom modules cannot be imported in custom grains

[dnd]

Currently with 2014.7 builtin salt modules can be loaded manually in grain files. Custom modules, however, seem to be unable to be imported. Is there something that can be done to add custom modules to python's load paths?

[pitatus]

editing your PYTHON_PATH to include them is a possibility, however, care needs to be taken because the loader will not have processed these modules.. test to know what your limitations are in this case.

[dnd]

I'm not terribly well versed with python. What exactly would I be setting PYTHON_PATH to?

[whiteinge]

@dnd what are you currently doing to manually load modules in a custom grain module? Are you calling salt.loader.raw_mod or something like that?

[dnd]

@whiteinge I've tried various import statements, as well as trying to use salt, but it looks like that isn't defined yet in the grains. I see in grains/core.py that there are some load order issues around this, so that wasn't surprising.

[whiteinge]

You're right that __salt__ is not available in grains since they're the first module type that is loaded (loading execution modules requires that __grains__ are already loaded). Also in general, using import to grab Salt modules is problematic.

Try this instead, of course substituting test for your custom module and test.ping for your custom module.function that you're trying to call.

import salt.loader

testmod = salt.loader.raw_mod(__opts__, 'test', None)
result = testmod['test.ping']()

[dnd]

That gives a NameError that __opts__ is not defined. I tried replacing that with an empty hash, but then things broke further along.

[whiteinge]

My bad. I thought that was available there. Add this:

import salt.config

__opts__ = salt.config.minion_config('/etc/salt/minion')

[whiteinge]

@dnd er, actually __opts__ should definitely already be available. Make sure you're doing this inside your custom grain function. Salt's loader only injects the special "dunder" variables into functions, it does not do anything at the module-level. Sorry for being unclear. Full example:

/srv/salt/_grains/yourcustomgrain.py:

import salt.loader

def yourcustomgrain():
    testmod = salt.loader.raw_mod(__opts__, 'test', None)
    result = testmod['test.ping']()

    return {
        'yourcustomgrain': result,
    }

[dnd]

Thanks. I did get it working, but was surprised by something. The custom grain appears to override grains set in /etc/salt/grains. I was expecting manually set grains to override any grains calculated by salt.

[whiteinge]

I believe the order of execution (and thus precedence) is minion conf ->
grains file -> core grains modules -> custom grains.

On Thu Jan 08 2015 at 11:02:09 AM Steve notifications@github.com wrote:

Thanks. I did get it working, but was surprised by something. The custom
grain appears to override grains set in /etc/salt/grains. I was expecting
manually set grains to override any grains calculated by salt.

—
Reply to this email directly or view it on GitHub
#19459 (comment).

[dnd]

Is there a way to get at the grains that salt already thinks it knows at
that point?

I would also argue that it would seem to make more sense that the grains
manually defined should take precedence over calculated ones.
On Jan 8, 2015 12:24 PM, "Seth House" notifications@github.com wrote:

I believe the order of execution (and thus precedence) is minion conf ->
grains file -> core grains modules -> custom grains.

On Thu Jan 08 2015 at 11:02:09 AM Steve notifications@github.com wrote:

Thanks. I did get it working, but was surprised by something. The custom
grain appears to override grains set in /etc/salt/grains. I was
expecting
manually set grains to override any grains calculated by salt.

—
Reply to this email directly or view it on GitHub
#19459 (comment).

—
Reply to this email directly or view it on GitHub
#19459 (comment).

[whiteinge]

Steve, after a little code diving followed by verification with some folk
at the Salt office it seems I was mistaken in the evaluation order. Custom
grains are indeed processed before anything else for the reason that they
should not be able to override any of the core grains. That would also
explain why you are seeing an empty grains dictionary. This needs to be
added to the grains docs.

The following should hopefully get you unblocked for the time being. Let me
know if it does not.

python
import yaml

def yourgrain():
    with open('/etc/salt/grains', 'rb') as f:
        static_grains = yaml.safe_load(f)
    return {
        'yourgrain': static_grains,
    }

On Thu Jan 08 2015 at 11:42:20 AM Steve notifications@github.com wrote:

Is there a way to get at the grains that salt already thinks it knows at
that point?

I would also argue that it would seem to make more sense that the grains
manually defined should take precedence over calculated ones.
On Jan 8, 2015 12:24 PM, "Seth House" notifications@github.com wrote:

I believe the order of execution (and thus precedence) is minion conf ->
grains file -> core grains modules -> custom grains.

On Thu Jan 08 2015 at 11:02:09 AM Steve notifications@github.com
wrote:

Thanks. I did get it working, but was surprised by something. The
custom
grain appears to override grains set in /etc/salt/grains. I was
expecting
manually set grains to override any grains calculated by salt.

—
Reply to this email directly or view it on GitHub
#19459 (comment).

—
Reply to this email directly or view it on GitHub
#19459 (comment).

—
Reply to this email directly or view it on GitHub
#19459 (comment).

[dnd]

That load order makes more sense. The problem though is that the grains I have defined in /etc/salt/grains are not overriding the grain set by the custom grain. I'm assuming the grain load order goes something like: custom > core > file? Maybe something is happening and the file based grains are on the wrong side of the merge?