State.highstate uses stale grain data.

[primechuck]

Using the python API

Salt: 2015.5.0
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: 0.9.1

Using the python api running.

saltCaller.function('grains.append', 'roles', 'master')

then immediately

saltCaller.function('state.highstate')

Highstate doesn't see the newly appended grain. If highstate is run again after a few seconds or after a sync it sees the grain. The previous behavior in 2014 always grabbed grains when running highstate.

[DmitryKuzmenko]

@primechuck I tried to reproduce the issue in the following way:
Master and minion are on different machines. Grains data are used in top.sls and in a target state sls file like this:

# top.sls
base:
    'node_type:test':
        - match: grain
        - test

    'node_type:foobar':
        - match: grain
        - foobar

# foobar.sls
/tmp/GRAINS-{{ grains.get('node_type', '') }}:
    file.managed:
        - source: salt://foobar.sls
        - require:
            - file: /tmp/testfile4

On minion side I used a simple script to test the issue:

    saltCaller = Caller()
    res = saltCaller.function('grains.append', 'node_type', 'foobar')
    print("===== grains.append ret:\n{0}".format(res))
    res = saltCaller.function('state.highstate')
    print("===== grains.get ret:\n{0}".format(res))

I used v2015.5.0 tag. My versions report:

           Salt: 2015.5.0
         Python: 2.7.9 (default, Dec 11 2014, 04:42:00)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: 1.4.2
         PyYAML: 3.11
          ioflo: 1.2.2
          PyZMQ: 14.6.0
           RAET: 0.6.3
            ZMQ: 4.0.5
           Mako: Not Installed

To let me continue investigation, could you please provide more details?

[primechuck]

There is a lot of missing logic, but the salt salls in the python look like this.

saltCaller = salt.client.Caller()
ret = saltCaller.function('grains.append', 'roles', 'master')
ret = saltCaller.function('grains.append', 'roles', 'test')
ret = saltCaller.function('grains.append', 'roles', 'mysql.server')
ret = saltCaller.function('state.highstate')

There is missing logic because the script is large, but those are all the salt calls in order. The highstate doesn't include the mysql.server role when it is run within the script, however, the grain exists in the minions grain file and if you call highstate again after the script is run, the mysql.server grain is read and the states completes with the grain data. When it did the highstate within python, the highstate doesn't include the appended data.

top.sls sample

base:
  '*':
  'roles:master':
    - match: grain
    - master
'roles:test':
    - match: grain
    - test
  'roles:mysql.server':
    - match: grain
    - mysql.server

[DmitryKuzmenko]

@primechuck thank you for details. I'll look at this.

[DmitryKuzmenko]

Exactly the same steps gave me the described results. Moreover I got the only 'roles:master' state in highstate.

[primechuck]

Glad I'm not the only one. We were testing migrating to 2015.5.0 from a 2014 and couldn't believe this was this issue :) If you need anything more let me know, I've tried to diagnose it to the best of my ability.

[primechuck]

The code sample I placed above is still failing for me with the same issue. High state isn't being passed the appended grains in the same script.
salt-call 2015.5.3 (Lithium)

[DmitryKuzmenko]

@primechuck there is no the fix in Lithium release

$ c=1d42ae598f83a1c56ea362b51ad783a99bee0172; git branch --contains $c; git tag --contains $c
  develop
v2015.8.0rc1

[primechuck]

Ah, saw the issue listed in the release notes without reading the fine print. Not used to github reports.

[cachedout]

@primechuck Did you get a chance to verify this fix prior to the release? We would like to close this issue if it's fixed and do any remaining work for the release if it isn't. Thanks!