weird salt behaviour for highstate with contradicting state.funcs

[noelmcloughlin]

Description of Issue/Question

Salt seems to get confused if highstate includes contradicting state.functions.

On #45510 I noted weird issue wrt service.dead and service.running in same highstate.

Replicating
State postgres.server.init (service.running watching pkg.installed) works.
State postgres.server.remove (service.dead watching pkg.removed) works.
Include both states in same highstate breaks service.running:

'The named service xxxxxxx is not available'

Today a 2nd weird issue wrt to alternatives.remove and alternatives.installed in same highstate.

Replicating
State postgres.server.remove.alternatives.remove is True (initdb symlink does not exist).
State postgres.server.init.pkg.installed is True (upstream postgres-server installs initdb symlink).
State postgres.server.init.alternatives.install is True (initdb symlink already exists).
State postgres.cmd.run.initdb is False (initdb symlink is missing Huh???)

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Both issues are related to #45510.
Both issues replicated when highstate invokes this code before standard install use cases execute in same highstate?

Masterless minion.

...
postgresql-{{ bin }}-removed:
alternatives.remove:
- path: {{ path }}
- require:
- pkg: postgresql-server-removed
- onlyif: test -f {{ path }}
...
{{ bin }}:
alternatives.install:
- link: {{ salt['file.join']('/usr/bin', bin) }}
- path: {{ path }}
- priority: {{ postgres.linux.altpriority }}
- onlyif: test -f {{ path }}
...

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Both issues happen if highstate includes postgres.server.remove and postgres.server.init states.

(1) The high.state called alternatives.remove which works as expected (initdb not found).

    Function: alternatives.remove
      Result: True
     Comment: onlyif execution failed
     Started: 17:22:41.408590
    Duration: 1978.029 ms
     Changes:

(2) Then high.state called pkg.installed which added upstream postgresql-server package. This package creates the initdb symlink during RPM post-installation.

(3) Then high.state called alternatives.install which noticed the symlink was pre-existing.

    Function: alternatives.install
      Result: True
     Comment: Alternatives for initdb is already set to /usr/pgsql-9.5/bin/initdb

(4) Then highstate called 'cmd.run.name: initdb ..' but symlink is gone? Huh????

ID: postgresql-cluster-prepared
    Function: cmd.run
        Name: initdb --pgdata=  /var/lib/pgsql/9.5/data
      Result: False
     Comment: Command "initdb --pgdata=  /var/lib/pgsql/9.5/data" run
     Started: 17:22:54.882354
    Duration: 6327.614 ms
     Changes:
              ----------
              pid:
                  2331
              retcode:
                  127
              stderr:
                  /bin/bash: initdb: command not found
              stdout:

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2017.7.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: 2.7.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

[noelmcloughlin]

If I split use case into two separate highstates everything works.

highstate1: postgres.server.remove & postgres.client.remove
highstate2: postgres & postgres.upstream

[noelmcloughlin]

Cannot replicate on Ubuntu.

[garethgreenaway]

@noelmcloughlin Initial thought is that initdb was in one path when your highstate begins and is in another path when the state that is attempting to run it, is being run. In the state that calls initdb can you provide the full path to the command and see if that affects anything?

[noelmcloughlin]

Thanks @garethgreenaway for the input. I was doing more alternatives testing this week on Centos/Fedora. Thealternatives states seems to behave better using onlyif: alternatives --display {{ path }} instead of onlyif: test -f {{path}}.

I'll look into this further over next few weeks when I get time.

[noelmcloughlin]

closing as stale issue. Will test again in future.