Regression in v3001.x Orchestrate runner fails on empty state

[uvsmtid]

If state renderer produces empty state (e.g. common case when Jinja template {% if ... %}...{% endif %} compiles out anything to do for specific minion), it can perfectly be run by salt command.

The same state run through orchestration (via salt-run command) results in error.

Easy to re-try:

• copy and paste all commands and file contents
• substitute minion_id by real minion id everywhere

Setup

empty_state.sls:

# empty_state.sls
#
# There is nothing here.

orchestrate_empty_state.sls:

# orchestrate_empty_state.sls
run_empty_state:
    salt.state:
        - tgt: minion_id
        - sls: empty_state

Results

• Run by normal state execution:

salt minion_id state.sls empty_state

minion_id:

Summary
-----------
Succeeded: 0
Failed:   0
-----------
Total:    0

• Run by orchestrate runner:

salt-run state.orchestrate orchestrate_empty_state

salt:
----------
          ID: run_empty_state
    Function: salt.state
      Result: False
     Comment: Run failed on minions: minion_id
              Failures:
                  minion_id:

                  Summary
                  -----------
                  Succeeded: 0
                  Failed:   0
                  -----------
                  Total:    0
     Changes:   

Summary
------------
Succeeded: 0
Failed:    1
------------
Total:     1

Versions

Both master and minion are deployed on the same Fedora 20 x86_64 host.

salt --versions-report

           Salt: 2014.1.7
         Python: 2.7.5 (default, Jun 25 2014, 10:19:55)
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
         PyYAML: 3.10
          PyZMQ: 13.0.2
            ZMQ: 3.2.4

Workaround

Simply add a dummy state to the empty_state.sls file:

dummy_step:
    cmd.run:
        - name: "echo dummy"

This time orchestrate runner results in success:

salt-run state.orchestrate orchestrate_empty_state

salt:
----------
          ID: run_empty_state
    Function: salt.state
      Result: True
     Comment: States ran successfully. Updating minion_id.
     Changes:   
              minion_id:
              ----------
                        ID: dummy_step
                  Function: cmd.run
                      Name: echo dummy
                    Result: True
                   Comment: Command "echo dummy" run
                   Changes:   
                            ----------
                            pid:
                                4921
                            retcode:
                                0
                            stderr:

                            stdout:
                                dummy

              Summary
              ------------
              Succeeded: 1
              Failed:    0
              ------------
              Total:     1

Summary
------------
Succeeded: 1
Failed:    0
------------
Total:     1

[cachedout]

Nice catch! We'll fix this.

[ssgward]

@uvsmtid - is this still an issue with the latest salt? If not an issue, can this be closed?

[uvsmtid]

I've just re-confirmed the issue on 2015.5.2.

Orchestrate runner still fails - only the output format changed (see below).

It fails without non-zero error code but it's a separate issue #18510.

Current output using 2015.5.2

salt-run state.orchestrate orchestrate_empty_state
[ERROR   ] Run failed on minions: projector
Failures:
    projector:
        Minion did not respond

minion_id:
    ----------
    salt_|-run_empty_state_|-run_empty_state_|-state:
        ----------
        __run_num__:
            0
        changes:
            ----------
        comment:
            Run failed on minions: minion_id
            Failures:
                ?[0;36mminion_id:?[0m
                ?[0;32m    Minion did not respond?[0m
        duration:
            1401.172
        name:
            run_empty_state
        result:
            False
        start_time:
            00:17:52.470817

Versions

 salt --versions-report
           Salt: 2015.5.2
         Python: 2.7.10 (default, May 27 2015, 18:11:38)
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.4.1
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: 1.0.1

[ssgward]

Thanks for the update.

[halfgaar]

I have this problem with 2016.3.2+ds-1. I just upgraded the master from some 2015 version and this showed up. We have a bunch of states that are empty on some minions, which is now becomming a problem with plain salt (not orchestrate).

The failed count may be zero, but we do get a non-zero exit code notice:

minionname:

Summary for minionname
-----------
Succeeded: 0
Failed:   0
-----------
Total states run:    0
ERROR: Minions returned with non-zero exit code

[madrisan]

Same issue with 2018.3.0 (Oxygen).

salt --versions-report
Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Nov 24 2017, 17:33:09)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.4 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-5-amd64
         system: Linux
        version: debian 9.4 

[mrshannon]

The workaround I am currently using is to put this at the top of my state, before any filtering:

fix empty state <state name>:
  cmd.run:
    - name: /bin/true
    - unless: /bin/true

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[amalaguti]

Opening this issue, is still found in 3001.1, in this case when it's run under normal salt to minion execution.

salt master_minion state.sls sched_state -l warning

No problems using salt-call, no problems running the state as an orchestration.

salt-call state.sls sched_state -l warning

local:

Summary for local

Succeeded: 0
Failed: 0

Total states run: 0
Total run time: 0.000 ms

salt master_minion state.sls sched_state -l warning

master_minion:

Summary for master_minion

Succeeded: 0
Failed: 0

Total states run: 0
Total run time: 0.000 ms
ERROR: Minions returned with non-zero exit code

salt-run state.orch sched_state -l info

[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'sched_state.sls'
ip-10-27-74-40.enterprise.pdx.dev.aws.saltstack.net_master:

Summary for ip-10-27-74-40.enterprise.pdx.dev.aws.saltstack.net_master

Succeeded: 0
Failed: 0

Total states run: 0
Total run time: 0.000 ms
[INFO ] Runner completed: 20200812152625634108

[anilsil]

Please update if you are still seeing this issue with 3004?

[waynew]

I was able to reproduce this and track down the underlying cause:

salt/salt/overstate.py

Lines 89 to 90 in a14fdf7

	if not running:
	return False

basically, because there are no states that are executing, the return code is 2.

That was originally used only for requisites, eventually it percolated it's way to where it lives and is used today. In fact it's pretty trivial:

empty.sls

# this space intentionally left blank

salt-call --local state.apply empty
echo $?

This will show an exit code of 2. I haven't found any indication that this was an intentional choice for an empty state run, just... how things were.

I don't know if anyone out there is relying on this behavior in Salt, but... it seems like it would be something possible to change 🤔

[anilsil]

Closing this due to age, the old version of Salt and Python 2.
Can you retest this with the latest release of Salt 3006.2 and if still an issue, please open a new issue, which will have metrics in tracking issues.