jobs.exit_success fails to report failure

[cruscio]

Description of Issue/Question

salt-run jobs.exit_success <jid> will return True for a minion that shows a non-zero failure count in salt-run jobs.lookup_jid <jid> --out=highstate --state_output=terse

This has happened to me with failures using file.managed & cmd.run state modules, and also when the minion returns a stack trace for an authentication timeout SaltClientError: Attempt to authenticate with the salt master failed with timeout error

jobs.lookup_jid:

<minion1>:
  [...]  
  Name: Move-Item -Force -Path [...] - Function: cmd.run - Result: Failed Started: - 13:46:19.801000 Duration: 547.0 ms
  [...]

Summary for <minion>
-------------
Succeeded: 16 (changed=17)
Failed:     1
-------------
<minion2>:
    The minion function caused an exception: Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt\minion.py", line 1606, in _thread_return
    return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
  File "c:\salt\bin\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt\modules\state.py", line 748, in apply_
    return sls(mods, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt\modules\state.py", line 1234, in sls
    initial_pillar=_get_initial_pillar(opts))
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 4037, in __init__
    BaseHighState.__init__(self, opts)
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 2902, in __init__
    self.opts = self.__gen_opts(opts)
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 2929, in __gen_opts
    mopts = self.client.master_opts()
  File "c:\salt\bin\lib\site-packages\salt\fileclient.py", line 1387, in master_opts
    return salt.utils.data.decode(self.channel.send(load)) if six.PY2 \
  File "c:\salt\bin\lib\site-packages\salt\utils\async.py", line 64, in wrap
    ret = self._block_future(ret)
  File "c:\salt\bin\lib\site-packages\salt\utils\async.py", line 74, in _block_future
    return future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt\transport\tcp.py", line 363, in send
    ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt\transport\tcp.py", line 340, in _crypted_transfer
    yield self.auth.authenticate()
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
SaltClientError: Attempt to authenticate with the salt master failed with timeout error

jobs.exit_success:

<minion1>: True
<minion2>: True

Versions Report

salt:/$ salt --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.15.0-1030-azure
         system: Linux
        version: Ubuntu 18.04 bionic

salt:/$ salt '<minion>' test.versions_report
<minion>:
    Salt Version:
               Salt: 2018.3.2

    Dependency Versions:
               cffi: 1.10.0
           cherrypy: 10.2.1
           dateutil: 2.6.1
          docker-py: Not Installed
              gitdb: 2.0.3
          gitpython: 2.1.3
              ioflo: Not Installed
             Jinja2: 2.9.6
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: Not Installed
               Mako: 1.0.6
       msgpack-pure: Not Installed
     msgpack-python: 0.4.8
       mysql-python: Not Installed
          pycparser: 2.17
           pycrypto: 2.6.1
       pycryptodome: Not Installed
             pygit2: Not Installed
             Python: 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
       python-gnupg: 0.4.1
             PyYAML: 3.12
              PyZMQ: 16.0.3
               RAET: Not Installed
              smmap: 2.0.3
            timelib: 0.2.4
            Tornado: 4.5.1
                ZMQ: 4.1.6

    System Versions:
               dist:
             locale: cp1252
            machine: AMD64
            release: 2012ServerR2
             system: Windows
            version: 2012ServerR2 6.3.9600  Multiprocessor Free

[cruscio]

In case it helps, the output of jobs.print_job, showing a retcode of 2 for the minion and a result: False for the individual step

        <minion1>:
            ----------
            out:
                highstate
            retcode:
                2
            return:
                ----------
[...]
                cmd_|-winlogbeat move from staging_|-Move-Item -Force -Path ***** -Destination *****_|-run:
                    ----------
                    __id__:
                        winlogbeat move from staging
                    __run_num__:
                        9
                    __sls__:
                        winlogbeat
                    changes:
                        ----------
                        pid:
                            4636
                        retcode:
                            1
                        stderr:
                            Move-Item : Access to the path
                            '*****' is denied.
                            At line:1 char:1
                            + Move-Item -Force -Path
                            ***** ...
                            + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                            ~~~
                                + CategoryInfo          : WriteError: (*****:D
                               irectoryInfo) [Move-Item], IOException
                                + FullyQualifiedErrorId : MoveDirectoryItemIOError,Microsoft.PowerShell.Co
                               mmands.MoveItemCommand
                        stdout:
                    comment:
                        Command "Move-Item -Force -Path ***** -Destination *****" run
                    duration:
                        547.0
                    name:
                        Move-Item -Force -Path ***** -Destination *****
                    result:
                        False
                    start_time:
                        13:46:19.801000
[...]

[dwoz]

@cruscio thank you for reporting this issue. I've confirmed this on the latest 2018.3. I used a state with a file being written to a location that results in a permissions error.