[REGRESSION] --output-diff doesn't display changes anymore with test=True since v2019.2.0

[tomlaredo]

Description of Issue/Question

--output-diff doesn't display changes anymore with test=True since v2019.2.0

Setup

# cat state.sls 
/tmp/dest_file.txt:
    file.managed:
        - source: /tmp/source_file.txt

Steps to Reproduce Issue

Initial setup

echo new_content > /tmp/source_file.txt
echo old_content > /tmp/dest_file.txt

With test=True

# salt-call state.apply state --output-diff test=True
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:21.779602
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] The file /tmp/dest_file.txt is set to be changed
[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:21.797227 (duration_in_ms=17.626)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
     Started: 16:36:21.779601
    Duration: 17.626 ms
     Changes:   

Summary for local
------------
Succeeded: 1 (unchanged=1)
Failed:    0
------------
Total states run:     1
Total run time:  17.626 ms

Diff isn't displayed anymore.

Without test=True

vs-salt-master:/srv/salt# salt-call state.apply state --output-diff
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:29.781272
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] File changed:
--- 
+++ 
@@ -1 +1 @@
-old_content
+new_content

[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:29.800569 (duration_in_ms=19.296)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/dest_file.txt updated
     Started: 16:36:29.781273
    Duration: 19.296 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -old_content
                  +new_content

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  19.296 ms

Diff is displayed.

Versions Report

# salt-call --versions-report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.8 
         locale: UTF-8
        machine: x86_64
        release: 4.15.18-9-pve
         system: Linux
        version: debian 9.8 

[garethgreenaway]

@saltstack/team-core Thoughts?

[thatch45]

We have gone back and forth on this one a lot over the years. The core argument has been that the changes dict should only have changes in it if changes actually happened. Thats why it keeps getting pulled out. The main argument is that this is the expected behavior.

The counter argument is that this is some very valuable information to have from a test=True run and that it should be made available.

The counter/counter to that is that we don't have the ability to get predictive changes out of all states, therefore the output becomes inconsistent.

Finally, it can also be argued that if the result is None then we should know for sure that we did not make changes, so any changes in the changes dict can be seen as predictive.

[garethgreenaway]

Brief discussion between @thatch45 @waynew and myself, we concluded that the diff should be shown and we'll look to see what changed from 2018.3.x to 2019.2.x to disable this.

[thatch45]

Agreed, and when we get it back, lets add a fat comment in there to keep it :)

[tomlaredo]

Thanks for those informations and for your reactivity.

Even if test=True isn't perfect and could miss some changes (generally due to triggers like watch / watch_in etc.), it is a huge help for the one who needs to check some particular changes instead of blindly applying states and just hope changes will be good.
A note could be displayed on test=True telling the user that changes are juste there for information and can be wrong in certain cases.

[waynew]

I'm a fan of adding a note in there, maybe something like Comment: File /tmp/dest_file.txt updated. Note: Actual changes may be different due to other states.