Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REGRESSION] --output-diff doesn't display changes anymore with test=True since v2019.2.0 #51932

Closed
tomlaredo opened this issue Mar 1, 2019 · 44 comments

Comments

@tomlaredo
Copy link

@tomlaredo tomlaredo commented Mar 1, 2019

Description of Issue/Question

--output-diff doesn't display changes anymore with test=True since v2019.2.0

Setup

# cat state.sls 
/tmp/dest_file.txt:
    file.managed:
        - source: /tmp/source_file.txt

Steps to Reproduce Issue

Initial setup

echo new_content > /tmp/source_file.txt
echo old_content > /tmp/dest_file.txt

With test=True

# salt-call state.apply state --output-diff test=True
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:21.779602
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] The file /tmp/dest_file.txt is set to be changed
[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:21.797227 (duration_in_ms=17.626)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
     Started: 16:36:21.779601
    Duration: 17.626 ms
     Changes:   

Summary for local
------------
Succeeded: 1 (unchanged=1)
Failed:    0
------------
Total states run:     1
Total run time:  17.626 ms

Diff isn't displayed anymore.

Without test=True

vs-salt-master:/srv/salt# salt-call state.apply state --output-diff
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:29.781272
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] File changed:
--- 
+++ 
@@ -1 +1 @@
-old_content
+new_content

[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:29.800569 (duration_in_ms=19.296)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/dest_file.txt updated
     Started: 16:36:29.781273
    Duration: 19.296 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -old_content
                  +new_content

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  19.296 ms

Diff is displayed.

Versions Report

# salt-call --versions-report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.8 
         locale: UTF-8
        machine: x86_64
        release: 4.15.18-9-pve
         system: Linux
        version: debian 9.8 
@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Mar 4, 2019

@thatch45
Copy link
Member

@thatch45 thatch45 commented Mar 4, 2019

We have gone back and forth on this one a lot over the years. The core argument has been that the changes dict should only have changes in it if changes actually happened. Thats why it keeps getting pulled out. The main argument is that this is the expected behavior.

The counter argument is that this is some very valuable information to have from a test=True run and that it should be made available.

The counter/counter to that is that we don't have the ability to get predictive changes out of all states, therefore the output becomes inconsistent.

Finally, it can also be argued that if the result is None then we should know for sure that we did not make changes, so any changes in the changes dict can be seen as predictive.

@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Mar 4, 2019

Brief discussion between @thatch45 @waynew and myself, we concluded that the diff should be shown and we'll look to see what changed from 2018.3.x to 2019.2.x to disable this.

@thatch45
Copy link
Member

@thatch45 thatch45 commented Mar 4, 2019

Agreed, and when we get it back, lets add a fat comment in there to keep it :)

@garethgreenaway garethgreenaway self-assigned this Mar 5, 2019
@tomlaredo
Copy link
Author

@tomlaredo tomlaredo commented Mar 5, 2019

Thanks for those informations and for your reactivity.

Even if test=True isn't perfect and could miss some changes (generally due to triggers like watch / watch_in etc.), it is a huge help for the one who needs to check some particular changes instead of blindly applying states and just hope changes will be good.
A note could be displayed on test=True telling the user that changes are juste there for information and can be wrong in certain cases.

@waynew
Copy link
Contributor

@waynew waynew commented Mar 5, 2019

I'm a fan of adding a note in there, maybe something like Comment: File /tmp/dest_file.txt updated. Note: Actual changes may be different due to other states.

@thatch45
Copy link
Member

@thatch45 thatch45 commented Mar 5, 2019

I like it, maybe say:
Note: No changes made, actual changes may be different due to other states.

@fedepires
Copy link

@fedepires fedepires commented Mar 5, 2019

If it's of any help, from our tests, looks like this commit might be the culprit:

98c61ae

#44353

@rhavenn
Copy link

@rhavenn rhavenn commented Mar 5, 2019

I would only add that a) This is a major feature when trying to merge a Salt state into systems that have not previously been managed by Salt and have years of manual changes in them. One never knows what setting you're going to be accidentally removing. b) If test=true isn't the grammitically correct way this is envisioned then adding a state.test that does the same thing is perhaps more grammatically correct?

@rhavenn
Copy link

@rhavenn rhavenn commented Mar 5, 2019

I can confirm reverting the change @fedepires mentions and restarting my remote minion "fixes" the issue and shows the diffs when running state.apply with test=true

@waynew
Copy link
Contributor

@waynew waynew commented Mar 6, 2019

@fedepires Nice find :) (and helpful for those who need it.

@rhavenn At least from those involved in the minor discussion we had, we didn't have strong feelings about calling it something else. My cursory search didn't find anything to the contrary. I'd say that my personal preference would be to fix this back (with the aforementioned comment), with the idea that test=True is basically equivalent to git's dry-run flag.

Maybe if someone really feels strongly about the semantics/grammar they could submit a PR to introduce a dry-run flag and re-un-restore the behavior for test=True :)

@tomlaredo
Copy link
Author

@tomlaredo tomlaredo commented Mar 6, 2019

To add a last thought to the @rhavenn 's comment, currently, if you misspell "test=True", the state will be executed without any error or message telling that we passed an unknown argument, which is very dangerous and confusing.

Then, replacing test=True by a special state.test (or maybe state.dryrun) would be a good thing I think,

@cjsin
Copy link

@cjsin cjsin commented Mar 19, 2019

Hello,

I agree that this is a terrible regression, it makes it impossible to actually see what changes are going to be made in test mode, almost defeating the entire purpose of the test mode (If you can't see potential changes, then the test mode becomes useful only as a syntax test, to check for errors in the states). We use test mode to capture differences in a nightly audit also which has now become useless without the details of the differences.

Although the change described by @fedepires and tested by @rhavenn supposedly fixes it, there is another alternative fix, potentially more suitable one, because the fix below restores the behaviour for handling all pchanges data from different states, whereas the fix above, only restores the changes displayed within the 'file' module. That fix also introduces the 'mixing' of 'changes' and 'pchanges', which people are concerned with above. However the fix below only affects the output produced (choosing to display pchanges if it is available).

The code in highstate.py, used to print 'pchanges' if it was present and there were no 'changes' but there are 'pchanges'.

This handling was removed in commit 98cfa1f "recursion for orchestration" (file salt/output/highstate.py, line 226), and may have simply been an oversight.

this fix is as follows (simply restoring the removed logic) from that commit

Around line 246 of salt/output/highstate.py, in salt version 2019.2.0, within function _format_host :

    schanged, ctext = _format_changes(ret['changes'])
    + if not ctext and 'pchanges' in ret:
    +     schanged, ctext = _format_changes(ret['pchanges'])
    nchanges += 1 if schanged else 0

@madrisan
Copy link
Contributor

@madrisan madrisan commented Mar 21, 2019

The @cjsin 's patch works perfectly for me. Thanks!

@dpkirchner
Copy link

@dpkirchner dpkirchner commented Apr 5, 2019

Can confirm, this works great. Is there a plan to merge this change into master? This was hands down the most valuable feature of salt (and of configuration management tools in general) so it would be great to have it back without manually patching.

@waynew
Copy link
Contributor

@waynew waynew commented Apr 5, 2019

@dpkirchner From what I can tell this should be landing in 2019.2.1, but @garethgreenaway can correct me if I'm wrong.

@garethgreenaway
Copy link
Member

@garethgreenaway garethgreenaway commented Apr 5, 2019

Correct, #51992 is labeled to be merged for the 2019.2.1 release.

@sagetherage
Copy link
Collaborator

@sagetherage sagetherage commented Jun 9, 2020

@marceliq can you give your set up and Salt version report, please?

@marceliq
Copy link

@marceliq marceliq commented Jun 18, 2020

@sagetherage sorry for late answer, here it is:

Salt is installed from pip.

salt-call --versions-report
Salt Version:
           Salt: 2019.2.5
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: 17.4.0
       dateutil: 2.8.1
      docker-py: 1.10.6
          gitdb: 2.0.6
      gitpython: 2.1.15
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.32.0
           Mako: 1.1.0
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: 3.9.4
         pygit2: Not Installed
         Python: 2.7.17 (default, May 14 2020, 06:37:18)
   python-gnupg: 0.4.6
         PyYAML: 5.2
          PyZMQ: 18.1.1
           RAET: Not Installed
          smmap: 3.0.4
        timelib: 0.2.4
        Tornado: 5.1.1
            ZMQ: 4.3.2
 
System Versions:
           dist: centos 7.7.1908 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

Note to mention, that file.serialize shows diffs, but file.manage dont.

@sagetherage
Copy link
Collaborator

@sagetherage sagetherage commented Jun 18, 2020

@marceliq what state are you using and where exactly are you see this happening? I want to gather as much information as I can so we can really narrow down where this is still occurring and get it corrected. Thank you!

@sagetherage
Copy link
Collaborator

@sagetherage sagetherage commented Dec 8, 2020

removing this from a release cycle until unblocked

@wwalker
Copy link

@wwalker wwalker commented Dec 16, 2020

@sagetherage what is the blocker on this bug? I would love to see it fixed. Anything I can do to help unBlock it?

@oeuftete
Copy link
Contributor

@oeuftete oeuftete commented Dec 16, 2020

@wwalker @sagetherage I'm going to dig in here to sort out where the issue stands.

@oeuftete oeuftete assigned oeuftete and unassigned sagetherage Dec 16, 2020
@oeuftete
Copy link
Contributor

@oeuftete oeuftete commented Jan 4, 2021

AFAICT, this is resolved in minions ≥ 2019.2.1, as noted in #51932 (comment).

It's possible the issue is still being seen based on an assumption that the master version matters here. It does not, the behaviour here is fully driven by the minions. Here's an example with a 3002.2 master with a 2018.3.5 minion (old) and a 3002.2 minion (master1).

# salt '*' state.apply gh51932.test test=True
old:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
     Started: 23:35:59.480332
    Duration: 12.141 ms
     Changes:   

Summary for old
------------
Succeeded: 1 (unchanged=1)
Failed:    0
------------
Total states run:     1
Total run time:  12.141 ms

master1:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 23:36:00.331591
    Duration: 8.46 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -dest
                  +source

Summary for master1
------------
Succeeded: 1 (unchanged=1, changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   8.460 ms

@oeuftete oeuftete closed this Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet