Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REGRESSION] --output-diff doesn't display changes anymore with test=True since v2019.2.0 #51932

Closed
tomlaredo opened this issue Mar 1, 2019 · 44 comments
Assignees
Labels
Bug broken, incorrect, or confusing behavior info-needed waiting for more info Regression The issue is a bug that breaks functionality known to work in previous releases. severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around ZD The issue is related to a Zendesk customer support ticket.
Milestone

Comments

@tomlaredo
Copy link

tomlaredo commented Mar 1, 2019

Description of Issue/Question

--output-diff doesn't display changes anymore with test=True since v2019.2.0

Setup

# cat state.sls 
/tmp/dest_file.txt:
    file.managed:
        - source: /tmp/source_file.txt

Steps to Reproduce Issue

Initial setup

echo new_content > /tmp/source_file.txt
echo old_content > /tmp/dest_file.txt

With test=True

# salt-call state.apply state --output-diff test=True
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:21.779602
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] The file /tmp/dest_file.txt is set to be changed
[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:21.797227 (duration_in_ms=17.626)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
     Started: 16:36:21.779601
    Duration: 17.626 ms
     Changes:   

Summary for local
------------
Succeeded: 1 (unchanged=1)
Failed:    0
------------
Total states run:     1
Total run time:  17.626 ms

Diff isn't displayed anymore.

Without test=True

vs-salt-master:/srv/salt# salt-call state.apply state --output-diff
[INFO    ] Got list of available master addresses: [u'salt-master', u'localhost']
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/dest_file.txt] at time 16:36:29.781272
[INFO    ] Executing state file.managed for [/tmp/dest_file.txt]
[INFO    ] File changed:
--- 
+++ 
@@ -1 +1 @@
-old_content
+new_content

[INFO    ] Completed state [/tmp/dest_file.txt] at time 16:36:29.800569 (duration_in_ms=19.296)
local:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/dest_file.txt updated
     Started: 16:36:29.781273
    Duration: 19.296 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -old_content
                  +new_content

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  19.296 ms

Diff is displayed.

Versions Report

# salt-call --versions-report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.8 
         locale: UTF-8
        machine: x86_64
        release: 4.15.18-9-pve
         system: Linux
        version: debian 9.8 
@garethgreenaway garethgreenaway added this to the Blocked milestone Mar 4, 2019
@garethgreenaway garethgreenaway added the Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged label Mar 4, 2019
@garethgreenaway
Copy link
Contributor

@saltstack/team-core Thoughts?

@thatch45
Copy link
Contributor

thatch45 commented Mar 4, 2019

We have gone back and forth on this one a lot over the years. The core argument has been that the changes dict should only have changes in it if changes actually happened. Thats why it keeps getting pulled out. The main argument is that this is the expected behavior.

The counter argument is that this is some very valuable information to have from a test=True run and that it should be made available.

The counter/counter to that is that we don't have the ability to get predictive changes out of all states, therefore the output becomes inconsistent.

Finally, it can also be argued that if the result is None then we should know for sure that we did not make changes, so any changes in the changes dict can be seen as predictive.

@garethgreenaway
Copy link
Contributor

Brief discussion between @thatch45 @waynew and myself, we concluded that the diff should be shown and we'll look to see what changed from 2018.3.x to 2019.2.x to disable this.

@thatch45
Copy link
Contributor

thatch45 commented Mar 4, 2019

Agreed, and when we get it back, lets add a fat comment in there to keep it :)

@garethgreenaway garethgreenaway self-assigned this Mar 5, 2019
@tomlaredo
Copy link
Author

tomlaredo commented Mar 5, 2019

Thanks for those informations and for your reactivity.

Even if test=True isn't perfect and could miss some changes (generally due to triggers like watch / watch_in etc.), it is a huge help for the one who needs to check some particular changes instead of blindly applying states and just hope changes will be good.
A note could be displayed on test=True telling the user that changes are juste there for information and can be wrong in certain cases.

@waynew
Copy link
Contributor

waynew commented Mar 5, 2019

I'm a fan of adding a note in there, maybe something like Comment: File /tmp/dest_file.txt updated. Note: Actual changes may be different due to other states.

@garethgreenaway garethgreenaway added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around severity-critical top severity, seen by most users, serious issues P2 Priority 2 v2019.2.1 unsupported version and removed Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged labels Mar 5, 2019
@thatch45
Copy link
Contributor

thatch45 commented Mar 5, 2019

I like it, maybe say:
Note: No changes made, actual changes may be different due to other states.

@ghost
Copy link

ghost commented Mar 5, 2019

If it's of any help, from our tests, looks like this commit might be the culprit:

98c61ae

#44353

@rhavenn
Copy link

rhavenn commented Mar 5, 2019

I would only add that a) This is a major feature when trying to merge a Salt state into systems that have not previously been managed by Salt and have years of manual changes in them. One never knows what setting you're going to be accidentally removing. b) If test=true isn't the grammitically correct way this is envisioned then adding a state.test that does the same thing is perhaps more grammatically correct?

@rhavenn
Copy link

rhavenn commented Mar 5, 2019

I can confirm reverting the change @fedepires mentions and restarting my remote minion "fixes" the issue and shows the diffs when running state.apply with test=true

@waynew
Copy link
Contributor

waynew commented Mar 6, 2019

@fedepires Nice find :) (and helpful for those who need it.

@rhavenn At least from those involved in the minor discussion we had, we didn't have strong feelings about calling it something else. My cursory search didn't find anything to the contrary. I'd say that my personal preference would be to fix this back (with the aforementioned comment), with the idea that test=True is basically equivalent to git's dry-run flag.

Maybe if someone really feels strongly about the semantics/grammar they could submit a PR to introduce a dry-run flag and re-un-restore the behavior for test=True :)

@tomlaredo
Copy link
Author

tomlaredo commented Mar 6, 2019

To add a last thought to the @rhavenn 's comment, currently, if you misspell "test=True", the state will be executed without any error or message telling that we passed an unknown argument, which is very dangerous and confusing.

Then, replacing test=True by a special state.test (or maybe state.dryrun) would be a good thing I think,

@cjsin
Copy link

cjsin commented Mar 19, 2019

Hello,

I agree that this is a terrible regression, it makes it impossible to actually see what changes are going to be made in test mode, almost defeating the entire purpose of the test mode (If you can't see potential changes, then the test mode becomes useful only as a syntax test, to check for errors in the states). We use test mode to capture differences in a nightly audit also which has now become useless without the details of the differences.

Although the change described by @fedepires and tested by @rhavenn supposedly fixes it, there is another alternative fix, potentially more suitable one, because the fix below restores the behaviour for handling all pchanges data from different states, whereas the fix above, only restores the changes displayed within the 'file' module. That fix also introduces the 'mixing' of 'changes' and 'pchanges', which people are concerned with above. However the fix below only affects the output produced (choosing to display pchanges if it is available).

The code in highstate.py, used to print 'pchanges' if it was present and there were no 'changes' but there are 'pchanges'.

This handling was removed in commit 98cfa1f "recursion for orchestration" (file salt/output/highstate.py, line 226), and may have simply been an oversight.

this fix is as follows (simply restoring the removed logic) from that commit

Around line 246 of salt/output/highstate.py, in salt version 2019.2.0, within function _format_host :

    schanged, ctext = _format_changes(ret['changes'])
    + if not ctext and 'pchanges' in ret:
    +     schanged, ctext = _format_changes(ret['pchanges'])
    nchanges += 1 if schanged else 0

@madrisan
Copy link
Contributor

The @cjsin 's patch works perfectly for me. Thanks!

@dpkirchner
Copy link

Can confirm, this works great. Is there a plan to merge this change into master? This was hands down the most valuable feature of salt (and of configuration management tools in general) so it would be great to have it back without manually patching.

@waynew
Copy link
Contributor

waynew commented Apr 5, 2019

@dpkirchner From what I can tell this should be landing in 2019.2.1, but @garethgreenaway can correct me if I'm wrong.

@garethgreenaway
Copy link
Contributor

Correct, #51992 is labeled to be merged for the 2019.2.1 release.

@sagetherage
Copy link
Contributor

@marceliq can you give your set up and Salt version report, please?

@marceliq
Copy link

@sagetherage sorry for late answer, here it is:

Salt is installed from pip.

salt-call --versions-report
Salt Version:
           Salt: 2019.2.5
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: 17.4.0
       dateutil: 2.8.1
      docker-py: 1.10.6
          gitdb: 2.0.6
      gitpython: 2.1.15
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.32.0
           Mako: 1.1.0
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: 3.9.4
         pygit2: Not Installed
         Python: 2.7.17 (default, May 14 2020, 06:37:18)
   python-gnupg: 0.4.6
         PyYAML: 5.2
          PyZMQ: 18.1.1
           RAET: Not Installed
          smmap: 3.0.4
        timelib: 0.2.4
        Tornado: 5.1.1
            ZMQ: 4.3.2
 
System Versions:
           dist: centos 7.7.1908 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

Note to mention, that file.serialize shows diffs, but file.manage dont.

@sagetherage
Copy link
Contributor

@marceliq what state are you using and where exactly are you see this happening? I want to gather as much information as I can so we can really narrow down where this is still occurring and get it corrected. Thank you!

@sagetherage sagetherage added severity-high 2nd top severity, seen by most users, causes major problems info-needed waiting for more info Magnesium Mg release after Na prior to Al and removed severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around ZRelease-Sodium retired label labels Jun 18, 2020
@sagetherage sagetherage modified the milestones: Approved, Magnesium, Blocked Jul 14, 2020
@sagetherage sagetherage removed this from Done in Sodium Jul 14, 2020
@sagetherage sagetherage removed the Magnesium Mg release after Na prior to Al label Oct 20, 2020
@sagetherage sagetherage added Aluminium Release Post Mg and Pre Si and removed Aluminium Release Post Mg and Pre Si labels Dec 7, 2020
@sagetherage
Copy link
Contributor

removing this from a release cycle until unblocked

@sagetherage sagetherage added severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around and removed severity-high 2nd top severity, seen by most users, causes major problems labels Dec 8, 2020
@sagetherage sagetherage self-assigned this Dec 10, 2020
@wwalker
Copy link

wwalker commented Dec 16, 2020

@sagetherage what is the blocker on this bug? I would love to see it fixed. Anything I can do to help unBlock it?

@oeuftete oeuftete added CS-R2 ZD The issue is related to a Zendesk customer support ticket. labels Dec 16, 2020
@oeuftete
Copy link
Contributor

@wwalker @sagetherage I'm going to dig in here to sort out where the issue stands.

@oeuftete oeuftete assigned oeuftete and unassigned sagetherage Dec 16, 2020
@oeuftete
Copy link
Contributor

oeuftete commented Jan 4, 2021

AFAICT, this is resolved in minions ≥ 2019.2.1, as noted in #51932 (comment).

It's possible the issue is still being seen based on an assumption that the master version matters here. It does not, the behaviour here is fully driven by the minions. Here's an example with a 3002.2 master with a 2018.3.5 minion (old) and a 3002.2 minion (master1).

# salt '*' state.apply gh51932.test test=True
old:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
     Started: 23:35:59.480332
    Duration: 12.141 ms
     Changes:   

Summary for old
------------
Succeeded: 1 (unchanged=1)
Failed:    0
------------
Total states run:     1
Total run time:  12.141 ms

master1:
----------
          ID: /tmp/dest_file.txt
    Function: file.managed
      Result: None
     Comment: The file /tmp/dest_file.txt is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 23:36:00.331591
    Duration: 8.46 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -dest
                  +source

Summary for master1
------------
Succeeded: 1 (unchanged=1, changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   8.460 ms

@oeuftete oeuftete closed this as completed Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior info-needed waiting for more info Regression The issue is a bug that breaks functionality known to work in previous releases. severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around ZD The issue is related to a Zendesk customer support ticket.
Projects
None yet
Development

Successfully merging a pull request may close this issue.