file.managed state should not download a file if the checksum did not change

[joelwhitehouse]

Description of Issue/Question

When using the file.managed state with a source HTTPS source url and a provided source_hash I would expect Salt to check the hash of an existing local file before downloading the file from the server. This appears to be a regression of #33831. Today, it's been eating my limited mobile data.

Setup

Salt Version: 2017.7.2 (Nitrogen) - in masterless local mode
Ubuntu 16.04 xenial

Steps to Reproduce Issue

• Create a salt state: file.managed with source: https://domain.tld/file.ext and source_hash: <hash of file>
• run salt-call state.highstate -l debug
• observe that salt's debug listing will include output like Requesting URL https://domain.tld/file.ext using GET method
• run salt-call state.highstate -l debug
• observe that salt's debug listing still includes output like Requesting URL https://domain.tld/file.ext using GET method despite the file already being present and in the correct state

Versions Report

Salt Version:
           Salt: 2017.7.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 20 2017, 18:23:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-103-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[garethgreenaway]

@terminalmage Thoughts on this one?

[terminalmage]

This is a duplicate of #38971 and was fixed several months ago in #43681.

[terminalmage]

The fix was made after the 2017.7.2 was frozen but before the release of 2017.7.2. Therefore the fix can be expected in 2017.7.3 (and the upcoming Oxygen release).

[zegl]

@terminalmage Thanks for fixing this bug. Do you have an estimate of when 2017.7.3 will be released?

[joelwhitehouse]

Another impact of this issue is that a pkg.installed state with http/s locations in sources also has its sources redownloaded every time. For example, this state:

vagrant:
  pkg.installed:
    - sources:
      - vagrant: https://releases.hashicorp.com/vagrant/2.0.1/vagrant_2.0.1_x86_64.deb

returns the following output:

          ID: vagrant
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 19:46:34.349830
    Duration: 854550.905 ms
     Changes:   
----------

That's 15 minutes just to verify that vagrant is already installed! I thought salt had completely hung but it turns out it's just this issue again.

[terminalmage]

2017.7.3 was released earlier this week.