archive.extracted unzips files with mode 000 in 2016.11

[onlyanegg]

Description of Issue/Question

archive.extracted state of zip archive sets file mode to 000 for all files in the archive on 2016.11
This works in 2016.3. It sets the directory modes to 755 and file modes to 644.

[minion]$ salt-call state.apply hornetq
local:
----------
          ID: install_hornetq
    Function: archive.extracted
        Name: /usr/local
      Result: True
     Comment: salt://hornetq/hornetq-2.1.1.zip extracted to /usr/local/
     Started: 14:54:27.643937
    Duration: 4551.343 ms
     Changes:
              ----------
              extracted_files:
                  - hornetq-2.1.1/
                  - hornetq-2.1.1/bin/
                  - hornetq-2.1.1/bin/build.bat
                  - hornetq-2.1.1/bin/build.sh
                  ...
              updated ownership:
                  True

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   4.551 s
[minion]$ ll -d hornetq-2.1.1/
d---------. 14 hornetq hornetq 4096 Dec 20 14:54 hornetq-2.1.1/
[minion]$ ll hornetq-2.1.1/
total 16
d---------. 3 hornetq hornetq 4096 Dec 20 14:54 bin
d---------. 4 hornetq hornetq   34 Dec 20 14:54 classes
d---------. 5 hornetq hornetq   71 Dec 20 14:54 config
d---------. 5 hornetq hornetq   56 Dec 20 14:54 data
d---------. 5 hornetq hornetq 4096 Dec 20 14:54 docs
d---------. 7 hornetq hornetq   80 Dec 20 14:54 examples
d---------. 2 hornetq hornetq 4096 Dec 20 14:54 lib
d---------. 2 hornetq hornetq   71 Dec 20 14:54 licenses
d---------. 2 hornetq hornetq   24 Dec 20 14:54 logs
d---------. 3 hornetq hornetq 4096 Dec 20 14:54 native-src
d---------. 2 hornetq hornetq   84 Dec 20 14:54 schemas
d---------. 3 hornetq hornetq   16 Dec 20 14:54 tools

Setup

[minion]$ grep baseurl /etc/yum.repos.d/saltstack.repo
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
[minion]$ rpm -qa | grep salt
salt-2016.11.1-1.el7.noarch
salt-minion-2016.11.1-1.el7.noarch

hornetq_state.sls

install_hornetq:
   archive.extracted:
     - name: {{ pillar.hornetq.install_dir }}
     - source: salt://hornetq/{{ pillar.hornetq.version }}.zip
     - archive_format: zip
     - if_missing: {{ pillar.hornetq.install_dir }}/{{ pillar.hornetq.version }}
     - user: {{ pillar.sys_users.hornetq.user }}
     - group: {{ pillar.sys_users.hornetq.group }}

hornetq_pillar.sls

 hornetq:
   install_dir: /usr/local
   version: 'hornetq-2.1.1'
 sys_users:
   hornetq:
     user: hornetq
     group: hornetq
     uid: 1001
     gid: 1001

Steps to Reproduce Issue

1. Install 2016.11
2. Run hornetq state

Versions Report

[master]$ salt --versions-report
Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.28.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

[minion]$ salt-call --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Sep 15 2016, 22:37:39)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.36.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

[Ch3LL]

@onlyanegg i'm having a hard time replicating this. It looks like your master is a lower version. If you upgrade the master does it then work?

Or maybe if you run it with salt-call --local on the minion do you still see hte same behavior?

Here is my state file:

extract_myapp:
  archive.extracted:
    - name: /var/www/myapp-16.2.4
    - source: salt://test.zip
    - archive_format: zip
    - user: www
    - group: www
    - if_missing: /var/www/myapp-16.2.4

And when i run salt-call --local state.sls test -ldebug I see correct permissions

[root@094cb91ffbb2 salt]# ll /var/www/myapp-16.2.4/
total 0
drwxr-xr-x. 5 www www 184 Dec 21 19:19 creatdir
[root@094cb91ffbb2 salt]# ll /var/www/myapp-16.2.4/creatdir/
total 0
-rw-r--r--. 1 www www 0 Dec 21 19:19 test1
-rw-r--r--. 1 www www 0 Dec 21 19:19 test10
-rw-r--r--. 1 www www 0 Dec 21 19:19 test2
-rw-r--r--. 1 www www 0 Dec 21 19:19 test3
-rw-r--r--. 1 www www 0 Dec 21 19:19 test4
-rw-r--r--. 1 www www 0 Dec 21 19:19 test5
-rw-r--r--. 1 www www 0 Dec 21 19:19 test6
-rw-r--r--. 1 www www 0 Dec 21 19:19 test7
-rw-r--r--. 1 www www 0 Dec 21 19:19 test8
-rw-r--r--. 1 www www 0 Dec 21 19:19 test9
drwxr-xr-x. 2 www www 6 Dec 21 19:19 testdir
drwxr-xr-x. 2 www www 6 Dec 21 19:19 testdir1
drwxr-xr-x. 2 www www 6 Dec 21 19:19 testdir2

Also here is a docker container to quickly replicate:

1. docker run -it -v /home/ch3ll/git/salt/:/testing/ ch3ll/issues:38384 (where /home/ch3ll/git/salt/ is a local cloned git repo)
2. rm -rf /var/www/myapp-16.2.4
   3.. salt-call --local state.sls test
3. ll /var/www/myapp-16.2.4/
4. ll /var/www/myapp-16.2.4/creatdir/

Also is there anything different with my test case then your issue?

[onlyanegg]

Thanks for taking a look at this @Ch3LL! It seems like this has to do with the combination of my zip archive and 2016.11. I was not able to reproduce it with your test.zip in either your docker container or my environment. When I revert to 2016.3.4, the zip is extracted with sane permissions.

[minion]$ ll -d /usr/local/hornetq-2.1.1/
drwxr-xr-x. 14 hornetq hornetq 4096 Nov 15  2011 /usr/local/hornetq-2.1.1/
[minion]$ ll /usr/local/hornetq-2.1.1/
total 16
drwxr-xr-x. 3 hornetq hornetq 4096 Feb  1  2011 bin
drwxr-xr-x. 4 hornetq hornetq   34 Jan 31  2011 classes
drwxr-xr-x. 5 hornetq hornetq   71 Jan 31  2011 config
drwxr-xr-x. 5 hornetq hornetq   56 Jan 31  2011 data
drwxr-xr-x. 5 hornetq hornetq 4096 Nov 15  2011 docs
drwxr-xr-x. 7 hornetq hornetq   80 Jan 31  2011 examples
drwxr-xr-x. 2 hornetq hornetq 4096 Jan 31  2011 lib
drwxr-xr-x. 2 hornetq hornetq   71 Jan 31  2011 licenses
drwxr-xr-x. 2 hornetq hornetq   24 Nov 15  2011 logs
drwxr-xr-x. 3 hornetq hornetq 4096 Jan 31  2011 native-src
drwxr-xr-x. 2 hornetq hornetq   84 Jan 31  2011 schemas
drwxr-xr-x. 3 hornetq hornetq   16 Jul  2  2010 tools
[minion]$ salt-call --version
salt-call 2016.3.4 (Boron)

To your question about my master version, I see this behavior in masterless configuration as well.
For the time being, I'm explicitly setting permissions after I extract the archive. I'll look into issues with my zip archive

[Ch3LL]

Okay great yeah if you can elaborate as to why our exact zip package is failing or can share the zip package so we can try to replicate this that would be great then we can dive deeper into the issue.

Might also be useful to do a git bisect to see where it broke but I can definitely do that if i can get a reproducable case. Want to get this fixed up for ya :)

[bfilipek]

I have the same issue. I have tried to extract zip archive on linux and windows.

test.sls:

test:
  archive.extracted:
    - name: '/tmp/'
    - if_missing: '/tmp/SERVER'
    - source: http://my-storage-server.com/file.zip
    - source_hash: md5=7e513cce724cfb66b5d9d197405def86
    - archive_format: zip
    - trim_output: True
    - enforce_toplevel: False

On linux looks like:

----------.  1 root root     27 Jan 11 05:27 autorun.inf
d---------.  4 root root   4096 Jan 11 05:27 folder
----------.  1 root root 151552 Jan 11 05:27 setup.exe

When i extract archive using unzip directly on machine, looks ok:

-rw-r--r--.  1 root root     27 Jan 11 05:27 autorun.inf
drwxr-xr-x.  4 root root   4096 Jan 11 05:27 folder
-rw-r--r--.  1 root root 151552 Jan 11 05:27 setup.exe

On windows files has read-only flags:

d-r--        11/01/2017     07:10            folder
-ar--        11/01/2017     07:06         69 autorun.inf
-ar--        11/01/2017     07:06        221 setup.exe
      

When i extract directly on windows:

d----        11/01/2017     07:10            folder
-a---        11/01/2017     07:06         69 autorun.inf
-a---        11/01/2017     07:06        221 setup.exe

The same situation when i set "extract_perms" to False

Salt Version:
           Salt: 2016.11.1
 
Dependency Versions:
           cffi: 0.8.2
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 4.2.0-42-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[onlyanegg]

@Ch3LL I worked around this by explicitly setting permissions with the file.directory state after extracting.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.