Packages DB refresh leads to all packages definitions erased on windows minion

[phil123456]

Description of Issue/Question

I tried to add my own package definition (winrar), but as I try to install it on minion, => package cannot be found in repository

digging further in the windows minion files

C:\salt\var\cache\salt\minion\files\base\win

only contains repo-ng that contains winrepo.p

and if I one it with an hex editor, it only contains character 0x80 (???)

it seems de db refresh erased all other packages, I refreshed the db without my winrar sls but still the same
all packages are GONE !

the minion log does not give any information about the package except it cannot find it

let me know if you need more infos

Setup

winrar:
  '5.01':
    full_name: 'WinRAR 5.01 (32-bit)'
    installer: 'salt://windows/packages/winrar-x64-531fr.exe'
    install_flags: '/s'
    uninstaller: 'C:\Program Files (x86)\WinRAR\uninstall.exe'
    uninstall_flags: '/s'
    msiexec: False
    locale: en_US
    reboot: False
  latest:
    full_name: 'WinRAR 5.31 (64-bit)'
    installer: 'salt://windows/packages/winrar-x64-531fr.exe'
    install_flags: '/s'
    uninstaller: 'C:\Program Files\WinRAR\uninstall.exe'
    uninstall_flags: '/s'
    msiexec: False
    locale: en_US
    reboot: False

Steps to Reproduce Issue

sudo salt-run winrepo.genrepo
event:
    ----------
    error:
        Failed to compile /srv/salt/windows_packages/repo/salt-winrepo/duplicati.sls.
suffix:
    progress
event:
    ----------
    error:
        Failed to compile /srv/salt/windows_packages/repo/salt-winrepo/duplicati_x86.sls.
suffix:
    progress
[ERROR   ] Rendering exception occurred: Jinja variable 'dict object' has no attribute 'os'
name_map:
    ----------
    7-Zip 16.00:
        7zip_x86
...
   winrar:
        ----------
        5.01:
            ----------
            full_name:
                WinRAR 5.01 (32-bit)
            install_flags:
                /s
            installer:
                salt://windows/packages/winrar-x32-501fr.exe
            locale:
                en_US
            msiexec:
                False
            reboot:
                False
            uninstall_flags:
                /s
            uninstaller:
                C:\Program Files (x86)\WinRAR\uninstall.exe
        latest:
            ----------
            full_name:
                WinRAR 5.31 (64-bit)
            install_flags:
                /s
            installer:
                salt://windows/packages/winrar-x64-531fr.exe
            locale:
                en_US
            msiexec:
                False
            reboot:
                False
            uninstall_flags:
                /s
            uninstaller:
                C:\Program Files\WinRAR\uninstall.exe
....

then I did a refresh that seemed successfull

salt -G 'os:windows' pkg.refresh_db

on the minion:

2016-07-06 11:14:37 [salt.minion    ][INFO    ] Starting a new job with PID 7448
2016-07-06 11:14:37 [salt.fileclient][INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
2016-07-06 11:14:39 [salt.minion    ][INFO    ] Returning information for job: 20160706111616222028
2016-07-06 11:14:42 [salt.minion    ][INFO    ] Starting a new job with PID 1564
2016-07-06 11:14:42 [salt.minion    ][INFO    ] Returning information for job: 20160706111621272031

pkg.available_version does not even show other packages before crashing:

C:\>salt-call pkg.available_version winrar
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
IndexError: pop from empty list
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-call", line 11, in <module>
        salt_call()
  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 345, in salt_call
    client.run()
  File "C:\salt\bin\lib\site-packages\salt\cli\call.py", line 58, in run
    caller.run()
  File "C:\salt\bin\lib\site-packages\salt\cli\caller.py", line 134, in run
    ret = self.call()
  File "C:\salt\bin\lib\site-packages\salt\cli\caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 97, in latest_version
    latest_available = _get_latest_pkg_version(pkg_info)
  File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 1099, in _get_latest_pkg_version
    return sorted(pkginfo, cmp=_reverse_cmp_pkg_versions).pop()
IndexError: pop from empty list
Traceback (most recent call last):
  File "C:\salt\bin\Scripts\salt-call", line 11, in <module>
    salt_call()
  File "C:\salt\bin\lib\site-packages\salt\scripts.py", line 345, in salt_call
    client.run()
  File "C:\salt\bin\lib\site-packages\salt\cli\call.py", line 58, in run
    caller.run()
  File "C:\salt\bin\lib\site-packages\salt\cli\caller.py", line 134, in run
    ret = self.call()
  File "C:\salt\bin\lib\site-packages\salt\cli\caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 97, in latest_version
    latest_available = _get_latest_pkg_version(pkg_info)
  File "C:\salt\bin\lib\site-packages\salt\modules\win_pkg.py", line 1099, in _get_latest_pkg_version
    return sorted(pkginfo, cmp=_reverse_cmp_pkg_versions).pop()
IndexError: pop from empty list

Versions Report

Salt Version:
Salt: 2016.3.1

Dependency Versions:
cffi: 0.8.6
cherrypy: Not Installed
dateutil: 2.2
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
Jinja2: 2.7.3
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.2
mysql-python: 1.2.3
pycparser: 2.10
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.9 (default, Mar 1 2015, 12:57:24)
python-gnupg: 0.3.6
PyYAML: 3.11
PyZMQ: 14.4.0
RAET: Not Installed
smmap: 0.8.2
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: debian 8.5
machine: x86_64
release: 3.16.0-4-amd64
system: Linux
version: debian 8.5

[Ch3LL]

@phil123456 i'm wondering if you ran into this issue that is explained in the windows software repository docs;

If you have customized software definition files that aren't maintained in a repository, those should be stored under win/repo for older minions and win/repo-ng for newer minions. The reason for this is that the contents of win/repo/salt-winrepo and win/repo-ng/salt-winrepo-ng are wiped out every time you run a winrepo.update_git_repos.

Additionally, when you run winrepo.genrepo and pkg.refresh_db the entire contents under win/repo and win/repo-ng, to include all subdirectories, are used to create the msgpack file.

Does the above information help? Looks like it needs to be located in a particular directory.

[phil123456]

no it does not help at all, there is clearly something broken in the db_refresh command

[phil123456]

can I have an update on this
do you need more infos ?
I really cannot push any windows package wich renders salt copletely useles on windows 7 minions

[phil123456]

hello ?

[morganwillcock]

@phil123456 The documentation says to put it in a specific location and your logs show it's trying to use this location.

2016-07-06 11:14:37 [salt.fileclient][INFO    ] Caching directory 'win/repo-ng/' for environment 'base'

Your sls files look to be elsewhere.

/srv/salt/windows_packages/repo/salt-winrepo/duplicati_x86.sls

Also, you don't need to use winrepo.genrepo if you are using repo-ng.

[thatch45]

@UtahDave @twangboy can you please take a look here?

[twangboy]

Please read the documentation at: https://docs.saltstack.com/en/latest/topics/windows/windows-package-manager.html

Assuming you haven't customized winrepo settings in the master config:

• Your master needs to be 2015.8.0 or newer
• On your master, run salt-run winrepo.update_git_repos to pull community definitions from git.
• Place your custom .sls files on your master in /srv/salt/win/repo-ng. They can be in subfolders or not. Everything in that folder will be used to create the winrepo.p on the minion.
• run salt -G 'os:windows' pkg.refresh_db to copy down .sls files to the minion and generate winrepo.p

[phil123456]

I have done all these... I am saying there is a bug in the packages system

• it used to list available packages then crashes with "pop from empty list" when searching for a non existing package
• now it crashes from the start with the same error

all I changed was the repos directory in the master config file to /srv/salt/windows_packages/

[twangboy]

@phil123456 Try setting that back to default and using the default paths...

[phil123456]

sure, but still it should work so this should be considered as a bug to fix

[morganwillcock]

Did you set winrepo_dir_ng (not winrepo_dir) in the master config file?

[phil123456]

winrepo_dir_ng: '/srv/salt/windows_packages/repo-ng'

[terminalmage]

The traceback has beeen fixed in #35471

[phil123456]

so do I download the exe and reinstall the minion on windows ?
I guess the empty repository issue has not beeing constated/fixed

[twangboy]

@phil123456 The patch will be in the next release of salt.