2019.2.1/2019.2.0 pip failures even when not using pip

[Reiner030]

While testing my setup with Debian Buster and "stable" Saltstack version (2019.2.0) on a vagrant instance I run into weird problems...
Especially i didn't use pip and most of the other modules notified in the failure messages I was search for over half hour for the cause:

# salt-call state.apply
[ERROR   ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 58, in <module>
    del pip
NameError: name 'pip' is not defined
[ERROR   ] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 79, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgrepo, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgrepo.py", line 93, in <module>
    from salt.exceptions import CommandExecutionError, SaltInvocationError
  File "/usr/lib/python2.7/dist-packages/salt/exceptions.py", line 9, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.portage_config. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.portage_config.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'portage_config', please fix this.
[ERROR   ] Failed to import states ports, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/ports.py", line 21, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_cluster. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_cluster.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_cluster', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_database. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_database.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_database', please fix this.
[ERROR   ] Failed to import states postgres_extension, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_extension.py", line 19, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_group, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_group.py", line 18, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_initdb. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_initdb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_initdb', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_language. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_language.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_language', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_privileges. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_privileges.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_privileges', please fix this.
[ERROR   ] Failed to import states postgres_schema, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_schema.py", line 16, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_tablespace, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_tablespace.py", line 23, in <module>
    import salt.utils.dictupdate as dictupdate
  File "/usr/lib/python2.7/dist-packages/salt/utils/__init__.py", line 18, in <module>
    from salt.ext import six
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 728, in <module>
    print_ = getattr(moves.builtins, "print", None)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 99, in __get__
    result = self._resolve()
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 122, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 90, in _import_module
    return sys.modules[name]
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_user, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_user.py", line 17, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

Passed invalid arguments: 'NoneType' object is not callable.

Usage:

    .. versionadded:: 2015.5.0

    This function will call :mod:`state.highstate
    <salt.modules.state.highstate>` or :mod:`state.sls
    <salt.modules.state.sls>` based on the arguments passed to this function.
    It exists as a more intuitive way of applying states.

    .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTATE <running-highstate>`)

    To apply all configured states, simply run ``state.apply``:

    .. code-block:: bash

        salt '*' state.apply

    The following additional arguments are also accepted when applying all
    states configured in top.sls:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    exclude
        Exclude specific states from execution. Accepts a list of sls names, a
        comma-separated string of sls names, or a list of dictionaries
        containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
        multiple states.

        .. code-block:: bash

            salt '*' state.apply exclude=bar,baz
            salt '*' state.apply exclude=foo*
            salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply localconfig=/path/to/minion.yml


    .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.modules.state.sls>`)

    To apply individual SLS files, pass them as a comma-separated list:

    .. code-block:: bash

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
        salt '*' state.apply stuff
        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
        # and salt://pkgs.sls (or salt://pkgs/init.sls).
        salt '*' state.apply stuff,pkgs

    The following additional arguments are also accepted when applying
    individual SLS files:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    concurrent : False
        Execute state runs concurrently instead of serially

        .. warning::

            This flag is potentially dangerous. It is designed for use when
            multiple state runs can safely be run at the same time. Do *not*
            use this flag for performance optimization.

    saltenv
        Specify a salt fileserver environment to be used when applying states

        .. versionchanged:: 0.17.0
            Argument name changed from ``env`` to ``saltenv``

        .. versionchanged:: 2014.7.0
            If no saltenv is specified, the minion config will be checked for an
            ``environment`` parameter and if found, it will be used. If none is
            found, ``base`` will be used. In prior releases, the minion config
            was not checked and ``base`` would always be assumed when the
            saltenv was not explicitly set.

    pillarenv
        Specify a Pillar environment to be used when applying states. This
        can also be set in the minion config file using the
        :conf_minion:`pillarenv` option. When neither the
        :conf_minion:`pillarenv` minion config option nor this CLI argument is
        used, all Pillar environments will be merged together.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply stuff localconfig=/path/to/minion.yml

    sync_mods
        If specified, the desired custom module types will be synced prior to
        running the SLS files:

        .. code-block:: bash

            salt '*' state.apply stuff sync_mods=states,modules
            salt '*' state.apply stuff sync_mods=all

        .. note::
            This option is ignored when no SLS files are specified, as a
            :ref:`highstate <running-highstate>` automatically syncs all custom
            module types.

        .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

After wasting my nightly time I found luckily that the system has installed a new version which is officially not yet relased: 2019.2.1

And thx to the autoupdate function now also my other instances where influenced and I have to manually downgrade dist to

deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch main

Interesting also that my common vagrant setup has setup the line with stretch and not buster, but this seems because you reuse stretch packages for buster ?

AFTER 10 MINUTES of REPAIRING EACH INSTANCE caused by addition broken package dependencies I can run my states again as usual:

# salt-call state.apply
[WARNING ] /usr/lib/python2.7/dist-packages/salt/modules/mysql.py:546: Warning: (1681L, "'PASSWORD' is deprecated and will be removed in a future release.")
  return cur.execute(qry, args)

local:

Summary for local
--------------
Succeeded: 140
Failed:      0
--------------
Total states run:     140
Total run time:     9.274 s

ah... now documentation shows the buggy version as released...

[garethgreenaway]

@Reiner030 thanks for reporting this. Would you be able to provide the state file in question that you were trying to run when this error happened? Thanks!

[tomlaredo]

I'm using SaltStack for some years now and I must admit that almost EVERY release comes with important and obvious regressions showing a bunch of errors.

Yes code quality control isn't a simple thing to deal with, yes SaltStack has a lot of modules witten by many people but what could be worse than wating for months (sometimes years) that a good PR is merged, packaged and deployed on one hand and then having bad troubles every time we update SaltStack on the other?

I know that this comment is a bit rude but please understand that this is very frustrating.

#34096

[r-lindner]

this bug was likely introduced with 238fd0f
The statefile is not relevant - it crashes if pip is not installed making it a grave bug:
It happens when the apache formula is used but not if the nginx formula is used, we are looking for a better test case.

#salt/states/pip_state.py
try:
....import pip
....HAS_PIP = True
except ImportError:
....HAS_PIP = False
....# [some more lines on this level]
....del pip # pip is not imported, cannot be deleted!

[robmp]

I'm hitting this issue with states using unless: if that helps to narrow it down.

[maennlse]

we also traced it down to that states using unless: are failing...

the following state is working:

vim:
  pkg.installed:
    - name: vim-enhanced

but it fails with as soon as unless: is added:

vim:
  pkg.installed:
    - name: vim-enhanced
    - unless: /bin/bash -c false

[Orlando-Richards]

A similar issue also affects anyone using pip between versions 10.0 and 18.1, due to this:

    if salt.utils.versions.compare(ver1=pip.__version__,
                                   oper='>=',
                                   ver2='18.1'):
        from pip._internal.exceptions import InstallationError

The exceptions module was moved from pip.exceptions to pip._internal.exceptions in pip version 10.0 - so the 18.1 above should probably read 10.0.

See: #54772

[stratusjerry]

Also seeing the issue when targeting state using "Unless", as well as with "OnlyIf" logic. However the root cause seems to be related to changes to the pip_state in 2019.2.1 See issue:
#53570

[maennlse]

I can confirm that installing python2-pip (on centos 7) fixes the issue with unless: or onlyif: used...

[smitelli]

Similar to @maennlse, this is fixed on Debian Buster by installing the python-pip package before running salt-call, etc. (https://packages.debian.org/buster/python-pip)

[mamorunl]

I can also confirm that it works on Ubuntu 16.04 when installing python-pip first.