New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utils/boto and utils/boto3 load error #47694

Closed
thuhak opened this Issue May 17, 2018 · 10 comments

Comments

Projects
None yet
4 participants
@thuhak

thuhak commented May 17, 2018

Description of Issue/Question

These errors appear when i run state module

[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: 'module' object has no attribute '__version__'
[WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.

The source code:

utils/boto3.py

def __virtual__():
    '''
    Only load if boto libraries exist and if boto libraries are greater than
    a given version.
    '''
    return salt.utils.versions.check_boto_reqs()

utils/versions.py

def check_boto_reqs(boto_ver=None,
                    boto3_ver=None,
                    botocore_ver=None,
                    check_boto=True,
                    check_boto3=True):
    '''
    Checks for the version of various required boto libs in one central location. Most
    boto states and modules rely on a single version of the boto, boto3, or botocore libs.
    However, some require newer versions of any of these dependencies. This function allows
    the module to pass in a version to override the default minimum required version.

    This function is useful in centralizing checks for ``__virtual__()`` functions in the
    various, and many, boto modules and states.

    boto_ver
        The minimum required version of the boto library. Defaults to ``2.0.0``.

    boto3_ver
        The minimum required version of the boto3 library. Defaults to ``1.2.6``.

    botocore_ver
        The minimum required version of the botocore library. Defaults to ``1.3.23``.

    check_boto
        Boolean defining whether or not to check for boto deps. This defaults to ``True`` as
        most boto modules/states rely on boto, but some do not.

    check_boto3
        Boolean defining whether or not to check for boto3 (and therefore botocore) deps.
        This defaults to ``True`` as most boto modules/states rely on boto3/botocore, but
        some do not.
    '''
    if check_boto is True:
        try:
            # Late import so we can only load these for this function
            import boto
            has_boto = True
        except ImportError:
            has_boto = False

        if boto_ver is None:
            boto_ver = '2.0.0'

        if not has_boto or version_cmp(boto.__version__, boto_ver) == -1:        #<=== exception came from here 
            return False, 'A minimum version of boto {0} is required.'.format(boto_ver)

    if check_boto3 is True:
        try:
            # Late import so we can only load these for this function
            import boto3
            import botocore
            has_boto3 = True
        except ImportError:
            has_boto3 = False

        # boto_s3_bucket module requires boto3 1.2.6 and botocore 1.3.23 for
        # idempotent ACL operations via the fix in https://github.com/boto/boto3/issues/390
        if boto3_ver is None:
            boto3_ver = '1.2.6'
        if botocore_ver is None:
            botocore_ver = '1.3.23'

        if not has_boto3 or version_cmp(boto3.__version__, boto3_ver) == -1:
            return False, 'A minimum version of boto3 {0} is required.'.format(boto3_ver)
        elif version_cmp(botocore.__version__, botocore_ver) == -1:
            return False, 'A minimum version of botocore {0} is required'.format(botocore_ver)

    return True

It seems absolute_import did not work, check_boto_reqs function loaded the wrong module which is utils/boto.py

These bug will be fixed if the boto package is installed or global variable __version__ is added in utils/boto.py

Versions Report

Salt Version:
Salt: 2018.3.0

Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.1
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.21.1.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented May 17, 2018

Is this installed on a new machine, or upgraded from a previous version?

Can you verify that there are no .pyc files left over?

Because this should stop at if not has_boto because that will be true, and then it will just execute.

>>> has_boto = True
>>> class Whatever(object):
...     pass
...
>>> this = Whatever()
>>> has_boto or this.those
True
>>> this.those
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Whatever' object has no attribute 'those'

Or if you could drop in a log.critical(has_boto) and a log.critical(boto.file) so we can see if has_boto is true or false, and then also what the boto module is, that would also be helpful.

Thanks
Daniel

@gtmanfred gtmanfred added this to the Blocked milestone May 17, 2018

@thuhak

This comment has been minimized.

thuhak commented May 18, 2018

It's upgraded.
And I did clean all pyc files in utils directory. But that did not help.

I add some code to where the exception come from, so that you can see it clearly.

def check_boto_reqs(boto_ver=None,
                    boto3_ver=None,
                    botocore_ver=None,
                    check_boto=True,
                    check_boto3=True):
    '''
    Checks for the version of various required boto libs in one central location. Most
    boto states and modules rely on a single version of the boto, boto3, or botocore libs.
    However, some require newer versions of any of these dependencies. This function allows
    the module to pass in a version to override the default minimum required version.

    This function is useful in centralizing checks for ``__virtual__()`` functions in the
    various, and many, boto modules and states.

    boto_ver
        The minimum required version of the boto library. Defaults to ``2.0.0``.

    boto3_ver
        The minimum required version of the boto3 library. Defaults to ``1.2.6``.

    botocore_ver
        The minimum required version of the botocore library. Defaults to ``1.3.23``.

    check_boto
        Boolean defining whether or not to check for boto deps. This defaults to ``True`` as
        most boto modules/states rely on boto, but some do not.

    check_boto3
        Boolean defining whether or not to check for boto3 (and therefore botocore) deps.
        This defaults to ``True`` as most boto modules/states rely on boto3/botocore, but
        some do not.
    '''
    if check_boto is True:
        try:
            # Late import so we can only load these for this function
            import boto
            has_boto = True
        except ImportError:
            has_boto = False

        if boto_ver is None:
            boto_ver = '2.0.0'

        try:
            if not has_boto or version_cmp(boto.__version__, boto_ver) == -1:
                return False, 'A minimum version of boto {0} is required.'.format(boto_ver)
        except:
            raise ValueError('has_boto: {}, boto file: {}'.format(has_boto, boto.__file__))

    if check_boto3 is True:
        try:
            # Late import so we can only load these for this function
            import boto3
            import botocore
            has_boto3 = True
        except ImportError:
            has_boto3 = False

        # boto_s3_bucket module requires boto3 1.2.6 and botocore 1.3.23 for
        # idempotent ACL operations via the fix in https://github.com/boto/boto3/issues/390
        if boto3_ver is None:
            boto3_ver = '1.2.6'
        if botocore_ver is None:
            botocore_ver = '1.3.23'

        if not has_boto3 or version_cmp(boto3.__version__, boto3_ver) == -1:
            return False, 'A minimum version of boto3 {0} is required.'.format(boto3_ver)
        elif version_cmp(botocore.__version__, botocore_ver) == -1:
            return False, 'A minimum version of botocore {0} is required'.format(botocore_ver)

    return True

Run 'salt-call state.apply'

The corresponding errors are:

[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto3. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.utils.boto3.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.utils.boto. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.utils.boto.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto3_sns. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto3_sns.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto3_sns', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_asg. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_asg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_asg', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cfn. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_cfn.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cfn', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudfront. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_cloudfront.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudfront', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_cloudwatch. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_cloudwatch.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_cloudwatch', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_dynamodb. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_dynamodb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_dynamodb', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_ec2. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_ec2.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_ec2', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elasticache. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_elasticache.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elasticache', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elb. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_elb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elb', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_elbv2. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_elbv2.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_elbv2', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_iam. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_iam.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_iam', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_kinesis. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_kinesis.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_kinesis', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_kms. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_kms.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_kms', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_route53. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_route53.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_route53', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_secgroup. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_secgroup.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_secgroup', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_sns. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_sns.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_sns', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.module.boto_sqs. Module will not be loaded: has_boto: True, boto file: /usr/lib/python2.7/site-packages/salt/utils/boto.pyc
[WARNING ] salt.loaded.int.module.boto_sqs.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'boto_sqs', please fix this.
@thuhak

This comment has been minimized.

thuhak commented May 18, 2018

By the way,there is absolute_import in utils/versions.py

from __future__ import absolute_import, print_function, unicode_literals

And I printed the sys.path variable, /usr/lib/python2.7/site-packages/salt/utils is not in that list.

I don't know why this happened.

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented May 18, 2018

can you do a find /usr/lib/python2.7/site-packages/salt -name \*.pyc -delete just to make sure, and restart?

@saltstack/team-core does anyone know why it would be importing salt.utils.boto when 'absolute_import` is turned on?

Thanks,
Daniel

@thuhak

This comment has been minimized.

thuhak commented May 18, 2018

I tried, restarted minion.
The problem still occurs

@terminalmage

This comment has been minimized.

Member

terminalmage commented May 18, 2018

@gtmanfred The loader appends the module dir to the end of sys.path temporarily. So therefore, if boto isn't installed, doing an import boto would import salt/utils/boto.py.

@terminalmage

This comment has been minimized.

Member

terminalmage commented May 18, 2018

We may need to rename the boto utils modules (which we could do while keeping the original virtualname to preserve __utils__['boto.whatever'] calls in the codebase).

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented May 18, 2018

@terminalmage

This comment has been minimized.

Member

terminalmage commented May 18, 2018

No problem. Remember when renaming the modules that you'll want to rename any corresponding test modules to preserve the naming convention.

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented May 18, 2018

I have opened #47726 to fix this issue.

Thanks for reporting,
Daniel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment