Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd_cache : Not working for mine #49281

Closed
aarnaud opened this issue Aug 23, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@aarnaud
Copy link
Contributor

commented Aug 23, 2018

Description of Issue/Question

Setup

Cluster of 3 etcd and 3 masters
/etc/salt/master.d/03-etcd.conf

etcd.host: 127.0.0.1
etcd.port: 2379
etcd.protocol: https
etcd.allow_reconnect: True
etcd.read_timeout: 60
etcd.cert: /etc/etcd/tmp.pem
etcd.ca_cert: /etc/etcd/ca.pem
cache: etcd

Steps to Reproduce Issue

salt * mine.update

Master log :

salt-master[3903]: [ERROR   ] Error in function _mine:
salt-master[3903]: Traceback (most recent call last):
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1776, in run_func
salt-master[3903]:     ret = getattr(self, func)(load)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1345, in _mine
salt-master[3903]:     return self.masterapi._mine(load, skip_verify=True)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/daemons/masterapi.py", line 607, in _mine
salt-master[3903]:     data = self.cache.fetch(cbank, ckey)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/cache/__init__.py", line 180, in fetch
salt-master[3903]:     return self.modules[fun](bank, key, **self._kwargs)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/cache/etcd_cache.py", line 150, in fetch
salt-master[3903]:     etcd_key, exc
salt-master[3903]: SaltCacheError: There was an error reading the key, /salt_cache/minions/ca.aws-cc1.dev.xxx.yyy.com/mine: Key not found : /salt_cache/minions/ca.aws-cc1.dev.xxx.yyy.com/mine
salt-master[3903]: [ERROR   ] Error in function _mine_get:
salt-master[3903]: Traceback (most recent call last):
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1776, in run_func
salt-master[3903]:     ret = getattr(self, func)(load)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/master.py", line 1331, in _mine_get
salt-master[3903]:     return self.masterapi._mine_get(load, skip_verify=True)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/daemons/masterapi.py", line 589, in _mine_get
salt-master[3903]:     fdata = self.cache.fetch('minions/{0}'.format(minion), 'mine')
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/cache/__init__.py", line 180, in fetch
salt-master[3903]:     return self.modules[fun](bank, key, **self._kwargs)
salt-master[3903]:   File "/usr/lib/python2.7/dist-packages/salt/cache/etcd_cache.py", line 150, in fetch
salt-master[3903]:     etcd_key, exc
salt-master[3903]: SaltCacheError: There was an error reading the key, /salt_cache/minions/ca.aws-cc1.dev.xxx.yyy.com/mine: Key not found : /salt_cache/minions/ca.aws-cc1.dev.xxx.yyy.com/mine

Versions Report

Salt Version:
Salt: 2018.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: 0.26.0
libnacl: Not Installed
M2Crypto: 0.27.0
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.2
Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 17.1.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 5.1
ZMQ: 4.2.5

System Versions:
dist: Ubuntu 18.04 bionic
locale: ANSI_X3.4-1968
machine: x86_64
release: 4.15.0-1019-aws
system: Linux
version: Ubuntu 18.04 bionic

python-etcd==0.4.5
etcd: v3.3.5

@aarnaud

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2018

There is issue in this case:

First before store something salt try to read the cache, we can fix that with:

def fetch(bank, key):
    '''
    Fetch a key value.
    '''
    _init_client()
    etcd_key = '{0}/{1}/{2}'.format(path_prefix, bank, key)
    try:
        value = client.get(etcd_key).value
        if value is None:
            return {}
        return __context__['serial'].loads(value)
    except etcd.EtcdKeyNotFound:  # Add this line
       return {}                  # Add this line
    except Exception as exc:
        raise SaltCacheError(
            'There was an error reading the key, {0}: {1}'.format(
                etcd_key, exc
            )
        )

Second:
It's seem binary data is not supported well with python-etcd or etcd
We can fix with base64 encode/decode

aarnaud added a commit to aarnaud/salt that referenced this issue Aug 23, 2018

Fix saltstack#49281 etcd_cache with mine cache
Signed-off-by: Anthony ARNAUD <github@anthony-arnaud.fr>

aarnaud added a commit to aarnaud/salt that referenced this issue Aug 23, 2018

Fix saltstack#49281 etcd_cache with mine cache
Signed-off-by: Anthony ARNAUD <github@anthony-arnaud.fr>
@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Aug 24, 2018

thanks for the PR! Since the PR resolves the issue i will go ahead and close this issue and we can keep the conversation in the PR but please let me know if you need to re-open this :)

@Ch3LL Ch3LL closed this Aug 24, 2018

@Ch3LL Ch3LL added this to the Approved milestone Aug 24, 2018

rallytime added a commit that referenced this issue Aug 31, 2018

Merge pull request #49283 from aarnaud/fix-etcd-cache
Fix #49281 etcd_cache with mine cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.