2015.8.3 pillar beacons bugged? #29451

githubcdr · 2015-12-04T21:08:03Z

Hi,

After upgrading to 2015.8.3 I get the following error spamming journald; My beacons are configured in pillar and worker in previous versions.

[CRITICAL] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1699, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 359, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

[root@flaim ~]# salt skynet.eigenhuis.lan pillar.item beacons --out=yaml
skynet.eigenhuis.lan:
  beacons:
    btmp: {}
    diskusage:
    - /: 80%
    - interval: 43200
    load:
    - 1m:
      - 2.0
      - 3.0
    - 5m:
      - 2.0
      - 3.5
    - 15m:
      - 2.1
      - 3.0
    network_info:
      eth0:
      - type: greater
      - errin: 50
      - errout: 50
      - dropin: 50
      - dropout: 50
    wtmp: {}

[root@flaim ~]# salt-minion --versions-report
Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: 1.0.1
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: 0.6.3
        Tornado: 4.2.1
            ZMQ: 4.1.2
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: 1.3.7
        libnacl: 1.4.3
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.6
   mysql-python: 1.2.5
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: 0.23.2
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: 0.2.4

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-71-generic
         system: Ubuntu 14.04 trusty

I tried changing the config and clearing the cache but the error stays.

The text was updated successfully, but these errors were encountered:

githubcdr · 2015-12-04T22:01:22Z

might be related to #27949, but this is from the documentation

jfindlay · 2015-12-04T22:16:42Z

@githubcdr, thanks for the report.

The-Loeki · 2015-12-15T12:09:28Z

Confirmed;

beacons:
  load:
    - interval: 5
    - 1m:
      - 0.0
      - 25.0
    - 5m:
      - 0.0
      - 100.0
    - 15m:
      - 0.0
      - 100.0

borks; it will be passed on to _determine_beacon_config as:

{'load': [{'interval': 5}, {'1m': [0.0, 25.0]}, {'5m': [0.0, 100.0]}, {'15m': [0.0, 100.0]}]}

Changing it to

beacons:
  load:
    interval: 5
    1m:
      - 0.0
      - 25.0
    5m:
      - 0.0
      - 100.0
    15m:
      - 0.0
      - 100.0

Works around the problem for now, passing on:

{'load': {'5m': [0.0, 100.0], '1m': [0.0, 25.0], 'interval': 5, '15m': [0.0, 100.0]}}

Which works as expected

The-Loeki · 2015-12-15T12:20:07Z

I have to say I'm not entirely sure what the exact problem is here; the first if in _determine_beacon_config tests whether beacons is a list or dict, but both conditions would bork on this construct, clearly broken.

OTOH wasn't that first syntax suppposed to deliver an ordereddict rather than a list of dicts???

fredrikaverpil · 2016-03-11T05:56:46Z

But basically, this makes it impossible to use beacons when defined in pillar (which is extremely useful). Am I right?

Has anyone found a fix or a workaround?

The-Loeki · 2016-03-11T10:47:38Z

@fredrikaverpil negative, just one of the syntaxes is broken. Use the 'alternative' syntax outlined earlier (which is the 'workaround' we use in production).

fredrikaverpil · 2016-04-04T13:55:21Z

I can't get this to generate anything in the event bus, which is from the docs:

beacons:
  diskusage:
    - /: 5%

...and changing it into the following still doesn't make anything show in the event bus:

beacons:
  diskusage:
    /: 5%

I'm viewing the event bus in order to catch any such event: salt-run state.event pretty=True and the minion has well over 5% disk usage.

asyncmind0 · 2016-05-16T22:21:15Z

the diskusage beacon expects a list of dictionaries instead of a dict, this is inconsistent with the other beacon configs, although seems to be fixed in the v2016

githubcdr · 2016-05-17T09:09:17Z

This state generates a unexpected configuration file.

diskusage:
  beacon.present:
    - /: 25%
    - interval: 20

The following file is created when using the beacons.save module, notice the list changed to a dict here.

beacons:
  diskusage:
    /: 25%
    interval: 20

End result is a non working setup, with a critical error as showed below.

May 17 11:07:38 salt-minion[24793]: [CRITICAL] The beacon errored:
May 17 11:07:38 salt-minion[24793]:     mount = diskusage.keys()[0]
May 17 11:07:38 salt-minion[24793]: AttributeError: 'str' object has no attribute 'keys'

This makes the diskusage beacon as described in the docs practically unusable :(

fredrikaverpil · 2016-05-24T14:09:37Z

I get errors in either case on 2015.8.8.2 with diskusage.beacon.

# /srv/pillar/top.sls

base:
  '*':
    - schedule.sls
    - beacons.sls

(schedule.sls runs just fine)

Alternative 1

# /srv/pillar/beacons.sls

beacons:
  diskusage:
    - /: 5%

Minion log:

2016-05-24 16:07:31,593 [salt.utils.lazy  ][DEBUG   ][10453] LazyLoaded diskusage.beacon
2016-05-24 16:07:31,593 [salt.minion      ][CRITICAL][10453] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

Alternative 2

# /srv/pillar/beacons.sls

beacons:
  diskusage:
    /: 5%

Minion log:

2016-05-24 15:53:38,752 [salt.minion      ][CRITICAL][1079] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 66, in beacon
    mount = diskusage.keys()[0]
AttributeError: 'str' object has no attribute 'keys'

Any idea of a workaround?

cachedout · 2016-05-24T14:32:09Z

@fredrikaverpil

Looks like the diskusage beacon was completely broken. I have submitted a patch to fix it here: #33474

Apologies for that.

fredrikaverpil · 2016-05-24T15:14:09Z

With the new patch, and using the syntax from the diskusage.py contents example, I now get this (as in the OP's traceback):

beacons:
  diskusage:
    - /: 5%

2016-05-24 16:55:01,775 [salt.minion      ][CRITICAL][14648] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

cachedout · 2016-05-24T15:14:55Z

@fredrikaverpil Yes, that is the same stacktrace as the original report. This appears to be related to the way the configuration is coming in from pillar.

fredrikaverpil · 2016-05-24T15:16:50Z

@cachedout Ok!

Also, not sure if related or not, but when you perform a salt 'someminion' beacons.list, you'll only get a proper response if the beacon was defined in the minion config. If defined in the pillar, the response is missing the beacon info. Perhaps that is expected, or should I open up a new issue for that?

cachedout · 2016-05-24T15:17:30Z

@fredrikaverpil Sounds related to this one. Let's fix this one first and if that issue persists after that, I'll ask you to open a separate issue. Thanks!

fredrikaverpil · 2016-05-24T15:19:06Z

@cachedout Sounds great, thank you! It's possible that that was intended behavior, as the beacons do turn up when you run a salt 'someminion' pillar.items when the beacons are defined in the pillar.

Fixes saltstack#29451

cachedout · 2016-05-24T15:27:50Z

OK @githubcdr and @fredrikaverpil give this a try: #33476

fredrikaverpil · 2016-05-25T06:10:31Z

@cachedout so I've downloaded the new __init__.py and the new diskusage.py into my /srv/salt/_beacons folder. I performed a salt 'myminion' saltutil.sync_all and I can see the new files being synced:

myminion
    ----------
    beacons:
        - beacons.__init__
        - becons.diskusage

I can also see them being synced in the minion log:

2016-05-25 07:56:36,528 [salt.fileclient  ][INFO    ][10340] Caching directory u'_beacons/' for environment 'base'
2016-05-25 07:56:36,555 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', looking at rel_path u'_beacons/__init__.py' to resolve u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,556 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', ** considering ** path u'/var/cache/salt/minion/files/base/_beacons/__init__.py' to resolve u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,557 [salt.fileclient  ][DEBUG   ][10340] Fetching file from saltenv 'base', ** attempting ** u'salt://_beacons/__init__.py'
2016-05-25 07:56:36,557 [salt.fileclient  ][DEBUG   ][10340] No dest file found
2016-05-25 07:56:36,579 [salt.fileclient  ][INFO    ][10340] Fetching file from saltenv 'base', ** done ** u'_beacons/__init__.py'
2016-05-25 07:56:36,605 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', looking at rel_path u'_beacons/diskusage.py' to resolve u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,605 [salt.fileclient  ][DEBUG   ][10340] In saltenv 'base', ** considering ** path u'/var/cache/salt/minion/files/base/_beacons/diskusage.py' to resolve u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,606 [salt.fileclient  ][DEBUG   ][10340] Fetching file from saltenv 'base', ** attempting ** u'salt://_beacons/diskusage.py'
2016-05-25 07:56:36,606 [salt.fileclient  ][DEBUG   ][10340] No dest file found
2016-05-25 07:56:36,634 [salt.fileclient  ][INFO    ][10340] Fetching file from saltenv 'base', ** done ** u'_beacons/diskusage.py'
2016-05-25 07:56:36,634 [salt.loaded.int.module.saltutil][DEBUG   ][10340] Local cache dir: '/var/cache/salt/minion/files/base/_beacons'
2016-05-25 07:56:36,635 [salt.loaded.int.module.saltutil][INFO    ][10340] Copying '/var/cache/salt/minion/files/base/_beacons/__init__.py' to '/var/cache/salt/minion/extmods/beacons/__init__.py'
2016-05-25 07:56:36,635 [salt.loaded.int.module.saltutil][INFO    ][10340] Copying '/var/cache/salt/minion/files/base/_beacons/diskusage.py' to '/var/cache/salt/minion/extmods/beacons/diskusage.py'

I then restarted the salt service on myminion. But I still get this in my minion log:

2016-05-25 08:04:00,646 [salt.utils.lazy  ][DEBUG   ][11398] LazyLoaded diskusage.beacon
2016-05-25 08:04:00,647 [salt.minion      ][CRITICAL][11398] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 55, in process
    interval = self._determine_beacon_config(mod, 'interval', b_config)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 105, in _determine_beacon_config
    config = config_mod[mod].get(val, False)
AttributeError: 'list' object has no attribute 'get'

So I checked the diff on /usr/lib/python2.7/site-packages/salt/beacons/__init__.py on my minion against the newly patched /srv/salt/_beacons/__init__.py on the master and the former is the old one, and hasn't been synced properly, it seems. And just like __init__.py, the diskusage.py file in /usr/lib/python2.7/site-packages/salt/beacons/ on the minion is also the old, broken, one.

What am I missing or doing wrong since these new patched files in _beacons doesn't make it onto the minion?

cachedout · 2016-05-25T14:42:45Z

The __init__.py file won't be synced into the correct location. You need to copy the updated version into /usr/lib/python2.7/site-packages/salt/beacons/ by hand or by using something like Salt's file.managed.

fredrikaverpil · 2016-05-26T08:54:43Z

@cachedout I copied the files in place by hand, per your instructions, and after restarting the salt-minion service I now get this on the minion:

2016-05-26 10:50:09,252 [salt.minion      ][CRITICAL][7636] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1750, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 364, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/var/cache/salt/minion/extmods/beacons/diskusage.py", line 69, in beacon
    _current_usage = psutil.disk_usage(mount)
  File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 1787, in disk_usage
    return _psplatform.disk_usage(path)
  File "/usr/lib64/python2.7/site-packages/psutil/_psposix.py", line 122, in disk_usage
    st = os.statvfs(path)
TypeError: must be string, not dict

githubcdr · 2016-05-31T21:51:13Z

Hi @cachedout,

Some errors here too after replacing "/usr/lib/python2.7/site-packages/salt/beacons/init.py"

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1757, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 365, in process_beacons
    return self.beacons.process(b_conf)
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 66, in beacon
    mount = diskusage.keys()[0]

zsh 2703 # salt skynet beacons.list
skynet
    beacons:
      diskusage:
        /: 25%

using this state

diskusage:
  beacon.present:
    - /: 25%
    - interval: 20

Salt Version:
           Salt: 2015.8.10

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.24.0
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.11 (default, Mar 31 2016, 06:18:34)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4
           cffi: 1.6.0
       cherrypy: 5.4.0
       dateutil: 2.5.3
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: 0.24.0
        libnacl: 1.4.5
   msgpack-pure: 0.1.3
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4

System Versions:
           dist:
        machine: x86_64
        release: 4.5.4-1-ARCH

githubcdr · 2016-06-06T18:29:15Z

@cachedout is there anything I can do to test this further? I got the feeling that a big part of the beacon functionality is broken on a low level, preventing any use.

It would be really nice if beacons received some love, because this is what makes salt dynamic compared to other systems.

githubcdr · 2016-06-09T19:56:25Z

Hi,

Sorry for re-opening this issue, but it seems the beacon emit function causes a critical error, the config part is loading correctly now.

I see this in /var/log/salt/minion

2016-06-09 21:51:29,089 [salt.minion      ][CRITICAL][19796] The beacon errored:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 2022, in handle_beacons
    beacons = self.process_beacons(self.functions)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 380, in process_beacons
    return self.beacons.process(b_conf)  # pylint: disable=no-member
  File "/usr/lib/python2.7/site-packages/salt/beacons/__init__.py", line 73, in process
    raw = self.beacons[fun_str](b_config[mod])
  File "/usr/lib/python2.7/site-packages/salt/beacons/diskusage.py", line 72, in beacon
    mount = diskusage.keys()[0]
AttributeError: 'str' object has no attribute 'keys'

I removed all .pyc leftovers and double checked the logs.

jfindlay added this to the Approved milestone Dec 4, 2015

jfindlay mentioned this issue Dec 4, 2015

Back-port #28388 to 2015.8 #28491

Merged

fredrikaverpil mentioned this issue Mar 9, 2016

Beacons in pillar doesn't trigger events #31766

Closed

jfindlay added P2 Priority 2 and removed P3 Priority 3 labels Mar 10, 2016

asyncmind0 mentioned this issue May 16, 2016

high salt-minion memory usage on debian jessie? #23167

Closed

jfindlay added Beacon P1 Priority 1 and removed P2 Priority 2 labels May 18, 2016

cachedout added fixed-pls-verify fix is linked, bug author to confirm fix and removed fixed-pls-verify fix is linked, bug author to confirm fix labels May 24, 2016

cachedout self-assigned this May 24, 2016

cachedout pushed a commit to cachedout/salt that referenced this issue May 24, 2016

Allow for config entry to be a list in a dict for beacons

c740c03

Fixes saltstack#29451

cachedout mentioned this issue May 24, 2016

Allow for config entry to be a list in a dict for beacons #33476

Merged

cachedout added the fixed-pls-verify fix is linked, bug author to confirm fix label May 24, 2016

rallytime closed this as completed in b4071b0 Jun 9, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2015.8.3 pillar beacons bugged? #29451

2015.8.3 pillar beacons bugged? #29451

githubcdr commented Dec 4, 2015

githubcdr commented Dec 4, 2015

jfindlay commented Dec 4, 2015

The-Loeki commented Dec 15, 2015

The-Loeki commented Dec 15, 2015

fredrikaverpil commented Mar 11, 2016

The-Loeki commented Mar 11, 2016

fredrikaverpil commented Apr 4, 2016

asyncmind0 commented May 16, 2016 •

edited

Loading

githubcdr commented May 17, 2016 •

edited

Loading

fredrikaverpil commented May 24, 2016 •

edited

Loading

cachedout commented May 24, 2016 •

edited

Loading

fredrikaverpil commented May 24, 2016 •

edited

Loading

cachedout commented May 24, 2016

fredrikaverpil commented May 24, 2016

cachedout commented May 24, 2016

fredrikaverpil commented May 24, 2016 •

edited

Loading

cachedout commented May 24, 2016

fredrikaverpil commented May 25, 2016 •

edited

Loading

cachedout commented May 25, 2016

fredrikaverpil commented May 26, 2016 •

edited

Loading

githubcdr commented May 31, 2016 •

edited

Loading

githubcdr commented Jun 6, 2016

githubcdr commented Jun 9, 2016

2015.8.3 pillar beacons bugged? #29451

2015.8.3 pillar beacons bugged? #29451

Comments

githubcdr commented Dec 4, 2015

githubcdr commented Dec 4, 2015

jfindlay commented Dec 4, 2015

The-Loeki commented Dec 15, 2015

The-Loeki commented Dec 15, 2015

fredrikaverpil commented Mar 11, 2016

The-Loeki commented Mar 11, 2016

fredrikaverpil commented Apr 4, 2016

asyncmind0 commented May 16, 2016 • edited Loading

githubcdr commented May 17, 2016 • edited Loading

fredrikaverpil commented May 24, 2016 • edited Loading

Alternative 1

Alternative 2

cachedout commented May 24, 2016 • edited Loading

fredrikaverpil commented May 24, 2016 • edited Loading

cachedout commented May 24, 2016

fredrikaverpil commented May 24, 2016

cachedout commented May 24, 2016

fredrikaverpil commented May 24, 2016 • edited Loading

cachedout commented May 24, 2016

fredrikaverpil commented May 25, 2016 • edited Loading

cachedout commented May 25, 2016

fredrikaverpil commented May 26, 2016 • edited Loading

githubcdr commented May 31, 2016 • edited Loading

githubcdr commented Jun 6, 2016

githubcdr commented Jun 9, 2016

asyncmind0 commented May 16, 2016 •

edited

Loading

githubcdr commented May 17, 2016 •

edited

Loading

fredrikaverpil commented May 24, 2016 •

edited

Loading

cachedout commented May 24, 2016 •

edited

Loading

fredrikaverpil commented May 24, 2016 •

edited

Loading

fredrikaverpil commented May 24, 2016 •

edited

Loading

fredrikaverpil commented May 25, 2016 •

edited

Loading

fredrikaverpil commented May 26, 2016 •

edited

Loading

githubcdr commented May 31, 2016 •

edited

Loading