Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting #54776

Open
javierbertoli opened this issue Sep 26, 2019 · 8 comments

Comments

@javierbertoli
Copy link
Contributor

commented Sep 26, 2019

Description of Issue

Due to errors in the the minion's function remove_periodic_callback(), when setting the parameter ping_interval in the minion configuration (ie, to check against a DDNS master, as described here), the minion fails to start and floods the logs with

[CRITICAL] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

At first it seems to be caused by a typo in remove_periodic_callbback() (note the 2 bb) but when fixing it, the error changes to

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1035, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2741, in tune_in
    self.remove_periodic_callback('ping', ping_master)

and does not start either.

Steps to Reproduce Issue

  1. Install the salt-minion v2019.2.1 packages
  2. Add the following config parameters
cat > /etc/salt/minion.d/test.conf << EOF
auth_tries: 2
master_tries: -1
auth_safemode: false
ping_interval: 5
EOF
  1. Run salt-minion -l debug to get the error.
  2. Remove the ping_interval parameter, repeat step 3 and it will run OK.

Versions Report

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Apr  3 2019, 05:39:12)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist: debian 10.1 
         locale: utf-8
        machine: x86_64
        release: 5.3.0-rc5-amd64
         system: Linux
        version: debian 10.1 
@cmcmarrow

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2019

I can confirm that this happens as on Ubuntu as well.

@morsik

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2019

Same on CentOS 7.6 with minions deployed via Salt Cloud.

@devopsdeluxe

This comment has been minimized.

Copy link

commented Sep 30, 2019

Master:

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1048-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Minion:

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-64-generic
         system: Linux
        version: Ubuntu 18.04 bionic

As mentioned in the original report, downgrading from to 2019.2.0 or removing the ping_interval configuration option resolves the issue

@sastorsl

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2019

Same on Red Hat Enterprise Linux Server 7 / RHEL7

# salt-call -V
Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.1.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.7 Maipo

Errors in /var/log/salt/minion

2019-10-01 09:15:18,923 [salt.minion      :2313][ERROR   ][2611] This Minion is already running. Not running Minion.tune_in()
2019-10-01 09:15:18,946 [salt.minion      :2692][INFO    ][2611] Minion is ready to receive requests!
2019-10-01 09:15:18,947 [salt.minion      :1049][CRITICAL][2611] Unexpected error while connecting to 10.246.104.52
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

NOT having this problem with salt-2019.2.0 on an identical server with redhat 7.6 Maipo running 3.10.0-957.12.2.el7.x86_64.

@hedrickbt

This comment has been minimized.

Copy link

commented Oct 1, 2019

I have created a support ticket for this:
https://help.saltstack.com/hc/en-us/requests/4183

@doesitblend doesitblend added the ZD label Oct 1, 2019
@mariusvw mariusvw referenced this issue Oct 8, 2019
@Xopherus

This comment has been minimized.

Copy link

commented Oct 17, 2019

Has there been a version released with this fix? If not, when can we expect that?

@hedrickbt

This comment has been minimized.

Copy link

commented Oct 17, 2019

@Xopherus ,

The following pull request is being tested. #54777

The plan is for the fix to be included in 2019.2.2. The last update I have from support is 10 days ago. I asked for an update today.

A notice has been put out to let people know to avoid going to 2019.2.1 at https://repo.saltstack.com/

@hedrickbt

This comment has been minimized.

Copy link

commented Oct 18, 2019

Be on the look out for a possible release mid-next-week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.