mine_collection not working via pillar in 2018.3

[aesposito91]

@### Description of Issue/Question
After migrating to 2018.3 I am not longer able to get mine collection to work automatically, it only works with a manual call of mine.update. It appears others are successful when putting mine_interval and mine_functions inside the proxy file whereas mine (as in me) are declared inside of pillar.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Proxy cfg -

$ cat salt-proxy/proxy | grep mine
mine_enabled: True

Relevant pillar -

cat salt-master/salt/pillar/globals/mine_data.sls
mine_functions:
  net.arp: []
  net.interfaces: []
  net.mac: []
  net.lldp: []
  net.facts: []    
  net.ipaddrs: []
  bgp.neighbors: []

mine_interval: 10

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Wait the mine_interval, and see that no mine data has been collected.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Proxy minion and master are both 2018.3.0

$salt --versions-report
Salt Version:
Salt: 2018.3.0

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.5.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Nov 24 2017, 17:33:09)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 17.0.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.1.6

System Versions:
dist: debian 9.3
locale: ANSI_X3.4-1968
machine: x86_64
release: 4.7.1-coreos
system: Linux
version: debian 9.3

[Ch3LL]

@aesposito91 was this working previously and if so what version?

There was this issue: #39775 which was fixed in 2017.7.3 by this PR: #41547 but that requires an extra setting. So I'm wondering if you are setting minion_interval in the pillar but its not being set correctly. If you check your minion scheduler does it show the correct amount of minutes you would expect?

[aesposito91]

I had success with this same layout in 2017.7.4 and confirmed with a friend on slack that is working on 2017.7.5 via same method.

It does appear to properly update the mine_interval on the proxy -

schedule:
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 10, return_job: false}
__proxy_keepalive:
enabled: true
function: status.proxy_reconnect
jid_include: true
kwargs: {proxy_name: napalm}
maxrunning: 1
minutes: 1
return_job: false

[Ch3LL]

ping @mirceaulinic are you seeing this behavior as well?

also @saltstack/team-network any ideas here as well?

[ngrundler]

Any movement on this issue? I'm seeing similar problems with my proxy minions on 2018.3.2.

[Ch3LL]

i have a proxy setup now. Can you clarify the exact steps you run to see this issue?

[ngrundler]

Hey @Ch3LL, thanks for the quick reply! For my environment, my proxy host has the following configured:

/etc/salt/proxy:

master: salt.my.domain
multiprocessing: False
mine_enabled: True

...and here's a snippet of my minion pillar:

/srv/pillar/my_minion.sls:

proxy:
  proxytype: napalm
  driver: junos
  host: 10.0.0.1
  username: foo
  optional_args:
    key_file: /root/.ssh/id_rsa
mine_interval: 10m
mine_functions:
  net.lldp: []

If I run a salt 'my_minion' net.lldp, I see the output I would expect. If I wait the mine interval or even run a salt 'my_minion' mine.update I never see anything in the output of salt '*' mine.get 'my_minion' net.lldp.

If I do a $ cat /etc/salt/proxy.d/my_minion/_schedule.conf from my proxy host, I can see the mine_interval parameter has been updated to reflect the time specified in the pillar like @aesposito91 noticed:

schedule:
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
    minutes: 10m, return_job: false}
  __proxy_keepalive:
    enabled: true
    function: status.proxy_reconnect
    jid_include: true
    kwargs: {proxy_name: napalm}
    maxrunning: 1
    minutes: 1
    return_job: false

Please let me know if I can provide any additional details.

[Ch3LL]

thanks for the use case :)

unfortunately i do not have access to a device that uses the net.lldp module. we would need someone from @saltstack/team-netapi that possibly might be able to help replicate this.

I did test with my proxy that uses a rest interface with this mine configuration:

mine_functions:
  test.ping: []
  network.ip_addrs:
    interface: eth0

and it worked just fine:

[root@9da0e3a04362 /]# salt '*' mine.get '*' network.ip_addrs
p8000:
    ----------
    p8000:
        - 172.17.0.2

This might be related to this issue: #48020 but I feel that might be a longshot. if you can comment out the lines here: #48020 (comment) that was the cause of that issue and see if you still are seeing the issue. Like i said this might be a long shot though.

Can you see if you add a different mine configuration if it works? I just want to see if its just these specific net modules that are the issue.

Also is there anything in the debug logs when this occurs?

[ngrundler]

I was able to get this working manually with mine.update, though I'm not sure why it suddenly started behaving. I'm going to assume it was an issue on my end, but scheduled collection is definitely still not working. No helpful errors when I run the proxy/master in debug mode either, and commenting out the lines in #48020 didn't have any effect.

[Ch3LL]

and if you use a different mine setting like test.ping for simplicity sake does it schedule the update?

Also do you see it attempting to run the schedule in the debug logs?

[ngrundler]

Just using test.ping doesn't show any difference.

I never see the schedule.handle_func calling mine.update in the debug log. Given the following /etc/salt/proxy.d/my-minion/_schedule.conf:

schedule:
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
    minutes: 1, return_job: false}
  __proxy_keepalive:
    enabled: true
    function: status.proxy_reconnect
    jid_include: true
    kwargs: {proxy_name: napalm}
    maxrunning: 1
    minutes: 1
    return_job: false

I do periodically see the following at the expected 1m interval:

[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {u'fun_args': [], u'jid': u'20180730234651878471', u'schedule': u'__proxy_keepalive', u'pid': 26461, u'fun': u'status.proxy_reconnect', u'id': 'my-minion'}

...but never for mine.update.

[Ch3LL]

so i did a bit more trying to replicate this and the closest i could get was I am seeing this error in the proxy log:

2018-08-01 15:28:58,292 [salt.minion :445 ][ERROR ][14952] Exception argument of type 'bool' is not iterable occurred in scheduled job

and i bisected it to this b23e5a7

using this pillar data:

[root@3c7ca8482634 testing]# cat /root/salt-manual-tests/proxy/pillar/proxy8000.sls 
proxy:
  proxytype: rest_sample
  url: http://localhost:8000/ 
mine_functions:
  test.ping: []
mine_interval: 1

but it seems its still trying to schedule the job, but can't because of this error on 2018.3

ping @ngrundler do you see this error at all when setting the same pillar data above?

Heres a docker container to easy replicate:

1. docker run -it -v ~/git/salt/:/testing/ ch3ll/salt-setup:proxy (where ~/git/salt/ is a local cloned git repo of salt)
2. add the following information to pillar file:

[root@3c7ca8482634 /]# cat /root/salt-manual-tests/proxy/pillar/proxy8000.sls
proxy:
  proxytype: rest_sample
  url: http://localhost:8000/ 
mine_functions:
  test.ping: []
mine_interval: 1

3. salt-master -d
4. cd ~/salt-contrib/proxyminion_rest_example/ && python rest.py --address=localhost --port=8000
5. salt-proxy --proxyid=p8000 -d

ping @garethgreenaway any ideas why i'm seeing this error?

[ezh]

Confirmed. mine_interval is not updated.
Minion saltenv=test, pillarenv=test

 $ pepper '*1' pillar.raw
us-west-1-test-0.3-mesos-mesosmaster-tf-1:
    ----------
    metadata_server_grains:
        True
    mine_functions:
        ----------
        grains.items:
        test.ping:
    mine_interval:
        1
    public-ipv4:
        NNN.183.237.37
    roles:
        test_pillar_ready
 $ pepper '*1' schedule.list show_all=True
us-west-1-test-0.3-mesos-mesosmaster-tf-1:
    schedule:
      __mine_interval:
        enabled: true
        function: mine.update
        jid_include: true
        maxrunning: 2
        minutes: 60
        name: __mine_interval
        return_job: false

I have a lot of 'Added new job __mine_interval to scheduler' messages in minion log (I changed pillar value).
I haven't any mine related settings in minion configuration.

[Ch3LL]

i know there was some work in 2018.3.3 around the mine any chance you can try that version?

[ezh]

We are using stable 2018.3.3

[Ch3LL]

im not able to replicate this on 2018.3.3. ping @garethgreenaway do you see anything obvious on the scheduler side of things?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.