New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
presence_events doesn't behave as indicated #45173
Comments
ETA: Tailing the events "properly" also results in the same misbehavior:
|
I am able to replicate this. Except i don't even see the master id in the present list. I don't ever see any minions show up in either present, or changes when minions are added. Thanks for reporting. |
@V3XATI0N What kind of environment are you running in? Is your salt master in a docker container? Thanks, |
No, it's a dedicated EC2 instance. |
@V3XATI0N Couple things to check. Can you see if your minions have the ipv4 grain available? Also check to see if either the |
All the minions have the ipv4 grain available and correctly defined. Some of them don't have the |
@V3XATI0N I spun up a couple EC2 instances, using both CentOS and Ubuntu images, to try to duplicate the issue that you're seeing and unfortunately I wasn't able to. Both instances are in the same availability zone and same subnet, they are in different security groups with the master group only allowing access to the Salt ports (4505 & 4506) to the minion group. Everything is working and I'm seeing events on the event bus when the presence of minions change. A couple additional question, is there anything different about your setup to what I've mentioned above that I could change in try and duplicate? Are the minions using IPv6 to connect to the master? Thanks! |
No one is using ipv6 for connecting to the master right now. The master is in AWS, but most minions are in our on-premises data center or scattered geographically in other data centers or offices, or carried around by end users and connecting sporadically. They're connected directly to the master, not through a Syndic. Anywhere from 50 to 150 of the minions may be offline at any given time during the normal course of business, and none of them are having any trouble getting connected or receiving salt commands when needed. They should be generating presence events all day long, but only the master ever shows up. |
@V3XATI0N Still trying to duplicate this issue so I have a couple more questions. Are you using any of the alternative backends for the minion cache on the master, eg. consul, etcd, etc. or are you using the stock file system based one? If you're using the file system one, can you check in |
I have Redis configured but I'm not using it (the server definition is there but the |
@V3XATI0N Would you be able to make a simple manual patch to a file on your master to see if we can see what is happening here? Make sure your log level is set to info and restart the master. |
I've applied that patch and restarted the master. I've set up a process tailing the master log and grepping for |
Awesome! Very interested to see if you see anything. You should see something if you run the salt runner with |
The log doesn't show anything at all, but |
@V3XATI0N Would you be able to paste a small sample? Feel free to scrub any data that you feel is sensitive. |
here's a snippet... the entire manage.present call generates a call about 8MB in size.
|
also mispelled mdata as mdate but still got the result ... so it is appearing in the master log, but had to drop the log level down so I didn't use up the entire disk in a few minutes. |
@V3XATI0N This is perfect. I wanted to confirm that there was something inside mdata and it was making it down into the code. Can you move the log line that you added down towards the bottom right above the |
apologies, I'm not quite catching what you mean. the EDIT: oh, change |
That just causes the function to crash:
|
@V3XATI0N Yup. that's the right line, I was at the HEAD of 2017.7 so I was a bit off. It looks like you're missing the |
You're right, I forgot the |
Okay that's good....we're narrowing down where the problem is and if you're seeing minions in that log statement then it's not in connected_minions. One quick question, are you using |
it's just zmq... but that's TCP, unless I'm confused. |
Okay good. You can remove the log calls from minions.py and let's move onto the runner, edit: |
yeah, returning minions from there (as it is just after it returns from the |
So you are seeing the minions listed on the standard output from the runner when running manage.present? |
Yes, just not in the presence events. |
Okay. Let's edit the master.py: We'll use warning here so it's a bit less chatting and doesn't fill up your logs. Once that's in place, restart the Salt master and then restart the salt-minion process on one of your minions and see if the presence event triggers and you see the output in the logs. |
tangential to this issue, is it a known issue that |
This could be related to the grains issue. Do they have the ipv4 grain available? |
yes. all the minions have the |
okay... here is the
But with those two logging lines, I only get the usual output with warning-level (or info, or even debug) command. These log statements don't output anything at all. |
To confirm, you've restarted the Salt master and you're looking at the master logs? Are you able to watch the event bus and see the presence events firing, even if they're empty or just have the master as before? |
yeah, there's no change at all. The master was restarted (twice) and the presence events in the log still only list the master. To be honest I'm ready to give up on this for now, since for my purposes I could just call |
Windows minions should definitely show up in the manage.present, that's a separate issue that needs to be addressed. If you wouldn't mind trying one more thing, change the |
Description of Issue/Question
I have set
presence_events
in the Master config file totrue
, but when thesalt/presence/present
event fires, only the Master is listed despite the fact that about 200 Minions are online at any given moment.Setup
Steps to Reproduce Issue
Tail the Master log, receive (for example):
despite the fact that ....
Versions Report
The text was updated successfully, but these errors were encountered: