[BUG] Unwanted targets selected by Salt

[jagatjyoti]

Description
Unwanted targets selected by Salt

Setup
Running a test.ping using IPv4 address of a minion returns two targets whereas it should return the result for the target being specified. This is happening only for a specific minion in our cluster.

Steps to Reproduce the behavior
Run command salt -S <IPv4 address> test.ping

Expected behavior
The salt command should have returned the liveness status of that specific minion as below (values altered for hiding sensitive info):

# salt -S "10.181.11.65" test.ping
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True

But it is selecting two minions which is unexpected as below:

# salt -v -S 10.181.11.65 test.ping
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True
cddprdd001-prod-usw2-node--i-05f513ee97536143873:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code

Debug output:

# salt -v -S 10.181.11.65 test.ping -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-176-33-42.cddprdd001-prod-usw2
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-176-3-42.cdprd01-prod-usw2
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'ip-10-176-33-42.cddprdd001-prod-usw2-master', u'tcp://10.176.33.42:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.176.33.42:4506
[DEBUG   ] Trying to connect to: tcp://10.176.33.42:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
Executing job with jid 20200925061940576513
-------------------------------------------

[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/56/fd0231ba974b9812c1449ec91cfe52d88b373162fb4fb35d2f3fab633e79a8/.minions.p
[DEBUG   ] get_iter_returns for jid 20200925061940576513 sent to set(['cddprdd001-prod-usw2-node--i-05f513ee97536143873', 'cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2']) will timeout at 06:19:45.677833
[DEBUG   ] jid 20200925061940576513 return from cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2
[DEBUG   ] return event: {u'cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2': {u'jid': u'20200925061940576513', u'retcode': 0, u'ret': True}}
[DEBUG   ] LazyLoaded nested.output
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True
[DEBUG   ] Checking whether jid 20200925061940576513 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'ip-10-176-33-42.cddprdd001-prod-usw2_master', u'tcp://10.176.33.42:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.176.33.42:4506
[DEBUG   ] Trying to connect to: tcp://10.176.33.42:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client.
[DEBUG   ] return event: {'cddprdd001-prod-usw2-node--i-05f513ee97536143873': {u'failed': True}}
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded no_return.output
cddprdd001-prod-usw2-node--i-05f513ee97536143873:
    Minion did not return. [No response]
[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code

Screenshots

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

# salt --versions-report
Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 20 2019, 20:27:34)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

There are no differences in master and minion versions. All are on 3000.3

Additional context
Have tried clearing the cache and re-running which doesn't help. The unwanted minion is alive too.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on Freenode
• SaltStack YouTube channel
• SaltStackInc Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at core@saltstack.com or reach out directly to the Community Manager, Cassandra Faris via Slack. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[Ch3LL]

The matchers in salt check the cache first to see if they match the criteria and then target those minions. So if you have a minion with cached data on the master (/var/cache/salt/master/minions/<minion name>/data.p) with that ip address it will try to target that minion. If you clear your cache do you still see this issue?

[jagatjyoti]

The same issue persists even after clearing the cache.

[jagatjyoti]

Any updates here ?

[vskubriev]

I can confirm that we also see it in our 3002.2. Hope next release should fix it. See #57750