New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calling salt '*' state.highstate on multi master, minions stop responding #19932

Closed
smitt04 opened this Issue Jan 22, 2015 · 12 comments

Comments

Projects
None yet
6 participants
@smitt04

smitt04 commented Jan 22, 2015

First of all I have a Multi-Master setup.
Only 1 of my masters have the state files, including the top.sls
if I call salt '*' state.highstate from the master with the files the minion stops responding and the call returns to the terminal with no error messages. I have determined that this is because the minion is asking the wrong master for the files. I think that the highstate command should be executed on the master that I ran it on, not check any available master.

@rallytime rallytime added this to the Blocked milestone Jan 22, 2015

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Jan 22, 2015

Contributor

Interesting. I haven't tried executing something like that before in a multi-master setup, but this would be good to reproduce. Can you post the the salt --versions-report for both of your masters as well as a salt <minion> test.versions_report for your minion?

When you say "returns with no error messages" do you mean the CLI is just returning and nothing is printed in the output? Have you tried checking your minion to see if the states are executed? When I am looking at multi-master issues, especially, I find it very useful to run the masters and the minion in debug-mode. That way it is easy to see if the minion is receiving the job from the master that initiated the command, and then returning the job status back to the master. (And then I also look to see where that return ended up.) Any logs along these lines displaying the behavior your seeing would also be helpful.

Contributor

rallytime commented Jan 22, 2015

Interesting. I haven't tried executing something like that before in a multi-master setup, but this would be good to reproduce. Can you post the the salt --versions-report for both of your masters as well as a salt <minion> test.versions_report for your minion?

When you say "returns with no error messages" do you mean the CLI is just returning and nothing is printed in the output? Have you tried checking your minion to see if the states are executed? When I am looking at multi-master issues, especially, I find it very useful to run the masters and the minion in debug-mode. That way it is easy to see if the minion is receiving the job from the master that initiated the command, and then returning the job status back to the master. (And then I also look to see where that return ended up.) Any logs along these lines displaying the behavior your seeing would also be helpful.

@smitt04

This comment has been minimized.

Show comment
Hide comment
@smitt04

smitt04 Jan 22, 2015

Here are the Versions:
Master 1

               Salt: 2014.7.0
             Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
             Jinja2: 2.2.1
           M2Crypto: 0.20.2
     msgpack-python: 0.1.13
       msgpack-pure: Not Installed
           pycrypto: 2.0.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.3.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: Not Installed

Master 2

           Salt: 2014.7.0
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: unknown
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 4.0.4
           Mako: Not Installed

Minion

               Salt: 2014.7.0
             Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
             Jinja2: 2.2.1
           M2Crypto: 0.20.2
     msgpack-python: 0.1.13
       msgpack-pure: Not Installed
           pycrypto: 2.0.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.3.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: Not Installed

Now I ran the salt *lint* state.highstate command a few times.

When it fails this is what I get:

Minion

[INFO    ] User sudo_ksmithson Executing command state.highstate with jid 20150122073556849877
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150122073556849877', 'tgt': '*lint*', 'ret': '', 'user': 'sudo_ksmithson', 'arg': [], 'fun': 'state.highstate'}
[INFO    ] Starting a new job with PID 26324
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Returning information for job: 20150122073556849877
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] User sudo_ksmithson Executing command saltutil.find_job with jid 20150122073601860534
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150122073601860534', 'tgt': '*lint*', 'ret': '', 'user': 'sudo_ksmithson', 'arg': ['20150122073556849877'], 'fun': 'saltutil.find_job'}
[INFO    ] Starting a new job with PID 26335
[INFO    ] Returning information for job: 20150122073601860534
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

Master

[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:21:59.764771', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:21:59.907732', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _pillar
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.341877', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.343037', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.532412', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.595558', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _mine
[INFO    ] AES payload received with command _mine
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:01.729758', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _minion_event
[DEBUG   ] Sending event - data = {'_stamp': '2015-01-22T07:22:01.826280', 'pretag': None, 'cmd': '_minion_event', 'tag': 'minion_start', 'data': 'Minion prod-lint-pk1.c1.cs1.sazze.com started at Thu Jan 22 07:22:01 2015', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:01.870372', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _minion_event
[DEBUG   ] Sending event - data = {'_stamp': '2015-01-22T07:22:01.966350', 'pretag': None, 'cmd': '_minion_event', 'tag': 'salt/minion/prod-lint-pk1.c1.cs1.sazze.com/start', 'data': 'Minion prod-lint-pk1.c1.cs1.sazze.com started at Thu Jan 22 07:22:01 2015', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:18.504390', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _master_opts
[INFO    ] AES payload received with command _file_list
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:18.652884', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _pillar
[INFO    ] AES payload received with command _file_hash
[INFO    ] AES payload received with command _serve_file
[INFO    ] AES payload received with command _ext_nodes
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:19.129302', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _return
[INFO    ] Got return from prod-lint-pk1.c1.cs1.sazze.com for job 20150122072217910825
[DEBUG   ] Sending event - data = {'fun_args': [], 'jid': '20150122072217910825', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:19.228299', 'fun': 'state.highstate', 'id': 'prod-lint-pk1.c1.cs1.sazze.com', 'out': 'highstate'}
[DEBUG   ] Sending event - data = {'fun_args': [], 'jid': '20150122072217910825', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:19.228771', 'fun': 'state.highstate', 'id': 'prod-lint-pk1.c1.cs1.sazze.com', 'out': 'highstate'}
[DEBUG   ] Sending event - data = {'comment': 'No Top file or external nodes data matches found', '_stamp': '2015-01-22T07:22:19.229020', 'retcode': 2, 'result': False, '__run_num__': 0, 'changes': {}, 'name': 'No States'}
[DEBUG   ] LazyLoaded local_cache.returner
[ERROR   ] An inconsistency occurred, a job was received with a job id that is not present in the local cache: 20150122072217910825
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:23.524875', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _return
[INFO    ] Got return from prod-lint-pk1.c1.cs1.sazze.com for job 20150122072222923633
[DEBUG   ] Sending event - data = {'fun_args': ['20150122072217910825'], 'jid': '20150122072222923633', 'return': {}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:23.623714', 'fun': 'saltutil.find_job', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[DEBUG   ] Sending event - data = {'fun_args': ['20150122072217910825'], 'jid': '20150122072222923633', 'return': {}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:23.624125', 'fun': 'saltutil.find_job', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[DEBUG   ] LazyLoaded local_cache.returner
[ERROR   ] An inconsistency occurred, a job was received with a job id that is not present in the local cache: 20150122072222923633

And on the master where I execute the command it just returns to the cli with no output, unless I add flag --show-timeout which I then get:

prod-lint-pk1.c1.cs1.sazze.com:
    Minion did not return. [No response]

smitt04 commented Jan 22, 2015

Here are the Versions:
Master 1

               Salt: 2014.7.0
             Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
             Jinja2: 2.2.1
           M2Crypto: 0.20.2
     msgpack-python: 0.1.13
       msgpack-pure: Not Installed
           pycrypto: 2.0.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.3.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: Not Installed

Master 2

           Salt: 2014.7.0
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: unknown
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 4.0.4
           Mako: Not Installed

Minion

               Salt: 2014.7.0
             Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
             Jinja2: 2.2.1
           M2Crypto: 0.20.2
     msgpack-python: 0.1.13
       msgpack-pure: Not Installed
           pycrypto: 2.0.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.3.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: Not Installed

Now I ran the salt *lint* state.highstate command a few times.

When it fails this is what I get:

Minion

[INFO    ] User sudo_ksmithson Executing command state.highstate with jid 20150122073556849877
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150122073556849877', 'tgt': '*lint*', 'ret': '', 'user': 'sudo_ksmithson', 'arg': [], 'fun': 'state.highstate'}
[INFO    ] Starting a new job with PID 26324
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Returning information for job: 20150122073556849877
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] User sudo_ksmithson Executing command saltutil.find_job with jid 20150122073601860534
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150122073601860534', 'tgt': '*lint*', 'ret': '', 'user': 'sudo_ksmithson', 'arg': ['20150122073556849877'], 'fun': 'saltutil.find_job'}
[INFO    ] Starting a new job with PID 26335
[INFO    ] Returning information for job: 20150122073601860534
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

Master

[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:21:59.764771', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:21:59.907732', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _pillar
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.341877', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.343037', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.532412', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:00.595558', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _mine
[INFO    ] AES payload received with command _mine
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:01.729758', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _minion_event
[DEBUG   ] Sending event - data = {'_stamp': '2015-01-22T07:22:01.826280', 'pretag': None, 'cmd': '_minion_event', 'tag': 'minion_start', 'data': 'Minion prod-lint-pk1.c1.cs1.sazze.com started at Thu Jan 22 07:22:01 2015', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:01.870372', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _minion_event
[DEBUG   ] Sending event - data = {'_stamp': '2015-01-22T07:22:01.966350', 'pretag': None, 'cmd': '_minion_event', 'tag': 'salt/minion/prod-lint-pk1.c1.cs1.sazze.com/start', 'data': 'Minion prod-lint-pk1.c1.cs1.sazze.com started at Thu Jan 22 07:22:01 2015', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:18.504390', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _master_opts
[INFO    ] AES payload received with command _file_list
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:18.652884', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _pillar
[INFO    ] AES payload received with command _file_hash
[INFO    ] AES payload received with command _serve_file
[INFO    ] AES payload received with command _ext_nodes
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:19.129302', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _return
[INFO    ] Got return from prod-lint-pk1.c1.cs1.sazze.com for job 20150122072217910825
[DEBUG   ] Sending event - data = {'fun_args': [], 'jid': '20150122072217910825', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:19.228299', 'fun': 'state.highstate', 'id': 'prod-lint-pk1.c1.cs1.sazze.com', 'out': 'highstate'}
[DEBUG   ] Sending event - data = {'fun_args': [], 'jid': '20150122072217910825', 'return': {'no_|-states_|-states_|-None': {'comment': 'No Top file or external nodes data matches found', '__run_num__': 0, 'changes': {}, 'result': False, 'name': 'No States'}}, 'retcode': 2, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:19.228771', 'fun': 'state.highstate', 'id': 'prod-lint-pk1.c1.cs1.sazze.com', 'out': 'highstate'}
[DEBUG   ] Sending event - data = {'comment': 'No Top file or external nodes data matches found', '_stamp': '2015-01-22T07:22:19.229020', 'retcode': 2, 'result': False, '__run_num__': 0, 'changes': {}, 'name': 'No States'}
[DEBUG   ] LazyLoaded local_cache.returner
[ERROR   ] An inconsistency occurred, a job was received with a job id that is not present in the local cache: 20150122072217910825
[INFO    ] Clear payload received with command _auth
[INFO    ] Authentication request from prod-lint-pk1.c1.cs1.sazze.com
[INFO    ] Authentication accepted from prod-lint-pk1.c1.cs1.sazze.com
[DEBUG   ] Sending event - data = {'id': 'prod-lint-pk1.c1.cs1.sazze.com', '_stamp': '2015-01-22T07:22:23.524875', 'result': True, 'pub': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA025cuHjHlHcGUDkqzkFp\n9t6/EmJSeWekYsiXGhZU6x8tYpOmTDpYVVaboejTrA0v+6VyulQcL2gBfuKg/7ds\nHNZl4KuPjd2TjPM2AShQlLTzoxO4w+L7rjNMguELVd1lVM35ZHe+Zi33V1JWkddc\nhTI1jw3AUaa9rfmgPJ0FM1LV7DyzZuaCl5fW8JrIbEiTyDzoHH/MZO4L7kqxL7k2\nxwqAWTP6wwqCb0/dwKi8iSuvPayypgjFg9GmqAuiO/kBZpFQNksGllS7+qWQirRI\n7d58sy/Ce8oR+kclUvrxNdONen/nAuMIlDzoLLEIRV1cV+lZgXTAZtR9d/p+Fj8V\nGE70P4noBnq9fliS1GybfowSvsfnbpigOQdsTxoThE02TsmVKC04TZ/9de55UgFR\nxYZtcv1SicaQ85GBJtNw3x7Cp84Jkjp2V5OFXv6IlXIOgVh0U8zt9W3c0Q18lJJE\n4igXPn25fJV9L3qEjVnn8tvNXzfBseYyvMHMJp/hoBxLiNjViz8cN5vk07WJtKtD\nzl9GHirEPhr9ExByR5UcqlwC1u25lMSa5BCaVevYIKKZUUjw+umQGb47KuHLm3Q4\nxI2XlRb6Fd2Gp6IjsxU3wRUfHuq4/8Di+HtMTdiJpiQgkck0tJgMH1AB0tEJ37yP\nwntrlbhVhy72+w+/ZkL2ulkCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'act': 'accept'}
[INFO    ] AES payload received with command _return
[INFO    ] Got return from prod-lint-pk1.c1.cs1.sazze.com for job 20150122072222923633
[DEBUG   ] Sending event - data = {'fun_args': ['20150122072217910825'], 'jid': '20150122072222923633', 'return': {}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:23.623714', 'fun': 'saltutil.find_job', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[DEBUG   ] Sending event - data = {'fun_args': ['20150122072217910825'], 'jid': '20150122072222923633', 'return': {}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2015-01-22T07:22:23.624125', 'fun': 'saltutil.find_job', 'id': 'prod-lint-pk1.c1.cs1.sazze.com'}
[DEBUG   ] LazyLoaded local_cache.returner
[ERROR   ] An inconsistency occurred, a job was received with a job id that is not present in the local cache: 20150122072222923633

And on the master where I execute the command it just returns to the cli with no output, unless I add flag --show-timeout which I then get:

prod-lint-pk1.c1.cs1.sazze.com:
    Minion did not return. [No response]
@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Jan 22, 2015

Contributor

Thanks for posting that information! It's very helpful. I have a very strong suspicion that #19790 has fixed this issue. This fix was made against the 2015.2 branch, but I am going to backport it to 2014.7. Are you in a position to test from 2015.2 to see if that resolves the issue for you? Otherwise this fix will be in the 2014.7.2 release.

Contributor

rallytime commented Jan 22, 2015

Thanks for posting that information! It's very helpful. I have a very strong suspicion that #19790 has fixed this issue. This fix was made against the 2015.2 branch, but I am going to backport it to 2014.7. Are you in a position to test from 2015.2 to see if that resolves the issue for you? Otherwise this fix will be in the 2014.7.2 release.

@smitt04

This comment has been minimized.

Show comment
Hide comment
@smitt04

smitt04 Jan 22, 2015

Thanks for getting back so quickly, Unfortunately I am not able to test that right now. Those machines are currently in a production space and the admins say they are not comfortable upgrading unless it is in a package so that they can downgrade easily if they need to. If you have a package for it then we can test otherwise we will wait until released. Thanks.

smitt04 commented Jan 22, 2015

Thanks for getting back so quickly, Unfortunately I am not able to test that right now. Those machines are currently in a production space and the admins say they are not comfortable upgrading unless it is in a package so that they can downgrade easily if they need to. If you have a package for it then we can test otherwise we will wait until released. Thanks.

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Jan 22, 2015

Contributor

Not a problem! We can leave this open until you've had a chance to give it a try. Our next release should be coming out any day now (2014.7.1) but the backported fix hasn't been merged yet. Keep an eye out for the 2014.7.2 or 2015.2.0 release, as the fix should be in both of those releases. Cheers!

Contributor

rallytime commented Jan 22, 2015

Not a problem! We can leave this open until you've had a chance to give it a try. Our next release should be coming out any day now (2014.7.1) but the backported fix hasn't been merged yet. Keep an eye out for the 2014.7.2 or 2015.2.0 release, as the fix should be in both of those releases. Cheers!

@jhenry82

This comment has been minimized.

Show comment
Hide comment
@jhenry82

jhenry82 Jan 28, 2015

I think we may be hitting this issue as well. Today I updated about 1000 minions from the 2014.1.x train to 2014.7.1. Our monitoring software periodically tests whether all minions are reachable from all masters via test.ping. Since upgrading, that test has been failing and we've had to play whack-a-mole restarting minions to get them to report in. But every time we fix "minionA not reachable from master1", a different master seems to lose the ability to talk to minionA instead.

Assuming it's the same root cause, could you please post here (or at least reference this issue in the commit) when the fix is backported to 2014.7? I'd like to manually patch our minions and see if that helps. I noticed that the first backport was reverted.

Thanks!

I think we may be hitting this issue as well. Today I updated about 1000 minions from the 2014.1.x train to 2014.7.1. Our monitoring software periodically tests whether all minions are reachable from all masters via test.ping. Since upgrading, that test has been failing and we've had to play whack-a-mole restarting minions to get them to report in. But every time we fix "minionA not reachable from master1", a different master seems to lose the ability to talk to minionA instead.

Assuming it's the same root cause, could you please post here (or at least reference this issue in the commit) when the fix is backported to 2014.7? I'd like to manually patch our minions and see if that helps. I noticed that the first backport was reverted.

Thanks!

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Feb 24, 2015

Contributor

@jhenry82 Oof that is not good! Assuming it's the same root cause and the fix that I mentioned above does indeed fix this, it actually won't be available until 2015.2.0 is released. The original fix was made on the develop branch on code that has been refactored since we cut the 2014.7 branch, so the fix didn't backport cleanly to 2014.7.

I have just now backported the aforementioned fix to 2015.2 in #20964. Is it possible for you to try to test the HEAD of 2015.2 in a smaller testing set up (once that gets merged) to see if this issue is fixed there for you?

Contributor

rallytime commented Feb 24, 2015

@jhenry82 Oof that is not good! Assuming it's the same root cause and the fix that I mentioned above does indeed fix this, it actually won't be available until 2015.2.0 is released. The original fix was made on the develop branch on code that has been refactored since we cut the 2014.7 branch, so the fix didn't backport cleanly to 2014.7.

I have just now backported the aforementioned fix to 2015.2 in #20964. Is it possible for you to try to test the HEAD of 2015.2 in a smaller testing set up (once that gets merged) to see if this issue is fixed there for you?

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Mar 17, 2015

Contributor

After testing this on the HEAD of 2015.2, this bug either has regressed or it wasn't quite fixed correctly. We're working on a fix here.

Contributor

rallytime commented Mar 17, 2015

After testing this on the HEAD of 2015.2, this bug either has regressed or it wasn't quite fixed correctly. We're working on a fix here.

@cachedout

This comment has been minimized.

Show comment
Hide comment
@cachedout

cachedout Mar 19, 2015

Contributor

Fix for this in #21795

Contributor

cachedout commented Mar 19, 2015

Fix for this in #21795

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Mar 20, 2015

Contributor

I have tested this particular issue with Multi-Master on both 2015.2 branch and 2014.7 and I can no longer reproduce it with the fixes that have been backported. @jhenry82 or @smitt04 can either of you give the HEAD of 2014.7 a try and confirm that this has been resolved?

Contributor

rallytime commented Mar 20, 2015

I have tested this particular issue with Multi-Master on both 2015.2 branch and 2014.7 and I can no longer reproduce it with the fixes that have been backported. @jhenry82 or @smitt04 can either of you give the HEAD of 2014.7 a try and confirm that this has been resolved?

@cachedout cachedout removed the Blocker label Mar 20, 2015

@rallytime

This comment has been minimized.

Show comment
Hide comment
@rallytime

rallytime Mar 21, 2015

Contributor

There is still one bug that we are working out that I recently discovered when running a highstate command in a particular manner. I'll post here with an update when we have it resolved.

Contributor

rallytime commented Mar 21, 2015

There is still one bug that we are working out that I recently discovered when running a highstate command in a particular manner. I'll post here with an update when we have it resolved.

@cachedout

This comment has been minimized.

Show comment
Hide comment
@cachedout

cachedout Mar 24, 2015

Contributor

I am confident this is now fixed on the HEAD of 2014.7. I am going to go ahead and close this issue. If anybody sees this crop up again, please feel free to leave a comment here and we'll be happy to re-open it. :]

Contributor

cachedout commented Mar 24, 2015

I am confident this is now fixed on the HEAD of 2014.7. I am going to go ahead and close this issue. If anybody sees this crop up again, please feel free to leave a comment here and we'll be happy to re-open it. :]

@cachedout cachedout closed this Mar 24, 2015

@jfindlay jfindlay added Core and removed Core labels May 26, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment