Salt master cored when accessing grains

[shealutton]

I tried to report the grains for a server. Salt cored out. I tried again, and got no answer. I tried a third time and was successful.

Try #1.
[root@jump ~]$ salt -v E7CH70101 grains.items
Bad file descriptor (epoll.cpp:81)
Aborted (core dumped)

Try #2 (about 10 seconds later)
[root@jump ~]$ salt -v E7CH70101 grain.items

Executing job with jid 20140801113621586265

E7CH70101:
'grain.items' is not available.

Try #3 (a few seconds later).
[root@jump ~]$ salt -v E7CH70101 grains.items

Executing job with jid 20140801113634054879

E7CH70101:
cpu_model: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
cpuarch: AMD64
... (Success)

I ran this:
gdb python core.27697
(gdb) bt full
#0 0x0000003a786328e5 in raise () from /lib64/libc.so.6

No symbol table info available.
#1 0x0000003a786340c5 in abort () from /lib64/libc.so.6

No symbol table info available.
#2 0x00007f890a294489 in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#3 0x00007f890a2942cd in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#4 0x00007f890a2a38b8 in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#5 0x00007f890a2a3a2c in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#6 0x00007f890a293ce2 in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#7 0x00007f890a2b11fb in ?? () from /usr/lib64/libzmq.so.3

No symbol table info available.
#8 0x0000003a78e07851 in start_thread () from /lib64/libpthread.so.0

No symbol table info available.
#9 0x0000003a786e894d in clone () from /lib64/libc.so.6

No symbol table info available.

Hope that helps.
Shea Lutton

[cachedout]

Intesting. Could we please see a salt --versions? A ZeroMQ upgrade may be in order but it's a little tough to tell.

[shealutton]

[root@jump ~]$ cat /etc/centos-release
CentOS release 6.4 (Final)
[root@jump ~]$ salt --versions
Salt: 2014.1.7
Python: 2.6.6 (r266:84292, Jul 10 2013, 22:48:45)
Jinja2: 2.2.1
M2Crypto: 0.20.2
msgpack-python: 0.1.13
msgpack-pure: Not Installed
pycrypto: 2.0.1
PyYAML: 3.10
PyZMQ: 2.2.0.1
ZMQ: 3.2.4

The minion was 2014.1.5 on Windows.

[cachedout]

There were some reports a long while ago about instability with the versions you're running on CentOS 6.4, though they didn't report a core dump like you are. Might be worth seeing if you can upgrade/downgrade the ZMQ pieces: #2674

[cachedout]

Since we didn't hear back here and this dump clearly shows ZMQ as being the culprit, I'm closing this since there isn't anything we can really do on the Salt side.