msgpack Exception: OverflowError: long too big to convert #12185

bbinet · 2014-04-22T12:48:20Z

Running salt-minion on Debian on a cubieboard v2 board, when running saltutil.kill_job function I get the following exception (this has also appeared once on a state.highstate run):

root@hl-srv-1-dev:~/salt-fileserver/pillar/master# salt '*' saltutil.kill_job 20140422142858458634
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
OverflowError: long too big to convert
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 135, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 138, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 553, in cmd_cli
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 272, in run_job
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1385, in pub
    payload = sreq.send(payload_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 93, in send
    return self._uncrypted_transfer(load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 86, in _uncrypted_transfer
    return self.sreq.send(self.crypt, load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 181, in send
    pkg = self.serial.dumps(payload)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 118, in dumps
    return msgpack.dumps(msg)
  File "_msgpack.pyx", line 169, in msgpack._msgpack.packb (msgpack/_msgpack.c:2384)
  File "_msgpack.pyx", line 153, in msgpack._msgpack.Packer.pack (msgpack/_msgpack.c:2020)
  File "_msgpack.pyx", line 136, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1761)
  File "_msgpack.pyx", line 136, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1761)
  File "_msgpack.pyx", line 142, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1862)
  File "_msgpack.pyx", line 105, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1345)
OverflowError: long too big to convert
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 135, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 138, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 553, in cmd_cli
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 272, in run_job
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1385, in pub
    payload = sreq.send(payload_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 93, in send
    return self._uncrypted_transfer(load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 86, in _uncrypted_transfer
    return self.sreq.send(self.crypt, load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 181, in send
    pkg = self.serial.dumps(payload)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 118, in dumps
    return msgpack.dumps(msg)
  File "_msgpack.pyx", line 169, in msgpack._msgpack.packb (msgpack/_msgpack.c:2384)
  File "_msgpack.pyx", line 153, in msgpack._msgpack.Packer.pack (msgpack/_msgpack.c:2020)
  File "_msgpack.pyx", line 136, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1761)
  File "_msgpack.pyx", line 136, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1761)
  File "_msgpack.pyx", line 142, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1862)
  File "_msgpack.pyx", line 105, in msgpack._msgpack.Packer._pack (msgpack/_msgpack.c:1345)
OverflowError: long too big to convert
root@hl-srv-1-dev:~/salt-fileserver/pillar/master#

I suppose this is because salt-minion is running on an armhf Debian system?
Is salt/msgpack supposed to work on armhf architectures?

Please tell me how can I help provide more information to investigate this issue.

For information, here are the versions reports for salt-minion:

root@cubie:~# salt --versions-report
           Salt: 2014.1.3
         Python: 2.7.3 (default, Mar 14 2014, 17:55:54)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.10
          PyZMQ: 13.1.0
            ZMQ: 3.2.3

And salt-master:

root@hl-srv-1-dev:~# salt-call --versions-report
           Salt: 2014.1.3
         Python: 2.7.3 (default, Mar 13 2014, 11:03:55)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.10
          PyZMQ: 13.1.0
            ZMQ: 3.2.3

The text was updated successfully, but these errors were encountered:

bbinet · 2014-04-22T12:53:23Z

Sorry, it hasn't occured on a state.highstate function but only when running either saltutil.find_job or saltutil.kill_job functions.

cachedout · 2014-04-22T16:52:46Z

This is quite interesting. The stacktrace is clearly coming from inside msgpack and so this may end up becoming an upstream bug with the msgpack folks but we'll investigate.

s0undt3ch · 2014-04-22T17:08:36Z

We're converting the jid to int right from the CLI when parsing args.

As a workaround we could try not to pass jid as an int for this specific platform?

Pedro Algarvio @ Phone

----- Reply message -----
From: "Mike Place" notifications@github.com
To: "saltstack/salt" salt@noreply.github.com
Subject: [salt] msgpack Exception: OverflowError: long too big to convert (#12185)
Date: Tue, Apr 22, 2014 17:52
This is quite interesting. The stacktrace is clearly coming from inside msgpack and so this may end up becoming an upstream bug with the msgpack folks but we'll investigate.

—
Reply to this email directly or view it on GitHub.

cachedout · 2014-04-22T19:51:59Z

@s0undt3ch That might work, though we'll have to catch it on the other side and make sure we have it cast it back correctly or we'll run into more problems potentially.

bbinet · 2014-04-23T08:15:33Z

Do you want me to test something to help with this issue?

cachedout · 2014-04-23T15:29:38Z

@bbinet We're not yet in agreement on the right approach here, I think. Once we figure out the best way to solve this we'll definitely appreciate your help in testing this thoroughly. Thanks!

aryanet · 2014-04-23T20:36:46Z

I have the same issue with the same version. Although I am running Ubuntu 12.04. Why not keep the jids as strings for all platforms as a solution?

s0undt3ch · 2014-04-23T22:15:01Z

@aryanet on which platform are you on? ARM?

aryanet · 2014-04-23T23:57:42Z

TL;DR: x86_64
Dedicated machine with this CPU:
Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
http://ark.intel.com/products/65727/Intel-Xeon-Processor-E3-1270-v2-8M-Cache-3_50-GHz
Kernel:
Linux 3.2.0-60-generic #91-Ubuntu SMP Wed Feb 19 03:54:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

basepi · 2014-04-24T22:27:20Z

This is due to a change we made with argument parsing. I think @terminalmage already fixed it, if I remember correctly. He took long types out of the list of things we wanted to convert to.

basepi · 2014-04-24T22:28:32Z

(I may be wrong, because I think his fix was in 2014.1.3, which is what you're running.)

replicant0wnz · 2014-04-30T19:46:43Z

Can confirm this bug in 2014.1.3 on x86_64 when doing a saltutil.find_job

replicant0wnz · 2014-04-30T19:52:34Z

Is there a quick fix for this I can apply to my own branch while the debate continues on what the proper fix is? :)

basepi · 2014-04-30T21:14:12Z

Yes, the temporary fix is to double quote your jid:

salt '*' saltutil.find_job '"20140422142858458634"'

This will ensure that it is consumed as a string.

basepi · 2014-04-30T21:17:07Z

@terminalmage's fix definitely made it in, so there must be an additional problem here. The double quoting should work for now, but we'll investigate an actual fix.

replicant0wnz · 2014-05-01T14:40:06Z

@basepi doh! Yup that did the trick. FYI it does seem to be via the command line only as it works via the API.

terminalmage · 2014-05-01T16:44:15Z

This has been found and fixed in #12460, and the fix will be in 2014.1.4. Here's an example of the now-fixed behavior:

root@saltmine:~erik/saltdev# salt saltmine saltutil.find_job 20140501112515756618
saltmine:
    ----------
    arg:
        - sleep 120
    fun:
        cmd.run
    jid:
        20140501112515756618
    pid:
        4559
    ret:

    tgt:
        saltmine
    tgt_type:
        glob
    user:
        root
root@saltmine:~erik/saltdev# salt saltmine saltutil.kill_job 20140501112515756618
saltmine:
    Signal 9 sent to job 20140501112515756618 at pid 4559
root@saltmine:~erik/saltdev# salt saltmine saltutil.find_job 20140501112515756618
saltmine:
    ----------

And in the terminal that originally ran the command:

root@saltmine:~erik/saltdev# salt saltmine cmd.run 'sleep 120' -v
Executing job with jid 20140501112515756618
-------------------------------------------

Execution is still running on saltmine
Execution is still running on saltmine
saltmine:
    Minion did not return

basepi · 2014-05-01T20:17:33Z

Awesome!

7oku · 2014-11-27T00:39:06Z

I have the same bug still in 2014.7.0 (but want to mention i could workaround this by wrapping it in quotes as mentioned above ...):

salt --version
salt 2014.7.0 (Helium)

salt \* saltutil.kill_job 20141125115342216725
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
OverflowError: long too big to convert
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 172, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 569, in cmd_cli
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 282, in run_job
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1379, in pub
    payload = sreq.send(payload_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 237, in send
    return self._uncrypted_transfer(load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 230, in _uncrypted_transfer
    return self.sreq.send(self.crypt, load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 202, in send
    pkg = self.serial.dumps(payload)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 115, in dumps
    return msgpack.dumps(msg)
  File "_packer.pyx", line 259, in msgpack._packer.packb (msgpack/_packer.cpp:259)
  File "_packer.pyx", line 184, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:184)
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  File "_packer.pyx", line 124, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:124)
OverflowError: long too big to convert
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 240, in salt_main
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/__init__.py", line 172, in run
    for full_ret in cmd_func(**kwargs):
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 569, in cmd_cli
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 282, in run_job
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1379, in pub
    payload = sreq.send(payload_kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 237, in send
    return self._uncrypted_transfer(load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/transport/__init__.py", line 230, in _uncrypted_transfer
    return self.sreq.send(self.crypt, load, tries, timeout)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 202, in send
    pkg = self.serial.dumps(payload)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 115, in dumps
    return msgpack.dumps(msg)
  File "_packer.pyx", line 259, in msgpack._packer.packb (msgpack/_packer.cpp:259)
  File "_packer.pyx", line 184, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:184)
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  File "_packer.pyx", line 159, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:159)
  File "_packer.pyx", line 173, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:173)
  File "_packer.pyx", line 124, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:124)
OverflowError: long too big to convert

basepi · 2014-12-01T20:17:03Z

Is your minion also on 2014.7.0, @7oku?

spo0nman · 2015-03-02T21:43:15Z

This is also a problem with custom grains. If a grain returned is long int, it breaks salt.

'''
File "/opt/blue-python/2.7/lib/python2.7/site-packages/msgpack/init.py", line 47, in packb
return Packer(**kwargs).pack(o)
File "_packer.pyx", line 224, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:224)
File "_packer.pyx", line 226, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:226)
File "_packer.pyx", line 185, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:185)
File "_packer.pyx", line 185, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:185)
File "_packer.pyx", line 185, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:185)
File "_packer.pyx", line 141, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:141)
OverflowError: long int too large to convert
'''

cachedout added Bug labels Apr 22, 2014

cachedout added this to the Approved milestone Apr 22, 2014

terminalmage mentioned this issue May 1, 2014

Fix CLI arg parsing #12460

Merged

basepi closed this as completed May 1, 2014

thenewwazoo mentioned this issue May 20, 2015

Bad pillar error caused by numerical pillar data (and msgpack exception) #23993

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

msgpack Exception: OverflowError: long too big to convert #12185

msgpack Exception: OverflowError: long too big to convert #12185

bbinet commented Apr 22, 2014

bbinet commented Apr 22, 2014

cachedout commented Apr 22, 2014

s0undt3ch commented Apr 22, 2014

cachedout commented Apr 22, 2014

bbinet commented Apr 23, 2014

cachedout commented Apr 23, 2014

aryanet commented Apr 23, 2014

s0undt3ch commented Apr 23, 2014

aryanet commented Apr 23, 2014

basepi commented Apr 24, 2014

basepi commented Apr 24, 2014

replicant0wnz commented Apr 30, 2014

replicant0wnz commented Apr 30, 2014

basepi commented Apr 30, 2014

basepi commented Apr 30, 2014

replicant0wnz commented May 1, 2014

terminalmage commented May 1, 2014

basepi commented May 1, 2014

7oku commented Nov 27, 2014

basepi commented Dec 1, 2014

spo0nman commented Mar 2, 2015

msgpack Exception: OverflowError: long too big to convert #12185

msgpack Exception: OverflowError: long too big to convert #12185

Comments

bbinet commented Apr 22, 2014

bbinet commented Apr 22, 2014

cachedout commented Apr 22, 2014

s0undt3ch commented Apr 22, 2014

cachedout commented Apr 22, 2014

bbinet commented Apr 23, 2014

cachedout commented Apr 23, 2014

aryanet commented Apr 23, 2014

s0undt3ch commented Apr 23, 2014

aryanet commented Apr 23, 2014

basepi commented Apr 24, 2014

basepi commented Apr 24, 2014

replicant0wnz commented Apr 30, 2014

replicant0wnz commented Apr 30, 2014

basepi commented Apr 30, 2014

basepi commented Apr 30, 2014

replicant0wnz commented May 1, 2014

terminalmage commented May 1, 2014

basepi commented May 1, 2014

7oku commented Nov 27, 2014

basepi commented Dec 1, 2014

spo0nman commented Mar 2, 2015