Crash on unicode scripts

[Wilfred]

Given the following Python script:

# coding=utf-8

def function():
    """€

    """
    pass

Putting point on function and editing it gives me:

deferred error : (error "UnicodeDecodeError('ascii', 'function()\\\\n\\\\n\\xe2\\x82\\xac\\\\n\\\\n    ', 14, 15, 'ordinal not in range(128)')")

I believe my emacs-jedi setup is correct, it generally works well. This bug is just a slight annoynce when I'm editing this file, since my minibuffer keeps flashing up.

[tkf]

Can you try dev version of Jedi? You need to clone https://github.com/davidhalter/jedi, check out dev branch and then install it. I guess this is could be the bug which is already fixed in the dev version.

[Wilfred]

The dev version of jedi gives me the same behaviour. I ran the following commands:

$ git clone git://github.com/davidhalter/jedi.git
$ cd jedi
$ git checkout dev
$ git show
commit 406e70d51ae15316adb827363bc75cdfa4131aad
Merge: 336d35e 07075a7
Author: David Halter <davidhalter88@gmail.com>
Date:   Sat Apr 20 12:49:30 2013 -0700

    Merge pull request #204 from davidhalter/testing

    Make run.py testing reusable
$ sudo pip install .

I still get the traceback above.

[tkf]

Hmm... Can you try to full stack trace from Python? You can get it by playing with the option for jediepcserver.py:
http://tkf.github.io/emacs-jedi/#jedi:server-args

Or pahapse the easiest (becase it is documented) way to do is to run in debug mode:
http://tkf.github.io/emacs-jedi/#jedi:toggle-debug-server

Also please run M-x jedi:show-jedi-version and paste the result here.

[Wilfred]

jedi version:

((:version "2.7.4 (default, Apr  6 2013, 19:20:36) \n[GCC 4.8.0]" :name "sys" :file nil)
 (:version "0.5.5" :name "jedi" :file "/usr/lib/python2.7/site-packages/jedi/__init__.pyc")
 (:version "0.0.3" :name "epc" :file "/usr/lib/python2.7/site-packages/epc/__init__.pyc")
 (:version "0.0.2" :name "sexpdata" :file "/usr/lib/python2.7/site-packages/sexpdata.pyc"))

I'm not sure how to get a stack trace. I ran M-x jedi:toggled-debug-server, then ran the command:

$ python2 jediepcserver.py --port-file jedi-port.log --ipdb

in a shell. I can then see the port in the jedi-port.log file. However, it's not clear to me how to get Emacs to talk to this jediepcserver instance.

I also tried executing M-x jedi:stop-server, M-x jedi:start-dedicated-server and passed --pdb as an argument. This doesn't seem to change anything. Passing --help to jedi:start-dedicated-server works as expected, so I'm not sure why I'm not getting a stacktrace.

[tkf]

Here is how to use jedi:toggle-debug-server:

1. Go to the buffer with the problematic code sample (the one with unicode).
2. Type M-x jedi:toggle-debug-server RET. It should copy command line to the kill-ring. You will see the message saying so.
3. Open terminal, paste the command line and hit enter. This should start the EPC server.
4. Go back to the buffer in Emacs then execute some Jedi commands. You should see many log lines in the terminal when you execute the commands.
5. Do some problematic command (requesting document for function with unicode document or something). This should print the stack trace in the terminal. Also you should see the pdb (or ipdb) prompt. At this point, Jedi stops working until you exit the pdb session by C-d (or q RET).

[Wilfred]

Perfect. Here's the traceback:

$ python2 jediepcserver.py --port-file jedi-port.log --ipdb
(AutoLog) Called: EPCHandler.setup()
(AutoLog) Returns: EPCHandler.setup(...) = None
(AutoLog) Called: EPCHandler.handle()
receiving...
received: length = 174; data = '(call 4 complete ("# coding=utf-8\n\nimport os\n\ndef functionjkjl():\n    \\"\\"\\"\xe2\x82\xacjjkl\n\n    \\"\\"\\"\n    os.path.split()\n    pass\n" 5 16 "/home/wilfred/scratch/some_unicode.py"))\n'
(AutoLog) Called: EPCHandler._handle('(call 4 complete ("# coding=utf-8\n\nimport os\n\ndef functionjkjl():\n    \\"\\"\\"\xe2\x82\xacjjkl\n\n    \\"\\"\\"\n    os.path.split()\n    pass\n" 5 16 "/home/wilfred/scratch/some_unicode.py"))\n')
(AutoLog) Called: EPCHandler._handle_call(4, Symbol(u'complete'), [u'# coding=utf-8\n\nimport os\n\ndef functionjkjl():\n    """\u20acjjkl\n\n    """\n    os.path.split()\n    pass\n', 5, 16, u'/home/wilfred/scratch/some_unicode.py'])
(AutoLog) Returns: EPCHandler._handle_call(...) = ['return', 4, [{'doc': 'functionjkjl()\n\n\xe2\x82\xacjjkl\n\n    ', 'symbol': 'f', 'word': u'functionjkjl', 'description': u'function: some_unicode.functionjkjl'}]]
(AutoLog) Called: EPCHandler._send('return', 4, [{'doc': 'functionjkjl()\n\n\xe2\x82\xacjjkl\n\n    ', 'symbol': 'f', 'word': u'functionjkjl', 'description': u'function: some_unicode.functionjkjl'}])
ERROR:epc:UnicodeDecodeError('ascii', 'functionjkjl()\\n\\n\xe2\x82\xacjjkl\\n\\n    ', 18, 19, 'ordinal not in range(128)')
UnicodeDecodeError('ascii', 'functionjkjl()\\n\\n\xe2\x82\xacjjkl\\n\\n    ', 18, 19, 'ordinal not in range(128)')
> /usr/lib/python2.7/site-packages/sexpdata.py(90)uformat()
     89     """Alias of ``unicode(s).format(...)``."""
---> 90     return unicode(s).format(*args, **kwds)
     91 

[tkf]

Thank you. I think now I know how to reproduce the problem in my machine. But I don't have enough time right now so it may take sometime to fix.

[tkf]

It turned out to be sexpdata's bug. Upgrade sexpdata then it should be fine. If you use make to setup Python requirements, use make clean requirements to upgrade.

[Wilfred]

Great, I can confirm this works. Could you please update epc to depend on sexpdata>=0.0.3? Otherwise, installing from the requirements doesn't change the installed version and users will stay on the old version.

[tkf]

Thanks. Good point. I will do it.