Scoping is weird with python3 in python interpolation and crashes.

[SirVer]

I tried list comprehensions inside python interpolated snippets and it seems not to work. When trying to access local variables (say for example the 're' module) it throws NameError and that the variable is not global (in the following case I defined arg = t[4] before the list comprehension).

An error occured. This is either a bug in UltiSnips or a bug in a
snippet definition. If you think this is a bug, please report it to
https://bugs.launchpad.net/ultisnips/+filebug.

Following is the full stack trace:
Traceback (most recent call last):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 23, in wrapper
return f(self, _args, *_kwds)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 602, in expand_or_jump
rv = self._try_expand()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 934, in _try_expand
self._do_snippet(snippet, before, after)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 908, in _do_snippet
si = snippet.launch(text_before, self._visual_content, None, start, end)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 420, in launch
last_re = self._last_re, globals = self._globals)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 34, in init
self.update_textobjects()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 70, in update_textobjects
if obj._update(done, not_done):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_python_code.py", line 216, in _update
compatible_exec(self._code, self._globals, local_d)
File "", line 9, in
File "", line 9, in
NameError: global name 'arg' is not defined

:py3 import sys; print(sys.version):
3.3.2+ (default, Oct 9 2013, 14:53:41)
[GCC 4.8.1]

Launchpad Details: #LP1259349 Răzvan-Cosmin Rădulescu - 2013-12-10 01:02:18 +0100

[SirVer]

We do not limit python in any way in the snippets. So you are able to use all of it.

It would be helpful if you could paste a minimal snippet that shows the problem you have, without an example I will have a hard time reproducing the bug.

Launchpad Details: #LPC SirVer - 2013-12-10 08:20:07 +0100

[SirVer]

Sorry I thought it's really clear from my comment. I define the RETURNS object in the global part, but the problem is with the "re" object, probably with RETURNS to but it doesn't go that far. Here is a non-working snippet:

snippet def "function with docstrings" b
def ${1:function_name}(!p if snip.indent: snip.rv = 'self' + (", " if len(t[2]) else "")${2:parameters}):
!p snip.rv = tripple_quotes(snip)${4:@todo:Summary for $1.}

${5:@todo:Description for $1.}`!p

snip.rv = ""
snip >> 1

this is the non-working part

print([re.search(r, 'return ksdjg.') for r in RETURNS])

end of non-working part

args = get_args(t[2])
if args:
write_docstring_args(args, snip)
function_returns = False
for regex in RETURNS:
if re.search(regex, t[4]) or re.search(regex, t[5]):
function_returns = True
break
if function_returns:
format_return(snip)
snip << 1
snip += tripple_quotes(snip) `

${0:pass}

endsnippet

And here is the error:

An error occured. This is either a bug in UltiSnips or a bug in a
snippet definition. If you think this is a bug, please report it to
https://bugs.launchpad.net/ultisnips/+filebug.

Following is the full stack trace:
Traceback (most recent call last):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 23, in wrapper
return f(self, _args, *_kwds)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 602, in expand_or_jump
rv = self._try_expand()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 934, in _try_expand
self._do_snippet(snippet, before, after)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 908, in _do_snippet
si = snippet.launch(text_before, self._visual_content, None, start, end)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 420, in launch
last_re = self._last_re, globals = self._globals)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 34, in init
self.update_textobjects()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 70, in update_textobjects
if obj._update(done, not_done):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_python_code.py", line 216, in _update
compatible_exec(self._code, self._globals, local_d)
File "", line 6, in
File "", line 6, in
NameError: global name 're' is not defined

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-10 11:47:38 +0100

[SirVer]

Other than this top notch job, I really like UltiSnips :).

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-10 11:49:12 +0100

[SirVer]

Glad you like the plugin. However, I can still not reproduce your problem as your example is still incomplete (i.e. there is no globals section and tripple_quotes is not defined). Try removing stuff from your example till you have a minimal failing case and we'll advance in this issue much quicker. Please also try to post a minimal case.

Launchpad Details: #LPC SirVer - 2013-12-11 12:52:46 +0100

[SirVer]

I attached my whole snippets file and hope it helps. I made only slight modifications to the original python.snippets file in the global portion. The part that gives the error is located at line 312 (the only print statement in the file to make it easy to find).

Thank you. Hope this is enough information...

Then to get the erorr I only: def... and then it gives me:

An error occured. This is either a bug in UltiSnips or a bug in a
snippet definition. If you think this is a bug, please report it to
https://bugs.launchpad.net/ultisnips/+filebug.

Following is the full stack trace:
Traceback (most recent call last):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 23, in wrapper
return f(self, _args, *_kwds)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 602, in expand_or_jump
rv = self._try_expand()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 934, in _try_expand
self._do_snippet(snippet, before, after)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 908, in _do_snippet
si = snippet.launch(text_before, self._visual_content, None, start, end)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/init.py", line 420, in launch
last_re = self._last_re, globals = self._globals)
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 34, in init
self.update_textobjects()
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_snippet_instance.py", line 70, in update_textobjects
if obj._update(done, not_done):
File "/home/razvan/.vim/bundle/ultisnips/plugin/UltiSnips/text_objects/_python_code.py", line 216, in _update
compatible_exec(self._code, self._globals, local_d)
File "", line 10, in
File "", line 10, in
NameError: global name 're' is not defined

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-11 14:53:43 +0100

[SirVer]

dude - what about the 'minimal example' did you not understand? I now have an error, but it is not the one that you see:

TypeError: expected string or buffer

and if I do not pass a list as second argument to re.search, but a string I see no more error and the snippet seems to work fine.

this is useless - finding the interaction problem that make the trouble in such a huge file is impossible for me - you have to do it. Reduce it to the smallest sample that still shows the error and we might make some progress.

Launchpad Details: #LPC SirVer - 2013-12-11 18:05:34 +0100

[SirVer]

ok, so I thought it didn't really matter how much clutter there is in the script since the only problem is with the comprehension lists. Sorry about that. Here, this is a non working minimal snippet that I have, this is the only thing in python.snippets:

snippet test "Test comprehension"
!p [re.search('a', i) for i in ['kdgj']]
endsnippet

and it throws me the error "NameError: global name 're' is not defined" when I do 'test'.

If I use:

snippet test "Test comprehension"
!p re.search('a', 'skljg')
endsnippet

then all is well, I don't get any error at all. I mentioned at the beginning that I use Python 3. Might this be causing the problem? Sorry again for not following your adivce but if you can't reproduce this then I don't know.

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-11 20:56:49 +0100

[SirVer]

The snippet does not fail for me. It is possible that python3 might be the culprit, but I have my doubts about this. If you have access to both python2 and 3 inside of vim you can try forcing it to be used: https://github.com/SirVer/ultisnips/blob/master/doc/UltiSnips.txt#L108

From where did you install the plugin and which version are you using? If you have it from github, which fork did you clone?

Launchpad Details: #LPC SirVer - 2013-12-12 07:52:37 +0100

[SirVer]

I just tried it and yes, with Python 2 I don't get the error. I couldn't do this from home cause over there I have VIM only with +python3. If I switch to Python 3 (let g:UltiSnipsUsePythonVersion = 3) then I get the error.

I'm using UltiSnips from https://github.com/SirVer/ultisnips the latest version (downloaded 2 days ago) with NeoBundle... if there's anything I can do let me know.

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-12 11:29:08 +0100

[SirVer]

I can reproduce the error now - I had to rebuild a Vim version with python 3. I cannot explain it though - the exec call gets the correct code which also works outside of Vim (in a standalone python3 session). I do not now what is going on, but I guess there is some bad interplay with the way python3 is embedded in Vim and UltiSnips snippet execution. This one will be very hard to trace down.

Launchpad Details: #LPC SirVer - 2013-12-14 16:20:56 +0100

[SirVer]

Hm... sorry for the bad news.

Since we are throwing ideas here... do you think this might be connected to how Python 3 interprets the strings as compared with 2?... since in Python 3 everything is an UTF-8 string by default. My thought goes this ways because I had an error when trying to load this plugin: https://github.com/marijnh/tern_for_vim besides the usual exception with commas and some import statements, I had to change the re.match call because the second argument was a bytes string instead... so I had to attach a .decode() to it. I know this isn't much to go on... but I'm hoping there might be something simple as this...

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-14 23:55:02 +0100

[SirVer]

I doubt that is it, because if you use re before the list comprehension, it just works. I have no idea why it does not work inside of the list comprehension - that is all weird to me.

Launchpad Details: #LPC SirVer - 2013-12-15 12:38:13 +0100

[SirVer]

Well, if there's anything else I can do to help with this let me know and thanks.

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2013-12-15 14:14:50 +0100

[SirVer]

[Expired for UltiSnips because there has been no activity for 60 days.]

Launchpad Details: #LPC Launchpad Janitor - 2014-02-14 05:18:50 +0100

[SirVer]

Sorry - I dropped the ball on that. I invested a few hours getting a python3 Vim running and could immediately reproduce the error. I believe the bug is due to scoping changes inside of python that have been done in py3 (see for example [1] and [2]). I believe there is also some interplay with the import hook inside of Vim - but I was not able to create a test case inside of Vim that showed the issues. The basic problem is that imports are done into the wrong dictionary when using exec() and then the list comprehension does not see it - not too sure why. Long story short, by only using one dictionary for exec() the problem is fixed - the cost is slightly bigger scope and longer execution type for snippet python code - but it is not noticeable for a human, so it should not matter. The fix is on github in sha 635dc63 and will be soon here on launchpad too.

sorry that you had to wait so long for a fix.

[1] http://stackoverflow.com/questions/13905741/accessing-class-variables-from-a-list-comprehension-in-the-class-definition
[2] http://docs.python.org/3/reference/compound_stmts.html#class-definitions

Launchpad Details: #LPC SirVer - 2014-02-14 19:28:46 +0100

[SirVer]

Not at all, thank you so much for your time on this. I'll be sure to check it out.

Launchpad Details: #LPC Răzvan-Cosmin Rădulescu - 2014-02-14 19:44:58 +0100