os.path.join returning TypeError in windows

[jsarver]

Shortly after updating to] 0.2.4 xonsh I noticed if If I try to run the following:

import os
os.path.join("one","two")

I get the error:
TypeError: 'Str' object is not iterable

If I exit xonsh and run it from a normal python prompt is I get the expected output
'one\two'

Was hoping some other windows user can confirm this issues I don't have another windows box to test this one. I uninstalled prompt-toolkit to see if it might be something with thatbut still get the same issue. So far can only confirm it is happening in windows. I'm running running python 3.4.3

[scopatz]

Out of curiosity have you set $FORCE_POSIX_PATHS = True?

[scopatz]

Thanks for reporting!

[jsarver]

I forgot to mention, to be on the safe side, I renamed my xonshrc to xonshrc.old so it wouldn't load on startup in case that was causing some weirdness so I'm pretty sure that was not enabled. I just tried setting it to True and it still give me the same error.

I'm curious, is there a way to simulate a xonsh environment and running a xonsh.shell command from within a script? I've had a hard time troubleshooting since I'm unsure how use this with a debugger. I also looked through your unit tests but didn't see how you would test this. Will look further when I get some more time.

Thanks!

[adqm]

I'm curious, is there a way to simulate a xonsh environment and running a xonsh.shell command from within a script?

Definitely! The default method of xonsh.shell.Shell is what gets called on each line of input. You can grab it like:

$ python3
>>> from xonsh.shell import Shell
>>> s = Shell()
>>> s.default('echo "hi"')
hi
>>> s.default('import os')
>>> s.default('os.path.join("hello", "hey")')
'hello/hey'

[scopatz]

I've had a hard time troubleshooting since I'm unsure how use this with a debugger.

Any python debugger should be usable with xonsh :).

[adqm]

Also, this is a very strange bug. I recently destroyed all my Windows environments, but I can try to get something set up in a VM tonight to test.

The error message makes me think that maybe starargs aren't being handled properly? But it doesn't make sense to me that that would behave differently on Windows...

[jsarver]

Thanks for the info! I'll try to do some more testing later tonight.

[adqm]

Hmmmm. I just tried and couldn't reproduce this. From Windows (Python 3.5.1):

adam@adam-PC ~\Documents\xonsh $ import os
adam@adam-PC ~\Documents\xonsh $ os.path.join("hey", "hi")
'hey\\hi'

[adqm]

This is a real mystery to me, @jsarver!

TypeError: 'Str' object is not iterable

I'm extra confused here, because str objects are iterable. Is Str really capitalized? Can you set $XONSH_SHOW_TRACEBACK = True and run again, and paste the whole output here?

There was only one other Google hit for "str object is not iterable", here. I think we can use the same workaround they did if we can figure out where in the code this is coming from.

[jsarver]

I had a chance to run it again with the stacktrace turned on, here is what it gave me.

 s.default('os.path.join("one","two")')
    Traceback (most recent call last):
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\base_shell.py", line 168, in push
        locs=self.ctx)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\execer.py", line 95, in compile
        tree = self.parse(input, ctx, mode=mode)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\execer.py", line 70, in parse
        tree = self._parse_ctx_free(input, mode=mode)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\execer.py", line 151, in _parse_ctx_free
        debug_level=self.debug_level)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\parser.py", line 265, in parse
        tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\ply\yacc.py", line 331, in parse
        return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\ply\yacc.py", line 1106, in parseopt_notrack
        p.callable(pslice)
      File "C:\Miniconda3\envs\sdmtools\lib\site-packages\xonsh\parser.py", line 2114, in p_arglist
        for arg in p1:
    TypeError: 'Str' object is not iterable

[scopatz]

So this is a Python v3.4 issue only, if you look at that part of the code. Still not sure what is wrong, though. Any chance you could run the through pdb and print p1 for us?

[jsarver]

I'm using the pycharm debugger, not sure if this exactly what you need but here is p1.

p1 = {Str} <_ast.Str object at 0x0000000004A8FCF8>
 _attributes = {tuple} ('lineno', 'col_offset')
  __len__ = {int} 2
  0 = {str} 'lineno'
  1 = {str} 'col_offset'
 _fields = {tuple} ('s',)
  __len__ = {int} 1
  0 = {str} 's'
 col_offset = {int} 18
 lineno = {int} 1
 s = {str} 'one'

[scopatz]

@jsarver, does the parser instance have a method called p_argument_comma() out of curiosity?

[jsarver]

On that particular instance it doesn't appear to have it. I set a breakpoint for when that method gets defined and it did define it at some point much earlier in the stack then later on right before it tries to iterate over p1, when I inspect that object it doesn't show the method. Pasted what it has below just to make sure I'm looking at the right thing.

self = {Parser} <xonsh.parser.Parser object at 0x00000000043CDC50>
 _augassign_op = {dict} {'*=': <class '_ast.Mult'>, '**=': <class '_ast.Pow'>, '@=': None, '>>=': <class '_ast.RShift'>, '+=': <class '_ast.Add'>, '&=': <class '_ast.BitAnd'>, '^=': <class '_ast.BitXor'>, '//=': <class '_ast.FloorDiv'>, '|=': <class '_ast.BitOr'>, '/=': <class '_
 _comp_ops = {dict} {'is': <class '_ast.Is'>, ('not', 'in'): <class '_ast.NotIn'>, '<': <class '_ast.Lt'>, 'in': <class '_ast.In'>, '<=': <class '_ast.LtE'>, '==': <class '_ast.Eq'>, '!=': <class '_ast.NotEq'>, ('is', 'not'): <class '_ast.IsNot'>, '>=': <class '_ast.GtE'>, '>
 _factor_ops = {dict} {'-': <class '_ast.USub'>, '+': <class '_ast.UAdd'>, '~': <class '_ast.Invert'>}
 _last_yielded_token = {NoneType} None
 _term_binops = {dict} {'+': <class '_ast.Add'>, '//': <class '_ast.FloorDiv'>, '-': <class '_ast.Sub'>, '%': <class '_ast.Mod'>, '/': <class '_ast.Div'>, '*': <class '_ast.Mult'>, '@': None}
 col = {int} 24
 lexer = {Lexer} <xonsh.lexer.Lexer object at 0x00000000043CDFD0>
 lineno = {int} 1
 parser = {LRParser} <ply.yacc.LRParser object at 0x0000000004B52E80>
 precedence = {tuple} (('left', 'PIPE'), ('left', 'XOR'), ('left', 'AMPERSAND'), ('left', 'EQ', 'NE'), ('left', 'GT', 'GE', 'LT', 'LE'), ('left', 'RSHIFT', 'LSHIFT'), ('left', 'PLUS', 'MINUS'), ('left', 'TIMES', 'DIVIDE', 'DOUBLEDIV', 'MOD'), ('left', 'POW'))
 tokens = {tuple} ('LE', 'TILDE', 'RSHIFT', 'MOD', 'PIPEEQUAL', 'INDENT', 'DEDENT', 'RARROW', 'LT', 'AT', 'RSHIFTEQUAL', 'MODEQUAL', 'ATEQUAL', 'TIMESEQUAL', 'GT', 'COMMA', 'GE', 'DOUBLEDIV', 'NE', 'ELLIPSIS', 'PERIOD', 'PLUS', 'XOREQUAL', 'AMPERSANDEQUAL', 'DIVEQUAL', 'DOU

[scopatz]

OK, can you maybe try removing line 2149 of parser.py that reads if VER_MAJOR_MINOR <= VER_3_4 and dedenting the p_argument_comma() method inside it and seeing if the code works as expected? I suspect that ply is doing something fancy that I don't understand.

[jsarver]

No luck, unfortunately. Removed the Version check, dendented and get the same error.

[adqm]

p1 = {Str} <_ast.Str object at 0x0000000004A8FCF8>

Well, there's the problem. Somehow an argument_comma_list_opt is being parsed out as an ast.Str instead of a list. So ignore my discussion about iterable earlier; that makes sense to me.

What's extra weird now is that we shouldn't even be entering that branch, since that should only happen if we are parsing out a "stararg" kind of thing. The branch we're failing in is here:

My mistake. Misread the != as ==. So at least that mystery is gone. Still not sure how argument_comma_list_opt is being parsed out as an ast.Str instead of a list.

            elif lenp == 4 and p2 != '**':
                for arg in p1:
                    self._set_arg(p0, arg)
                self._set_arg(p0, p2)

I'll install Python 3.4 today and see if I can reproduce.

[adqm]

I can't reproduce, even with Python 3.4.3 installed. I just checked, and everything seems to be handled correctly on Windows under Python 3.4.3 from python.org. Perhaps conda is doing something weird?

[jsarver]

Installed vanilla 3.4.3 python and os.path.join works fine using xonsh now so I think you're right @adqm, it does appear to be conda specific. I'm going to try this on conda 3.5 to see if that version is affected too.

[scopatz]

Installed vanilla 3.4.3 python and os.path.join works fine using xonsh now so I think you're right @adqm, it does appear to be conda specific.

What the...?! Could it be a difference in PLY versions between the vanilla python and the conda python?

[jsarver]

It works fine in Conda Python 3.5.1 as well.

So the main difference between I've noticed between these parser objects is for some reason the arguments are getting transposed.

When I run in in conda 3.4.3 this is what the parser object looks like and what all the p* assignments looks like.

lenp = {int} 4
p = {YaccProduction} <ply.yacc.YaccProduction object at 0x00000000044E5E80>
 lexer = {Lexer} <xonsh.lexer.Lexer object at 0x000000000435D630>
 parser = {LRParser} <ply.yacc.LRParser object at 0x0000000004C58CF8>
 slice = {list} [arglist, argument, comma_argument_list, comma_opt]
 stack = {list} [$end, atom, trailer_list, LexToken(LPAREN,'(',1,12)]
p0 = {dict} {'kwargs': None, 'keywords': [], 'args': [], 'starargs': None}
p1 = {Str} <_ast.Str object at 0x000000000545A160>
p2 = {list} [<_ast.Str object at 0x000000000545A4E0>]
self = {Parser} <xonsh.parser.Parser object at 0x000000000435DE10>

And on the version of python where it works, here is what is looks like.

p = {YaccProduction} <ply.yacc.YaccProduction object at 0x0000000003EE7F60>
 lexer = {Lexer} <xonsh.lexer.Lexer object at 0x0000000003507CF8>
 parser = {LRParser} <ply.yacc.LRParser object at 0x0000000004246D30>
 slice = {list} [arglist, argument_comma_list, argument, comma_opt]
 stack = {list} [$end, atom, trailer_list, LexToken(LPAREN,'(',1,12)]
p0 = {dict} {'starargs': None, 'keywords': [], 'kwargs': None, 'args': []}
p1 = {list} [<_ast.Str object at 0x0000000004FC2F60>]
p2 = {Str} <_ast.Str object at 0x0000000004FC2E80>
self = {Parser} <xonsh.parser.Parser object at 0x0000000003507FD0>

It appears to me that for some reason p1 and p2 are getting switched around but still unsure why at this point.

[jsarver]

@scopatz I'm running ply 3.8 in both scenarios.

[scopatz]

Thanks @jsarver - That is weird. In the broken case it looks like it is matching against:

| argument comma_argument_list_opt comma_opt

Which is not a rule that is implemented.

[scopatz]

I have no idea how it got there, but the hacky fix is to put in a some code that ensures that p1 is a list and p2 is not.

[jsarver]

I just upgraded to 3.5 as my solution :). I'm willing to attempt your suggested fix and submit a pr if that's the route you'd like to go to resolve it though.

[scopatz]

I just upgraded to 3.5 as my solution :).

Works for me :)

I'm willing to attempt your suggested fix and submit a pr if that's the route you'd like to go to resolve it though.

That would be awesome if you could!

[den-run-ai]

not reproduced on windows 7 64 bit, python 3.4.3 from conda environment, 32bit version python:

-DENIS ~ # import os
-DENIS ~ # os.path.join("one","two")
'one\\two'
-DENIS ~ # python
Python 3.4.3 |Continuum Analytics, Inc.| (default, Dec  1 2015, 11:59:33) [MSC v
.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

[jsarver]

Such a curious issue. The only difference I can think of is I'm running 64 bit miniconda version of python not 32 bit but not sure how that would affect things. It works fine if I drop down to version 0.2.3 on my machine though so some change is adversely affecting it.

What is even stranger is if I download the source and install xonsh via myenv\python.exe setup.py install it works just fine so I have no idea what is going on. I guess if no one else reports the issue perhaps I have a hosed machine but I've built several clean new envs with conda just to test this.

[schnitzelbub]

Unfortunately not:

$ import os
$ os.path.join("/one", "/two")
TypeError: 'Str' object is not iterable
$ os.path.join("/one", "/two" )
TypeError: 'Str' object is not iterable
$ os.path.join(["/one", "/two"])
['/one', '/two']

but pathlib.Path is not affected:

$ import pathlib
$ pathlib.Path("/one").joinpath("./two")
PosixPath('/one/two')

[jsarver]

I don't want to muddy the waters any further but here is something I find really odd. If install xonsh 0.2.4 in python 3.4.2 it fails

activate xonshtest
pip install xonsh==0.2.4
python

>> import xonsh
>> from xonsh import Shell
>> print(xonsh.__file__)
>> s = Shell()
>> s.default("import os")
>> s.default("os.path.join('one', 'two')")

C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\__init__.py
TypeError: 'Str' object is not iterable

However, if I uninstall xonsh, and just run xonsh from withing the xonsh_master 0.2.4 code it works fine

pip uninstall xonsh
cd c:\xonsh_master
python
>> import xonsh
>> from xonsh import Shell
>> print(xonsh.__file__)
>> s = Shell()
>> s.default("import os")
>> s.default("os.path.join('one', 'two')")

C:\xonsh-master\xonsh\__init__.py
'one\\two'

[scopatz]

More importantly, does this affect xonsh v0.2.6?

[jsarver]

I get the same issue in 0.2.6 (using version installed via pip) and python 3.4.2 but 0.2.6 does seem to work fine in python 3.5.

[scopatz]

Ahh ok. Does this still happen on Python 3.4.2 on xonsh master then?

[jsarver]

I need to test that for 0.2.6, will let you know in a few.

[jsarver]

Works just fine in the xonsh_master

pip uninstall xonsh
cd c:\xonsh_master
python

import xonsh
from xonsh.shell import Shell
print(xonsh.__file__)
print(xonsh.__version__)
s = Shell()
s.default("import os")
s.default("os.path.join('one', 'two')")

c:\xonsh-master\xonsh\__init__.py
0.2.6
'one\\two'

[scopatz]

Works just fine in the xonsh_master

Great! I think that we will just wait until the next release then, if that is OK with everyone.

[jsarver]

That's fine with me, I use 3.5 mostly anyways :), should I reopen this issue then?

I'm gonna tinker with this and see if I can narrow down why it works in source and not in the pip version. Is there a way pass XONSH_SHOW_TRACEBACK = True to the Shell() or do I need to pass that info somewhere else? It didn't seem to work when putting that in my default .xonshrc file.

[schnitzelbub]

Ok, this is not specific to os.path.join but rather an issue of str.replace or argument passing in general!?

xonsh:

$ "x".replace("x","y")
TypeError: 'Str' object is not iterable

vs.

ipython:

In [1]: "x".replace("x","y")
Out[1]: 'y'

Thus the impact might be slightly bigger.

[schnitzelbub]

But on the other hand other str methods with similar interfaces will do fine:

$ ",".join("ab")
'a,b'

$ "a,b,c,d,".rstrip(",")
'a,b,c,d'

Possible naming collision?

[schnitzelbub]

By the way, I can confirm that this affects pip installed versions from PyPi (v0.2.6) as well as the latest from master:

pip3 install --upgrade git+git://github.com/scopatz/xonsh.git@master

[scopatz]

Is there a way pass XONSH_SHOW_TRACEBACK = True to the Shell() or do I need to pass that info somewhere else?

@jsarver, no, not currently. But you could add it right after creating the Shell instance:

from xonsh.shell import Shell
s = Shell()
__xonsh_env__['XONSH_SHOW_TRACEBACK'] = True
s.default("import os")

It didn't seem to work when putting that in my default .xonshrc file.

Odd. That sounds like a bug w.r.t. https://github.com/scopatz/xonsh/blob/master/xonsh/shell.py#L113

[scopatz]

Ok, this is not specific to os.path.join but rather an issue of str.replace or argument passing in general!?

@schnitzelbub That is very interesting.

[scopatz]

Knowing exactly what line is failing would really help. I just checked again, and everything in the argument handling seems correct. Could you both run the xonfig command too and report the output here?

[schnitzelbub]

started from /usr/local/bin/xonsh:

+------------------+-------+
| xonsh            | 0.2.6 |
| Python           | 3.4.2 |
| PLY              | 3.7   |
| have readline    | True  |
| prompt toolkit   | False |
| pygments         | False |
| on posix         | True  |
| on linux         | True  |
| on arch          | False |
| on windows       | False |
| on mac           | False |
| are root user    | False |
| default encoding | utf-8 |
+------------------+-------+

started within a virtualenv:

+------------------+-------+
| xonsh            | 0.2.6 |
| Python           | 3.4.2 |
| PLY              | 3.7   |
| have readline    | True  |
| prompt toolkit   | 0.57  |
| pygments         | 2.1.1 |
| on posix         | True  |
| on linux         | True  |
| on arch          | False |
| on windows       | False |
| on mac           | False |
| are root user    | False |
| default encoding | utf-8 |
+------------------+-------+

The virtualenv one fails, but the system wide actually works!

I removed the differing packages from the virtualenv :

+------------------+-------+
| xonsh            | 0.2.6 |
| Python           | 3.4.2 |
| PLY              | 3.7   |
| have readline    | True  |
| prompt toolkit   | False |
| pygments         | False |
| on posix         | True  |
| on linux         | True  |
| on arch          | False |
| on windows       | False |
| on mac           | False |
| are root user    | False |
| default encoding | utf-8 |
+------------------+-------+

in order to have parity, but unfortunately to no avail.

[scopatz]

What. i. going. on?! This is totally crazy

[scopatz]

I wonder if it is a bug in PLY

[jsarver]

This is the config where i get the error

+------------------+-------+
| xonsh            | 0.2.6 |
| Python           | 3.4.2 |
| PLY              | 3.7   |
| have readline    | False |
| prompt toolkit   | 0.57  |
| pygments         | 2.1.1 |
| on posix         | False |
| on linux         | False |
| on arch          | False |
| on windows       | True  |
| on mac           | False |
| are root user    | True  |
| default encoding | utf-8 |
+------------------+-------+

[scopatz]

@jsarver do you have a full traceback from when it fails?

[gforsyth]

@schnitzelbub -- I don't think this will yield anything but it's worth trying.
Can you post the output of

import sys
sys.version

from within the virtualenv and without?

[jsarver]

It's odd that the master source doesn't work for @schnitzelbub, I just installed to latest and it seems to work for me.

@scopatz to answer your question, here is the traceback for when it does fail in 0.2.6, also the traceback occurs when running os.path.join and "x".replace("x","y") like @schnitzelbub was encountering.

os.path.join("one","two")
Traceback (most recent call last):
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\ptk\shell.py", line 89, in push
    locs=self.ctx)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\execer.py", line 94, in compile
    tree = self.parse(input, ctx, mode=mode)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\execer.py", line 69, in parse
    tree = self._parse_ctx_free(input, mode=mode)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\execer.py", line 150, in _parse_ctx_free
    debug_level=self.debug_level)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\parser.py", line 289, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\ply\yacc.py", line 1106, in parseopt_notrack
    p.callable(pslice)
  File "C:\Miniconda3\envs\xonshtest\lib\site-packages\xonsh\parser.py", line 2238, in p_arglist
    for arg in p1:
TypeError: 'Str' object is not iterable

[schnitzelbub]

@gforsyth here's the output of sys.version:

within virtualenv:

$ import sys
$ sys.version
'3.4.2 (default, Oct  8 2014, 10:45:20) \n[GCC 4.9.1]'

outside virtualenv:

@gforsyth

$ import sys
$ sys.version
'3.4.2 (default, Oct  8 2014, 10:45:20) \n[GCC 4.9.1]'

[schnitzelbub]

Here's my traceback from within a virtualenv:

$ "x".replace("a","b")
Traceback (most recent call last):
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/base_shell.py", line 175, in push
    locs=self.ctx)
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/execer.py", line 94, in compile
    tree = self.parse(input, ctx, mode=mode)
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/execer.py", line 69, in parse
    tree = self._parse_ctx_free(input, mode=mode)
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/execer.py", line 150, in _parse_ctx_free
    debug_level=self.debug_level)
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/parser.py", line 289, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/home/hank/venv1/lib/python3.4/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/hank/venv1/lib/python3.4/site-packages/ply/yacc.py", line 1106, in parseopt_notrack
    p.callable(pslice)
  File "/home/hank/venv1/lib/python3.4/site-packages/xonsh/parser.py", line 2238, in p_arglist
    for arg in p1:
TypeError: 'Str' object is not iterable

it is almost identical to @jsarver 's output, except for ptk\shell.py being base_shell.py in my case.

[scopatz]

Thanks for those tracebacks @jsarver and @schnitzelbub. Neither of those are for the master branch. The parser.py file is now only 8 lines. Can you reproduce this error on master?

[schnitzelbub]

Sorry, my virtualenv had version 0.2.6 from PyPi installed and running xonsh --version was misleading (('xonsh/0.2.6',)).

I ran

pip3 install --upgrade git+git://github.com/scopatz/xonsh.git@master

in the virtualenv and retried again and now it is working here as well:

$ "x".replace("x","y")
'y'

So I can confirm that it is fixed in master. Sorry for the inconvenience!

Thank you so much for your support, can't wait for the next release 👍

[jsarver]

Can you reproduce this error on master?

No it has been working for me on master, I have only been having issue with the 0.2.6 and 0.2.4 releases.

[scopatz]

Excellent! I'd like to see #672 and #695 get merged in for the next release, but after that I would be happy with another release :) Please feel free to re-open if this can be reproduced on master.