Create python3.8 parser

[gforsyth]

This builds on earlier work by @carmenbianca -- I've moved the 3.8+
specific parser code (involving the posonlyargs) to a python 3.8
parser class that inherits from the 3.6 parser.

[scopatz]

Should be for 3.8 😉

[scopatz]

This is awesome! Thanks @gforsyth! Do we need any more tests here?

Also, should probably enable CI for 3.8 before merging. I belive that conda-forge has packages for everything we need for xonsh now for v3.8

[gforsyth]

Hrrrr,

so yeah, we need more tests. Also, on 3.8 current parser tests are failing. I believe this is due to a change in the AST in python 3.8 -- there are now two additional attributes for ast.stmt and ast.expr: end_lineno and end_col_offset.

little bit of info here: https://docs.python.org/3.8/library/ast.html#ast.AST.lineno

Also looks like ast.Num, ast.Str, ast.Bytes, ast.NameConstant, and ast.Ellipsis are being deprecated in favor of ast.Constant

6 tests are failing because AssertionError: Ast nodes do not have the same line number : 2 != 1

[gforsyth]

Hmm, interesting. All of the failures involve decorators:

============================================================= FAILURES =============================================================
__________________________________________________________ test_decorator __________________________________________________________

    def test_decorator():
>       check_stmts("@g\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1931: AssertionError
_________________________________________________________ test_decorator_2 _________________________________________________________

    def test_decorator_2():
>       check_stmts("@h\n@g\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 3 != 1

tests/test_parser.py:1935: AssertionError
_______________________________________________________ test_decorator_call ________________________________________________________

    def test_decorator_call():
>       check_stmts("@g()\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1939: AssertionError
_____________________________________________________ test_decorator_call_args _____________________________________________________

    def test_decorator_call_args():
>       check_stmts("@g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1943: AssertionError
___________________________________________________ test_decorator_dot_call_args ___________________________________________________

    def test_decorator_dot_call_args():
>       check_stmts("@h.g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1947: AssertionError
_________________________________________________ test_decorator_dot_dot_call_args _________________________________________________

    def test_decorator_dot_dot_call_args():
>       check_stmts("@i.h.g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1951: AssertionError

[gforsyth]

Ok, this is ready for another look.

I think that at this stage it actually doesn't need more tests, in that all this PR is doing is making sure that xonsh can run on Python 3.8. Python 3.8 is now enabled in CI and tests are passing, so huzzah! And @scopatz, it looks like 3.8 changed a bit of the AST involving decorator position that you found a bit odd, as per this comment: https://github.com/xonsh/xonsh/blob/master/xonsh/parsers/base.py#L823-L826

There is a much bigger lift which is adding support for positional-only parameter support which landed in 3.8: https://www.python.org/dev/peps/pep-0570/#syntax-and-semantics

This is going to require a bit of extra noodling in the parser to handle all the new possible function signatures (I'm playing with this but someone else should feel free to take a crack at it, too).

[gforsyth]

Ping @xonsh/xore

[gforsyth]

Anyone in @xonsh/xore available to take a look at this?

[melund]

Thanks for all the effort @gforsyth . I have read through the code and tried it with python3.8. I can merge this.

I still think we have problem with releasing, since #3342 may have broken other things. @scopatz is probably the one to judge this.