Xonsh broken on Python 3.8

[carmenbianca]

xonfig

Irrelevant because it doesn't run.

Expected Behavior

Xonsh works correctly when you run it using a Python 3.8 interpreter.

Current Behavior

Xonsh crashes upon starting.

Traceback (if applicable)

$ python3 scripts/xonsh                                                                                                                                              
xonsh: To log full traceback to a file set: $XONSH_TRACEBACK_LOGFILE = <filename>
Traceback (most recent call last):
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/proc.py", line 1709, in wait
    r = self.f(self.args, stdin, stdout, stderr, spec, spec.stack)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/proc.py", line 1204, in proxy_two
    return f(args, stdin)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/xontribs.py", line 178, in xontribs_main
    return _MAIN_XONTRIB_ACTIONS[ns.action](ns)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/xontribs.py", line 98, in _load
    xontribs_load(ns.names, verbose=ns.verbose)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/xontribs.py", line 90, in xontribs_load
    update_context(name, ctx=ctx)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/xontribs.py", line 69, in update_context
    modctx = xontrib_context(name)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/xontribs.py", line 33, in xontrib_context
    m = importlib.import_module(spec.name)
  File "/home/carmenbianca/.pyenv/versions/3.8-dev/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 773, in exec_module
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/imphooks.py", line 127, in get_code
    code = execer.compile(src, glbs=ctx, locs=ctx)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/execer.py", line 138, in compile
    code = compile(tree, filename, mode)
TypeError: required field "type_ignores" missing from Module
Traceback (most recent call last):
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/main.py", line 402, in main
    return main_xonsh(args)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/main.py", line 431, in main_xonsh
    shell.shell.cmdloop()
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/ptk2/shell.py", line 189, in cmdloop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/home/carmenbianca/Projektoj/xonsh/xonsh/ptk2/shell.py", line 158, in singleline
    line = self.prompter.prompt(**prompt_args)
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 738, in prompt
    return run_sync()
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 727, in run_sync
    return self.app.run(inputhook=self.inputhook, pre_run=pre_run2)
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 709, in run
    return run()
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 683, in run
    return f.result()
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/eventloop/future.py", line 149, in result
    raise self._exception
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/eventloop/coroutine.py", line 90, in step_next
    new_f = coroutine.throw(exc)
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 658, in _run_async2
    result = yield f
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/eventloop/coroutine.py", line 86, in step_next
    new_f = coroutine.send(None)
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 597, in _run_async
    with self.input.raw_mode():
  File "/home/carmenbianca/.virtualenvs/xonsh38/lib/python3.8/site-packages/prompt_toolkit/input/vt100.py", line 228, in __enter__
    termios.tcsetattr(self.fileno, termios.TCSANOW, newattr)
termios.error: (4, 'Interrupted system call')
Xonsh encountered an issue during launch
Failback to /bin/sh
sh: ne prosperis atribui grupon (-1) de terminala procezo: Nevalida 'ioctl' por aparato
sh: Ĉi tiu ŝelo ne disponigas laborregadon
sh-5.0$ exit

The relevant lines are:

  File "/home/carmenbianca/Projektoj/xonsh/xonsh/execer.py", line 138, in compile
    code = compile(tree, filename, mode)
TypeError: required field "type_ignores" missing from Module

Steps to Reproduce

Obtain a version of Python 3.8, and run xonsh using that interpreter.

More info

Bug report in Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1703776

This issue is basically identical to the one found here: ipython/ipython#11590

Which links to an upstream bug report here: https://bugs.python.org/issue35894

Which is closed as notabug, because the ast library isn't stable.

I narrowed down the problem to one specific line, xonsh/parsers/base.py::714:

    def p_file_input(self, p):
        """file_input : file_stmts"""
        p[0] = ast.Module(body=p[1])

If you change that line to:

    def p_file_input(self, p):
        """file_input : file_stmts"""
        p[0] = ast.Module(body=p[1], type_ignores=[])

The interpreter no longer crashes when you launch it.

I have no confidence that this is the correct fix, however. When I run the tests after making that change, I get the following errors:

================================================================================================================== FAILURES ==================================================================================================================
_______________________________________________________________________________________________________________ test_decorator _______________________________________________________________________________________________________________

    def test_decorator():
>       check_stmts("@g\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1949: AssertionError
______________________________________________________________________________________________________________ test_decorator_2 ______________________________________________________________________________________________________________

    def test_decorator_2():
>       check_stmts("@h\n@g\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 3 != 1

tests/test_parser.py:1953: AssertionError
____________________________________________________________________________________________________________ test_decorator_call _____________________________________________________________________________________________________________

    def test_decorator_call():
>       check_stmts("@g()\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1957: AssertionError
__________________________________________________________________________________________________________ test_decorator_call_args __________________________________________________________________________________________________________

    def test_decorator_call_args():
>       check_stmts("@g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1961: AssertionError
________________________________________________________________________________________________________ test_decorator_dot_call_args ________________________________________________________________________________________________________

    def test_decorator_dot_call_args():
>       check_stmts("@h.g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1965: AssertionError
______________________________________________________________________________________________________ test_decorator_dot_dot_call_args ______________________________________________________________________________________________________

    def test_decorator_dot_dot_call_args():
>       check_stmts("@i.h.g(x, y=10)\ndef f():\n  pass", False)
E       AssertionError: Ast nodes do not have the same line number : 2 != 1

tests/test_parser.py:1969: AssertionError

Perhaps someone more familiar with the codebase knows more, because this is where my knowledge ends.

[gaborbernat]

Yeah just noticed this in virtualenv CI -
TypeError: required field "type_ignores" missing from Module

[gforsyth]

xonsh should start up in 3.8 now that #3116 has been merged, but we haven't implemented the rest of the 3.8 changes syntax-wise

[anki-code]

This looks solved now and should be closed.