Add support for Python 3.8

[PCManticore]

We do have a dozen of tests currently failing on Python 3.8. We should add support for it before the official launch in autumn.

• Positional only arguments

[encukou]

Maintainers of Python projects are strongly encouraged to test with Python's beta and RC releases. Many projects use astroid/pylint as part of their CI; they can't fully start testing without 3.8-compatible linters.
Please consider releasing well before Python's official launch in autumn, to unblock the ecosystem and to have more time to find any remaining 3.8 bugs in both Python and astroid.

[tacaswell]

It looks like the actual parsing of the python code is delegated to typed_ast which has an open issue for supporting the new syntax python/typed_ast#118 .

[MatthijsBlom]

Here is another 3.8-related failure mode: astroid reports f"{3=}" as a syntax error.
(Feature announced here.)

It looks like the actual parsing of the python code is delegated to typed_ast which has an open issue for supporting the new syntax python/typed_ast#118 .

The README says that it will not support Python 3.8+. Instead (I think), typed_ast has been merged back into ast. (bpo-35766)

[MatthijsBlom]

Seeing as typed_ast was recently merged back into ast (which is supposedly more up to date), I thought replacing imports of typed_ast with imports of ast might solve some problems.

I tried just replacing the one typed_ast.ast3 in _ast.py with simply ast. Below are test results.

Before:

> py -3.7 -m pytest astroid
= 842 passed, 43 skipped, 15 xfailed, 1 xpassed, 2 warnings in 26.57 seconds =

> py -3.8 -m pytest astroid
= 47 failed, 797 passed, 41 skipped, 15 xfailed, 1 xpassed, 2 warnings in 34.10 seconds =

After:

> py -3.8 -m pytest astroid
= 5 failed, 839 passed, 41 skipped, 15 xfailed, 1 xpassed, 2 warnings in 27.46 seconds =

While I lack any further knowledge of astroid's inner workings, this certainly looks promising to me :-)

It seems that this change fixes both positional-only arguments and debug f-strings.

[PCManticore]

@MatthijsBlom Thanks for the test! We only use typed_ast for Python 3.5-3.7, not for 3.8. The errors you are seeing with py -3.8 might just be the unsupported features of Python 3.8 that astroid currently don't handle. I'm not sure though how you got these errors, as the CI is green on 3.8. Maybe you are using a particular Python 3.8 version that is not available on Travis?

[MatthijsBlom]

@PCManticore I use a fresh Python 3.8.0b2 on Windows. I ran the tests again, against commit 34bd444. Here are the complete logs: before.txt, after.txt.
(I notice the number of failures has increased in both cases.)

[PCManticore]

@MatthijsBlom FWIW, I cannot reproduce these errors using the latest 3.8 beta on Mac. Not sure if these test failures are specific to Windows as I don't have access to this OS, and AppVeyor doesn't have support for Python 3.8 just yet. Is it possible to see if the latest master gives you the same amount of errors without any changes to typed_ast?

[MatthijsBlom]

@PCManticore Did you mean without changes to _ast.py? I tested a fresh 516197d; results here:
windows.txt. This time, there is only one failure. I tested on Ubuntu on WSL as well; those tests were clean.

[PCManticore]

@MatthijsBlom Yes that's right. Thanks for doing that test! 28fc86f should fix that test failure for you.

[MatthijsBlom]

@PCManticore It does. No failures remaining now on my end.

[PCManticore]

Thank you for testing it @MatthijsBlom I think we can close this now. My plan is to release a new version in the next two weeks, but I'll have a dev release before then.