Importing bs4 fails with -3 option in Python 2.7.15

[fschulze]

BPO	33645
Nosy	@serhiy-storchaka, @fschulze
PRs	[2.7] bpo-33645: Fix an "unknown parsing error" in the parser. #7119

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-31.04:42:58.421>
created_at = <Date 2018-05-25.09:17:41.454>
labels = ['interpreter-core', 'type-bug']
title = 'Importing bs4 fails with -3 option in Python 2.7.15'
updated_at = <Date 2018-05-31.04:42:58.420>
user = 'https://github.com/fschulze'

bugs.python.org fields:

activity = <Date 2018-05-31.04:42:58.420>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-31.04:42:58.421>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2018-05-25.09:17:41.454>
creator = 'fschulze'
dependencies = []
files = []
hgrepos = []
issue_num = 33645
keywords = ['patch']
message_count = 7.0
messages = ['317664', '317675', '317678', '317681', '317697', '318245', '318248']
nosy_count = 2.0
nosy_names = ['serhiy.storchaka', 'fschulze']
pr_nums = ['7119']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33645'
versions = ['Python 2.7']

[fschulze]

Since Python 2.7.15 import bs4 (BeautifulSoup4) fails when using the -3 option on the Python binary.

...
'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'<>'You need to convert the code, either by installing it (python setup.py install) or by running 2to3 (2to3 -w bs4).'
^
SyntaxError: unknown parsing error

With 2.7.14 this works fine and I get the expected deprecation warning.

A workaround is to import the package without -3 first, so the .pyc file is generated and used in subsequent imports.

[serhiy-storchaka]

The Python interpreter itself does not have references to Beautiful Soup. This error is generated by third-party code. Report a bug on the Beautiful Soup bug tracker.

[fschulze]

Yes, it's third party code, but that worked up until Python 2.7.14 and only broke with 2.7.15, which is why I reported it here.

The expected behaviour is not a SyntaxError, but a DeprecationWarning, which is what you get pre Python 2.7.15. So this is a regression.

[serhiy-storchaka]

Ah, I was fooled by the message about Beautiful Soup. It is not a message of an error, it is a part of source line, printed for a SyntaxError. Seems sources of Beautiful Soup intentionally contain a code invalid in Python 3.

There is a bug in Python. The tokenizer returns error E_OK, but it should never return such error code. The AST parser is confused and raises a SyntaxError with the message "unknown parsing error".

This bug is reproduced when run Python wish both options -3 and -We and parse the "<>" operator.

$ ./python -3 -We -c '[] <> []'
error=10
  File "<string>", line 1
    [] <> []
        ^
SyntaxError: unknown parsing error

But it is reproduced with 2.7.14 too.

[fschulze]

You are right, this actually happens with Python 2.7.14 as well. I was fooled by a warnings.filterwarnings matching to a path in my codebase. That one did match for my Python 2.7.15 testing but didn't for my Python 2.7.14 testing, because those were done in a temporary path which didn't match. I should have tried to narrow it down further first.

I'm glad you found the necessary -We option.

Thanks for looking into this and for the patch!

I'm currently unable to test the patch though.

[serhiy-storchaka]

New changeset d5e7556 by Serhiy Storchaka in branch '2.7':
bpo-33645: Fix an "unknown parsing error" in the parser. (GH-7119)
d5e7556

[serhiy-storchaka]

I'm not sure that this will fix your case (there may be other bug in BeautifulSoup4 or other changes in your environment), but it fixed a bug in Python. Thank you for your report.