None.splitlines raises AttributeError in email.feedparser

[sshnaidm]

BPO	34102
Nosy	@warsaw, @terryjreedy, @bitdancer, @tirkarthi, @sshnaidm
PRs	bpo-34102: Check if data is not None before split #8266

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-13.21:34:39.417>
created_at = <Date 2018-07-12.12:13:49.863>
labels = ['type-feature', 'invalid', 'expert-email']
title = 'None.splitlines raises AttributeError in email.feedparser'
updated_at = <Date 2018-07-13.21:44:17.250>
user = 'https://github.com/sshnaidm'

bugs.python.org fields:

activity = <Date 2018-07-13.21:44:17.250>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-13.21:34:39.417>
closer = 'terry.reedy'
components = ['email']
creation = <Date 2018-07-12.12:13:49.863>
creator = 'sshnaidm'
dependencies = []
files = []
hgrepos = []
issue_num = 34102
keywords = ['patch']
message_count = 7.0
messages = ['321544', '321547', '321553', '321556', '321562', '321635', '321636']
nosy_count = 5.0
nosy_names = ['barry', 'terry.reedy', 'r.david.murray', 'xtreak', 'sshnaidm']
pr_nums = ['8266']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34102'
versions = ['Python 2.7']

[sshnaidm]

sudo pip install ansible-lint
Requirement already satisfied: ansible-lint in /usr/lib/python2.7/site-packages (3.4.21)
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/usr/lib/python2.7/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/lib/python2.7/site-packages/pip/_internal/resolve.py", line 262, in _resolve_one
    check_dist_requires_python(dist)
  File "/usr/lib/python2.7/site-packages/pip/_internal/utils/packaging.py", line 46, in check_dist_requires_python
    feed_parser.feed(metadata)
  File "/usr/lib64/python2.7/email/feedparser.py", line 178, in feed
    self._input.push(data)
  File "/usr/lib64/python2.7/email/feedparser.py", line 99, in push
    print(parts)
UnboundLocalError: local variable 'parts' referenced before assignment

pip2 --version
pip 10.0.1 from /usr/lib/python2.7/site-packages/pip (python 2.7)

[tirkarthi]

Hi,

I couldn't see the print statement in Lib/email/feedparser.py with latest 2.7 branch and also git log -p [Lib/email/feedparser.py](https://github.com/python/cpython/blob/main/Lib/email/feedparser.py) | grep print doesn't return me anything. pip install ansible-lint also works fine with Python 2.7. I think this statement was added by mistake for debugging by the developer before parts = data.splitlines(True). Can you please point me to the commit where the print statement was added or a test case to reproduce this?

Thanks

[sshnaidm]

@XTreak , sorry, you're right, I pasted a wrong traceback.
This is original traceback:

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 346, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 666, in _prepare_file
    check_dist_requires_python(dist)
  File "/usr/lib/python2.7/site-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
    feed_parser.feed(metadata)
  File "/usr/lib64/python2.7/email/feedparser.py", line 177, in feed
    self._input.push(data)
  File "/usr/lib64/python2.7/email/feedparser.py", line 99, in push
    parts = data.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'

And I pasted by mistake the one I used to debug.

[tirkarthi]

Thanks @sshnaidm. The patch makes sense. Can you add a test case for this that fails without the patch? I have tried pip install ansible-lint and it works fine for me in Python 2.7.

I think a NEWS entry will help too : https://devguide.python.org/committing/#what-s-new-and-news-entries

Thanks

[bitdancer]

This appears to be a bug in pip or the ansible-lint package, similar, and possibly identical, to bpo-31361. I believe None is being passed in as the argument to feed_parser.feed, which is an invalid use of that API and so correctly results in an error.

[terryjreedy]

An exception is not a crash. An intended exception is not even a bug. A bug is a discrepancy between behavior and doc. In this case, email.parser.Feedparser is imported from email.feedparser. Its doc
https://docs.python.org/2.7/library/email.parser.html#feedparser-api
says
"feed(data)
Feed the FeedParser some more data. data should be a string containing one or more lines."
In addition, html.parser.HTMLParser.feed(data) say "data must be str."

For either, behavior upon passing anything else is undefined. Defining the behavior for None, and defining it to be 'return something' rather than 'raise something', would be an enhancement. Such enhancements are only added, if deemed desirable, in future versions.

(Note: I recommend against resubmitting this for 3.8. Masking bugs in 3rd party code is contrary to policy. There would have to be a positive use-case of benefit large enough to multiple users to overcome the masking deficit.)

[terryjreedy]

What you can do is report to pip and ansible-lint that there is a bug somewhere in one of the two packages, or try to figure out which and report it to one of them.