site.abs_paths should handle None __cached__ type

[demianbrecht]

BPO	33399
Nosy	@brettcannon, @ncoghlan, @ericsnowcurrently, @demianbrecht
PRs	bpo-33399: Handle when module __cache__ is None in site.abs_path() #6675

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-01.17:56:11.751>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'site.abs_paths should handle None __cached__ type'
updated_at = <Date 2018-05-15.18:41:25.876>
user = 'https://github.com/demianbrecht'

bugs.python.org fields:

activity = <Date 2018-05-15.18:41:25.876>
actor = 'brett.cannon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-05-01.17:56:11.751>
creator = 'demian.brecht'
dependencies = []
files = []
hgrepos = []
issue_num = 33399
keywords = ['patch']
message_count = 2.0
messages = ['316008', '316692']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'ncoghlan', 'eric.snow', 'demian.brecht']
pr_nums = ['6675']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33399'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[demianbrecht]

Echoing an email sent to python-list:

I recently ran into an issue using the site package and I wanted to confirm that I'm not doing something wrong here before creating an issue:

I have a need to git subtree modules into a Django project and then add the dependencies such that they're discoverable in code. I also need to have consistent behavior between python setup.py (develop|install) and pip install -r requirements.txt (the latter is used by Heroku for project installation). An example project that demonstrates the issue is here: https://github.com/demianbrecht/python-sitehooks-example.

In the example, I've subtree'd requests into _vendor and forced site hooks to fire using site.main() in testme/settings.py: demianbrecht/python-sitehooks-example@1b81e15. Forcing the execution is not needed when the package is actually installed using setup.py, but is needed when using pip install -r requirements.txt. This works great when running within a virtualenv, but I get the following on system python:

$ python3.6 ./manage.py test
Traceback (most recent call last):
 File "./manage.py", line 22, in <module>
   execute_from_command_line(sys.argv)
 File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
   utility.execute()
 File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 308, in execute
   settings.INSTALLED_APPS
 File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
   self._setup(name)
 File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
   self._wrapped = Settings(settings_module)
 File "/usr/local/lib/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
   mod = importlib.import_module(self.SETTINGS_MODULE)
 File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
   return _bootstrap._gcd_import(name[level:], package, level)
 File "<frozen importlib._bootstrap>", line 978, in _gcd_import
 File "<frozen importlib._bootstrap>", line 961, in _find_and_load
 File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 678, in exec_module
 File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
 File "/<path>/<to>/testme/testme/settings.py", line 13, in <module>
   site.main()
 File "/usr/local/lib/python3.6/site.py", line 525, in main
   abs_paths()
 File "/usr/local/lib/python3.6/site.py", line 110, in abs_paths
   m.__cached__ = os.path.abspath(m.__cached__)
 File "/usr/local/lib/python3.6/posixpath.py", line 369, in abspath
   path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Using venv and system comparisons for debugging doesn't help much as site.py is different in the two environments. Any insight into what may be going on here or what I'm doing wrong would be much appreciated. I realize that I could just muck with PYTHONPATH, but I thought this approach would be a little nicer as it's transparent to the user.

Digging a little deeper, it seems that site.abs_paths fails when a module's __cached__ value is None. I haven't spent enough time digging into it to understand why this is valid behavior, hoping to get some insight from folks who have spent more time on it.

[brettcannon]

I didn't bother digging as I assume this was done for a reason. I left a request for changes on the PR to make the code has a stricter try block.

[furkanonder]

@brettcannon The problem appears to be resolved. I think the issue can be closed.