New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove a same-variable-name class hierarchy #1092

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@pradyunsg
Member

pradyunsg commented Jul 18, 2017

pypa/pip#4545 is an attempt to add mypy type-checking to pip. In that PR, the vendored version of pkg_resources is patched (with this same change) since it is needed for mypy to run on the codebase.

This patch is really a workaround for mypy's crashing behaviour; as can be seen in https://travis-ci.org/pypa/pip/jobs/243330633#L120.

It would be nice if pip's vendoring task wouldn't have to do such a thing, given the triviality of this change.

@jaraco Thoughts?

Remove a same-variable-name class hierarchy
The modified definition makes it possible to use `pkg_resources` with `mypy`, since `mypy` currently fails since it detects a cycle in the class's hierarchy.

Additionally, this also makes the intent slightly clearer IMO.
@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

My initial instinct is that Setuptools shouldn't be adopting a workaround for a defect in another system without a clear acknowledgement that it's a workaround. I don't yet understand how the current implementation is deficient or how changing it fixes things. Can you provide a distilled, minimal example of why the current behavior is problemmatic? I tried using _get_mro with mypy installed and I had no problems:

$ rwt mypy
Collecting mypy
  Downloading mypy-0.521-py3-none-any.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 1.2MB/s 
Collecting typed-ast<1.1.0,>=1.0.4 (from mypy)
  Downloading typed_ast-1.0.4-cp36-cp36m-macosx_10_11_x86_64.whl (211kB)
    100% |████████████████████████████████| 215kB 3.7MB/s 
Installing collected packages: typed-ast, mypy
Successfully installed mypy-0.521 typed-ast-1.0.4
Python 3.6.1 (v3.6.1:69c0db5050, Mar 21 2017, 01:21:04) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class MyClass:
...   pass
... 
>>> import pkg_resources
>>> pkg_resources._get_mro(MyClass)
(<class '__main__.MyClass'>, <class 'object'>)

But I can't tell from the mypy docs how to activate it or use it.

I do think that the change you're proposing here looks sound. I'd just like to have an understanding why it's superior before accepting it.

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Jul 26, 2017

I'd just like to have an understanding why it's superior before accepting it.

Running mypy with pkg_resources vendored causes a mypy to abort (see here - https://travis-ci.org/pypa/pip/jobs/243330633#L120).

mypy thinks there's a cycle in the class hierarchy, at the class definition line (that has been changed in this PR) since the new class refers to the old class and has the same name. By changing the name of the new class, this issue is worked around.

I felt that this change was too trivial to deserve a comment in the sources.

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Jul 26, 2017

Can you provide a distilled, minimal example of why the current behavior is problemmatic?

If you still want one after the previous post, I'll happily look into make a reproducible example. :)

I tried using _get_mro with mypy installed and I had no problems:

That's not how to trigger this. mypy refusing to check types on anything that refers to pkg_resources through an import - because it detects a cycle in the class hierarchy.

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

Here's running mypy on a file importing pkg_resources:

$ cat > test-issue-1092.py
import pkg_resources
$ rwt mypy -- -m mypy test-issue-1092.py                      
Collecting mypy
  Using cached mypy-0.521-py3-none-any.whl
Collecting typed-ast<1.1.0,>=1.0.4 (from mypy)
  Using cached typed_ast-1.0.4-cp36-cp36m-macosx_10_11_x86_64.whl
Installing collected packages: typed-ast, mypy
Successfully installed mypy-0.521 typed-ast-1.0.4
test-issue-1092.py: error: Name '__builtins__' is not defined
test-issue-1092.py:1: error: No library stub file for module 'pkg_resources'
test-issue-1092.py:1: note: (Stub files are from https://github.com/python/typeshed)
test-issue-1092.py:1: error: Cannot find module named 'builtins'
test-issue-1092.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
$ echo $?
1

I do see an error when running against pkg_resources package itself.

$ rwt mypy -- -m mypy --show-traceback ~/m/setuptools/pkg_resources/__init__.py
Collecting mypy
  Using cached mypy-0.521-py3-none-any.whl
Collecting typed-ast<1.1.0,>=1.0.4 (from mypy)
  Using cached typed_ast-1.0.4-cp36-cp36m-macosx_10_11_x86_64.whl
Installing collected packages: typed-ast, mypy
Successfully installed mypy-0.521 typed-ast-1.0.4
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:1: error: Cannot find module named 'builtins'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:18: error: Cannot find module named '__future__'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:18: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:20: error: Cannot find module named 'sys'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:21: error: No library stub file for standard library module 'os'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:21: note: (Stub files are from https://github.com/python/typeshed)
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:22: error: No library stub file for standard library module 'io'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:23: error: Cannot find module named 'time'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:24: error: No library stub file for standard library module 're'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:25: error: No library stub file for standard library module 'types'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:26: error: No library stub file for standard library module 'zipfile'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:27: error: Cannot find module named 'zipimport'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:28: error: No library stub file for standard library module 'warnings'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:29: error: No library stub file for standard library module 'stat'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:30: error: No library stub file for standard library module 'functools'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:31: error: No library stub file for standard library module 'pkgutil'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:32: error: No library stub file for standard library module 'operator'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:33: error: No library stub file for standard library module 'platform'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:34: error: No library stub file for standard library module 'collections'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:35: error: No library stub file for standard library module 'plistlib'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:36: error: No library stub file for standard library module 'email'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:36: error: No library stub file for standard library module 'email.parser'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:37: error: No library stub file for standard library module 'tempfile'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:38: error: No library stub file for standard library module 'textwrap'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:39: error: Cannot find module named 'itertools'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:43: error: Cannot find module named '_imp'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:46: error: No library stub file for standard library module 'imp'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:49: error: No library stub file for module 'pkg_resources.extern.six.moves'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:61: error: No library stub file for standard library module 'os.path'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:64: error: No library stub file for standard library module 'importlib'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:64: error: No library stub file for standard library module 'importlib.machinery'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:467: error: No library stub file for standard library module 'sysconfig'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:469: error: Cannot find module named 'distutils.util'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:650: error: Cannot find module named '__main__'
/Users/jaraco/m/setuptools/pkg_resources/__init__.py:1504: error: No library stub file for standard library module 'linecache'
/Users/jaraco/m/setuptools/pkg_resources/extern/__init__.py:1: error: Cannot find module named 'builtins'
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py: error: Name '__builtins__' is not defined
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py:1: error: Cannot find module named 'builtins'
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py:2: error: Cannot find module named 'errno'
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py:6: error: Name 'False' is not defined
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py:6: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.521
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/main.py", line 97, in type_check_only
    options=options)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/build.py", line 2044, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/build.py", line 2147, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/build.py", line 1716, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 185, in check_first_pass
    self.accept(d)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 273, in accept
    stmt.accept(self)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/nodes.py", line 565, in accept
    return visitor.visit_func_def(self)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 519, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 574, in check_func_item
    typ = self.function_type(defn)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 2554, in function_type
    return function_type(func, self.named_type('builtins.function'))
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 2423, in named_type
    sym = self.lookup_qualified(name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-w99m5jw9/mypy/checker.py", line 2488, in lookup_qualified
    n = self.modules[parts[0]]
KeyError: 'builtins'
/Users/jaraco/m/setuptools/pkg_resources/py31compat.py:6: note: use --pdb to drop into pdb
@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

Applying the patch, however, I get the same result.

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

Perhaps that _get_mro issue only occurs when an old-style class is present, which would only happen on Python 2.

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

As I look at it, I see there's another _get_mro function in setuptools.monkey, which does mostly the same thing, but addresses different issues. Probably these functions need to be consolidated.

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Jul 26, 2017

Perhaps that _get_mro issue only occurs when an old-style class is present, which would only happen on Python 2.

mypy only runs on Py3.

jaraco added a commit that referenced this pull request Jul 26, 2017

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 26, 2017

I've pushed an alternate approach in 35fa31f.

@jaraco jaraco closed this Jul 26, 2017

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Jul 26, 2017

It's way past midnight - I'll let you know if this works for the motivating PR tomorrow evening. :)

@pradyunsg pradyunsg deleted the pradyunsg:patch-1 branch Jul 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment