importlib has become part of the "builtins" cycle

[gvanrossum]

The recent changes to typeshed that added importlib have unfortunately extended the import cycle containing the builtins module from 5 to 7 modules, and somehow this has broken incremental mode.

Calling out @brettcannon (importlib author) and @Michael0x2a (who's going to work on incremental soon).

FYI the existing cycle contains these modules: abc typing types sys builtins. Since every module depends on builtins, every module depends on all of these. The importlib changes added importlib._modulespec importlib.abc to the cycle. I don't yet understand how this broke incremental mode. but when I run my -i -c 'import typing' with a warm cache I get this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/guido/src/mypy/mypy/__main__.py", line 5, in <module>
    main(None)
  File "/Users/guido/src/mypy/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/guido/src/mypy/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/Users/guido/src/mypy/mypy/build.py", line 177, in build
    dispatch(sources, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1315, in dispatch
    process_graph(graph, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1451, in process_graph
    process_fresh_scc(graph, scc)
  File "/Users/guido/src/mypy/mypy/build.py", line 1510, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "/Users/guido/src/mypy/mypy/build.py", line 1178, in fix_cross_refs
    fixup_module_pass_one(self.tree, self.manager.modules)
  File "/Users/guido/src/mypy/mypy/fixup.py", line 17, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "/Users/guido/src/mypy/mypy/fixup.py", line 75, in visit_symbol_table
    stnode = lookup_qualified_stnode(self.modules, cross_ref)
  File "/Users/guido/src/mypy/mypy/fixup.py", line 237, in lookup_qualified_stnode
    assert '.' in head, "Cannot find %s" % (name,)
AssertionError: Cannot find importlib

(Michael: I can help debugging this.)

[brettcannon]

So the dependency issue here is that types.ModuleType depends on importlib.abc.Loader and importlib.machinery.ModuleSpec (the latter which is kept in importlib._modulespec). But then importlib.abc.Loader depends on importlib.machinery.ModuleSpec and vice-versa. Since Loader and ModuleSpec are the leaves of this dependency you could move importlib.abc.Loader into importlib._modulespec and then only import that in types (if I remember correctly PEP 484 specifies a way to list a type that isn't declared yet?).

[gvanrossum]

OK, I think I have a fix. It moves the cycle ModuleSpec/ModuleType/Loader to a top-level private module (_importlib_modulespec.pyi), adds from ... import X as X statements to types.pyi and importlib/{abc,machinery}.pyi, and cleans up minor details. This fixes the mypy -i issue (though I'm not sure I understand how or why) -- the import cycle is now 6 large (the original 5 + _importlib_modulespec).