Add a new, lower priority for imports inside "if MYPY" #2167

Merged
merged 7 commits into from Oct 27, 2016

Projects

None yet

4 participants

@gvanrossum
Member

@elazarg Do you have time to review this? I am missing tests -- do you understand how to add tests for this?

-# Inferred value of an expression.
-ALWAYS_TRUE = 0
-ALWAYS_FALSE = 1
-TRUTH_VALUE_UNKNOWN = 2
@elazarg
elazarg Sep 22, 2016 edited Contributor

Minor comments: uniform prefix will make it slightly easier to search and possibly replace, as will happen eventually when it will become an Enum. Not that TRUTH_VALUE_UNKNOWN is a hard name to search :)

How about giving this type a dedicated alias until then? making it under the namespace class Truth: ... already, before its values gets spread any further.

@gvanrossum
gvanrossum Sep 23, 2016 Member

Yeah, but that would just cause more code churn and the benefit would be minuscule.

+ TRUTH_VALUE_UNKNOWN: TRUTH_VALUE_UNKNOWN,
+ MYPY_TRUE: MYPY_FALSE,
+ MYPY_FALSE: MYPY_TRUE,
+}
@elazarg
elazarg Sep 22, 2016 edited Contributor

Do you consider this numbering more readable than the range [-2, 2], with the obvious negation? An explicit mapping can still be used just to be sure. TRUTH_UNKNOWN should be zero this way or the other,

@gvanrossum
gvanrossum Sep 23, 2016 Member

The numbers are entirely uninteresting. I intentionally changed them around a bit to show that this is so.

mypy/semanal.py
# The condition is always false, so we skip the if/elif body.
mark_block_unreachable(s.body[i])
- elif result == ALWAYS_TRUE:
+ elif result in (ALWAYS_TRUE, MYPY_TRUE):
# This condition is always true, so all of the remaining
# elif/else bodies will never be executed.
@elazarg
elazarg Sep 22, 2016 edited Contributor

The phrasing here is now slightly inaccurate, and perhaps the previous comment too.

@gvanrossum
gvanrossum Sep 23, 2016 Member

Depending on the meaning of "always". :-)

@@ -3024,12 +3039,9 @@ def infer_if_condition_value(expr: Node, pyversion: Tuple[int, int], platform: s
elif name == 'PY3':
result = ALWAYS_TRUE if pyversion[0] == 3 else ALWAYS_FALSE
elif name == 'MYPY' or name == 'TYPE_CHECKING':
- result = ALWAYS_TRUE
+ result = MYPY_TRUE
@elazarg
elazarg Sep 22, 2016 Contributor

I prefer TYPECHECK_TRUE or TYPE_CHECK_TRUE to be intuitively linked to the TYPE_CHECK flag, especially if you intend to phase MYPY flag out. (Similarly MarkImportsTypechekOnlyVisitor, mark_block_typecheck_only, Import.is_typecheck_only, and related documentation.)

@gvanrossum
gvanrossum Sep 23, 2016 Member

Well, we don't really care about other type checkers in this codebase, do we? :-)

@elazarg
elazarg Sep 25, 2016 Contributor

No. My concern was greppability, but it is a minor one (it's only the connection between MYPY_TRUE and TYPE_CHECKING that is not immediate; each is perfectly greppable on its own).

+ def visit_import_all(self, node: ImportAll) -> None:
+ node.is_mypy_only = True
+
+
def is_identity_signature(sig: Type) -> bool:
"""Is type a callable of form T -> T (where T is a type variable)?"""
if isinstance(sig, CallableType) and sig.arg_kinds == [ARG_POS]:
@elazarg
elazarg Sep 22, 2016 Contributor

(Completely off topic but such comparisons assumes that arg_kinds is a list, in a way that mypy does not warn about. Feels like a subtle bug waiting to happen).

@gvanrossum
gvanrossum Sep 23, 2016 Member

Yeah, we have a bug open about that already.

mypy/build.py
@@ -308,9 +308,22 @@ def default_lib_path(data_dir: str, pyversion: Tuple[int, int]) -> List[str]:
PRI_MED = 10 # top-level "import X"
PRI_LOW = 20 # either form inside a function
PRI_INDIRECT = 30 # an indirect dependency
+PRI_MYPY = 40 # inside "if MYPY" or "if typing.TYPE_CHECKING"
PRI_ALL = 99 # include all priorities
@elazarg
elazarg Sep 22, 2016 edited Contributor

Why only a single additional priority? Looks like the concerns are pretty independent. I would think a more detailed set of priorities, e.g. lexicographical ordering of the form

(is_mypy, is_toplevel, is_from)

might help the algorithm breaking cycles less arbitrarily, Alternatively,

(is_mypy, nesting_level, is_from)

perhaps even more so (the actual tuple is of course unimportant; I only talk about the implied ordering).

I don't know if all this will have any effect on any real codebase, but the current system feels somewhat arbitrary.

@gvanrossum
gvanrossum Sep 23, 2016 Member

It would still be arbitrary. There are ideas for something better in the tracker, but I'd like to see how things work out with this small improvement first.

@@ -394,20 +407,21 @@ def correct_rel_imp(imp: Union[ImportFrom, ImportAll]) -> str:
for imp in file.imports:
if not imp.is_unreachable:
if isinstance(imp, Import):
- pri = PRI_MED if imp.is_top_level else PRI_LOW
+ pri = import_priority(imp, PRI_MED)
+ ancestor_pri = import_priority(imp, PRI_LOW)
@elazarg
elazarg Sep 22, 2016 edited Contributor

I don't understand the story of ancestor_pri. I'm sure I can figure it out but it feels like it deserves a comment - how ancestor_pri is pri with a different default? It's not obvious to me.

@gvanrossum
gvanrossum Sep 23, 2016 Member

It's only used once, for the ancestors. The reason is that if you have "import x.y" you depend on both x.y and on x, but the priority for x should be less than for x.y. There are many other ways we could potentially improve the heuristics here but they're still just heuristics, so I don't really care. The change here is really just that if we decide an import is inside if MYPY it should get an even lesser priority.

@elazarg
Contributor
elazarg commented Sep 22, 2016

I have the time, sure.

I can't say I already know how to test this; I will study it tomorrow.

@elazarg
Contributor
elazarg commented Sep 22, 2016

order_ascc() docstring should be updated: N=4

@elazarg
Contributor
elazarg commented Sep 22, 2016

The new truth values suddenly makes it hard to be sure what's happening in other functions that return truth values. I don't have any solution for that but I do think it's a problem.

@gvanrossum
Member
@elazarg
Contributor
elazarg commented Sep 22, 2016

Do you want me to add tests to test/testgraph.py?

I am not familiar enough with the code, but I couldn't find anything like a test/testmoduledep.py with moduledep-*.test files structured like

[case testSomeDep]

[module x]
class Base: pass

def foo() -> None:
    import y
    reveal_type(y.Sub.attr)  # E: Revealed type is builtins.int

[module y]
import x
class Sub(x.Base):
    attr = 0

[check y]

This is more general that this PR however.

I think this PR should also be asserted in the assertion pass somehow, verifying that nested dependencies are prioritized lower than dependencies on enclosing scope, for example. Of course there are no assertion passes yet.

@gvanrossum
Member
@elazarg
Contributor
elazarg commented Sep 22, 2016

I missed that, sorry. I'm on it.

mypy/semanal.py
@@ -2983,12 +2993,16 @@ def infer_reachability_of_if_statement(s: IfStmt,
platform: str) -> None:
for i in range(len(s.expr)):
result = infer_if_condition_value(s.expr[i], pyversion, platform)
- if result == ALWAYS_FALSE:
+ if result in (ALWAYS_FALSE, MYPY_FALSE):
# The condition is always false, so we skip the if/elif body.
mark_block_unreachable(s.body[i])
@elazarg
elazarg Sep 23, 2016 Contributor

Just to make sure, the intention is that if not TYPE_CHECKING: means "don't type check"? That's fine but not explicitly documented in PEP-484.

In this case

if TYPE_CHECKING:
    import expensive_mod
else:
    # I don't care about runtime annotations
    expensive_mod = CleverPseudoModule('SomeClass')

def a_func(arg: expensive_mod.SomeClass) -> None:
    ...

The else branch will not be checked at all. Again, it's fine, but I think it should be documented.

@gvanrossum
gvanrossum Sep 23, 2016 Member

Even though PEP 484 only gives a positive example, it clearly states that TYPE_CHECKING

is considered True during type checking (or other static analysis) but False at runtime.

I don't see any ambiguity there.

@gvanrossum

Thanks for the review. I'll push a new version with some docstring/comment updates. Do you have unit tests for this yet?

mypy/build.py
@@ -308,9 +308,22 @@ def default_lib_path(data_dir: str, pyversion: Tuple[int, int]) -> List[str]:
PRI_MED = 10 # top-level "import X"
PRI_LOW = 20 # either form inside a function
PRI_INDIRECT = 30 # an indirect dependency
+PRI_MYPY = 40 # inside "if MYPY" or "if typing.TYPE_CHECKING"
PRI_ALL = 99 # include all priorities
@gvanrossum
gvanrossum Sep 23, 2016 Member

It would still be arbitrary. There are ideas for something better in the tracker, but I'd like to see how things work out with this small improvement first.

@@ -394,20 +407,21 @@ def correct_rel_imp(imp: Union[ImportFrom, ImportAll]) -> str:
for imp in file.imports:
if not imp.is_unreachable:
if isinstance(imp, Import):
- pri = PRI_MED if imp.is_top_level else PRI_LOW
+ pri = import_priority(imp, PRI_MED)
+ ancestor_pri = import_priority(imp, PRI_LOW)
@gvanrossum
gvanrossum Sep 23, 2016 Member

It's only used once, for the ancestors. The reason is that if you have "import x.y" you depend on both x.y and on x, but the priority for x should be less than for x.y. There are many other ways we could potentially improve the heuristics here but they're still just heuristics, so I don't really care. The change here is really just that if we decide an import is inside if MYPY it should get an even lesser priority.

-# Inferred value of an expression.
-ALWAYS_TRUE = 0
-ALWAYS_FALSE = 1
-TRUTH_VALUE_UNKNOWN = 2
@gvanrossum
gvanrossum Sep 23, 2016 Member

Yeah, but that would just cause more code churn and the benefit would be minuscule.

+ TRUTH_VALUE_UNKNOWN: TRUTH_VALUE_UNKNOWN,
+ MYPY_TRUE: MYPY_FALSE,
+ MYPY_FALSE: MYPY_TRUE,
+}
@gvanrossum
gvanrossum Sep 23, 2016 Member

The numbers are entirely uninteresting. I intentionally changed them around a bit to show that this is so.

mypy/semanal.py
@@ -2983,12 +2993,16 @@ def infer_reachability_of_if_statement(s: IfStmt,
platform: str) -> None:
for i in range(len(s.expr)):
result = infer_if_condition_value(s.expr[i], pyversion, platform)
- if result == ALWAYS_FALSE:
+ if result in (ALWAYS_FALSE, MYPY_FALSE):
# The condition is always false, so we skip the if/elif body.
mark_block_unreachable(s.body[i])
@gvanrossum
gvanrossum Sep 23, 2016 Member

Even though PEP 484 only gives a positive example, it clearly states that TYPE_CHECKING

is considered True during type checking (or other static analysis) but False at runtime.

I don't see any ambiguity there.

mypy/semanal.py
# The condition is always false, so we skip the if/elif body.
mark_block_unreachable(s.body[i])
- elif result == ALWAYS_TRUE:
+ elif result in (ALWAYS_TRUE, MYPY_TRUE):
# This condition is always true, so all of the remaining
# elif/else bodies will never be executed.
@gvanrossum
gvanrossum Sep 23, 2016 Member

Depending on the meaning of "always". :-)

@@ -3024,12 +3039,9 @@ def infer_if_condition_value(expr: Node, pyversion: Tuple[int, int], platform: s
elif name == 'PY3':
result = ALWAYS_TRUE if pyversion[0] == 3 else ALWAYS_FALSE
elif name == 'MYPY' or name == 'TYPE_CHECKING':
- result = ALWAYS_TRUE
+ result = MYPY_TRUE
@gvanrossum
gvanrossum Sep 23, 2016 Member

Well, we don't really care about other type checkers in this codebase, do we? :-)

+ def visit_import_all(self, node: ImportAll) -> None:
+ node.is_mypy_only = True
+
+
def is_identity_signature(sig: Type) -> bool:
"""Is type a callable of form T -> T (where T is a type variable)?"""
if isinstance(sig, CallableType) and sig.arg_kinds == [ARG_POS]:
@gvanrossum
gvanrossum Sep 23, 2016 Member

Yeah, we have a bug open about that already.

@elazarg
Contributor
elazarg commented Sep 24, 2016

I didn't find any useful test case yet. Sorry. I could use pointers if you (or @JukkaL) have any.

@gvanrossum
Member
@elazarg
Contributor
elazarg commented Sep 25, 2016

I think there's some mismatch between the unit tests and an installed mypy (I'm embarrassed to say I don't know how to run mypy without actually installing it :\ ). Running mypy, Jukka's example from Aug 12 #2016 fails without this PR and passes with it, but when I try to transform it into a unit test it crashes. I will investigate this crash.

@elazarg
Contributor
elazarg commented Sep 25, 2016 edited

Turns out it was the usual missing fixture which pops up in the wrong places. Test case:

[case testTypeCheckPrio]
# cmd: mypy -m part1 part2 part3 part4

[file part1.py]
from part3 import Thing
class FirstThing: pass

[file part2.py]
from part4 import part4_thing as Thing

[file part3.py]
from part2 import Thing
reveal_type(Thing)

[file part4.py]
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from part1 import FirstThing
def part4_thing(a: int) -> str: pass

[out]
tmp/part3.py:2: error: Revealed type is 'def (a: builtins.int) -> builtins.str'

[builtins fixtures/bool.pyi]
@gvanrossum
Member
@elazarg
Contributor
elazarg commented Sep 25, 2016

What do you think about the idea of permuting the names in command line arguments? I've said it has significant slowdown but it's only so for a handful of cases - the cases for which it is important - so in total it's not a significant hit.

@gvanrossum
Member
@elazarg
Contributor
elazarg commented Sep 25, 2016 edited

I wrote it as an edit, sorry... Here's the cut&paste:

Since the original problem is sensitive to order of command line arguments, I have tried running all possible permutations, and it passes them all. I'm not sure whether this is a useful addition, but if you want to give it quick try, just replace test/testcheck.py:111 with:

        from itertools import permutations
        for module_data in permutations(self.parse_module(original_program_text, incremental)):
            # indent the rest of the function

I think it is a nice sanity check (unless I miss something and these were all actually the same test somehow).

@gvanrossum
Member
@gvanrossum gvanrossum changed the title from Add a new, lower priority for imports inside "if MYPY" to [WIP] Add a new, lower priority for imports inside "if MYPY" Sep 29, 2016
@gvanrossum
Member

I'm tempted to abandon this. I tried a bunch of variations, including increasing the priorities for code inside if MYPY, decreasing them but keeping the differentiation between import forms, and the results are discouraging. On one of our internal codebases ("S") none of this makes a difference. On the other codebase ("C") every tweak causes a different set of errors to occur. This latter codebase has more annotated code and a very large import cycle (> 400). I think we may need a different strategy.

@elazarg
Contributor
elazarg commented Sep 29, 2016 edited

Personally I believe that the real solution requires taking incremental to the extreme, type checking expressions on demand (i.e explicitly building the dataflow graph and checking/inferring by need, caching whenever possible). This will require an even larger change than #1292, so I guess it will never happen.

Back to the real world: is it possible to extract the dependency graphs from actual codebases, so they can be visualized and analyzed independently of the code, and perhaps we can find a common pattern? If we build such a tool, it can be applied any time someone opens an issue on the subject.

@gvanrossum
Member

Personally I believe that the real solution requires taking incremental to the extreme, type checking expressions on demand (i.e explicitly building the dataflow graph and checking/inferring by need, caching whenever possible).

Agreed. I am planning to round up the resources to do this, but it will be a while.

This will require an even larger change than #1292, so I guess it will never happen.

It will.

Back to the real world: is it possible to extract the dependency graphs from actual codebases, so they can be visualized and analyzed independently of the code, and perhaps we can find a common pattern? If we build such a tool, it can be applied any time someone opens an issue on the subject.

Extracting imports is pretty simple; I've written throwaway tools for this several times. I've also tried to visualize the dependency graphs, but the problem is that once you have a graph of several 100 nodes (like we do) a traditional node + arrow diagram just looks like an entirely black page. I've looked into other visualization styles too, and even once wrote something that can show dependencies as an Adjacency Matrix (Google it) and lets you interactively show and hide certain parts of the graph. But I wrote it in JS using D3, which is a bit out of my regular skill set, so I gave up before I got it working well.

@JukkaL
Collaborator
JukkaL commented Sep 29, 2016

@gvanrossum Being somewhat familiar with the "C" codebase, I suspect the problems may be at least partially due to there being a "hub" module that gets imported by a huge number of modules in the cycle using if MYPY imports. With this change the hub module will perhaps be type checked as one of the last things in the cycle, whereas otherwise it might be somewhere in the middle. Since a lot of modules use the hub module, having inferred types available earlier will be useful. There probably isn't any single cause, though :-(

Here's one idea that might be worth trying, if you haven't tried it yet: decrease the priority of from x import statements within if MYPY blocks to be the same as import x, but otherwise don't treat them specially compared to other imports. That it, there would still be the same set of import priorities as now, but everything inside if MYPY would have (up to) PRI_MED priority. This would be a more conservative change but it might still be helpful.

@elazarg Fine-grained incrementalism would certainly solve this problem. We'd basically type check the entire cycle as a single unit, in dependency order. However, you are correct that it would be a big change, and we should first try any simpler approaches that are available to us. It's still perhaps the best thing to do -- the original design didn't anticipate large import cycles, and so it wasn't designed to cope with them.

A simpler option would be run multiple type checking passes over each cycle until all the needed types have been inferred. We already do this over individual files, and it shouldn't be very hard to generalize this to cycles -- it would almost certainly be much easier that doing "real" finer-grained incrementalism.

@elazarg
Contributor
elazarg commented Sep 29, 2016 edited

@gvanrossum Having a database containing dependency graphs "from the wild" can be a nice playground for experimenting with heuristics and other solutions for breaking cycles. I don't know what kind of information is needed to be there (the shape alone is not enough), so it might not be practical; but currently the only codebase available for me to look at is mypy.

@JukkaL Fine-grained incrementalism can be helpful for IDEs too.

@JukkaL
Collaborator
JukkaL commented Sep 29, 2016

@elazarg Yeah, that's part of what originally motivated the idea. IDE support is definitely on our radar.

@gvanrossum
Member
@gvanrossum
Member
@JukkaL
Collaborator
JukkaL commented Sep 29, 2016

But if we don't do something to improve things (even it requires a significant set of tweaks to migrate to the new version) they'll have to keep adding those hacks in the future. Once the module ordering within the cycle is more stable, there should less trouble in the future (though there still likely is going to be some trouble). One-time, significant pain can be better than continuous, low-level pain.

I wonder if it would be possible to look at the ordering of the import cycle historically, using both the old sorting method and some of the proposed variants. For example, pick 5 revisions over some time period, sort the biggest import cycle in each revision using all the algorithms, and see which algorithm produces the most stable module orderings, using some relevant metric. This wouldn't be perfect but at least it would be independent of the hacks they've employed.

Stable ordering could mean that the relative order of as few modules as possible is changed: i.e., for each pair (x, y) of modules, we'd calculate whether x comes before y (or vice versa) in both orderings. Stability between two orderings would be the fraction of pairs that have the same order in both orderings (we'd only consider pairs that appear in both orderings).

@elazarg
Contributor
elazarg commented Sep 29, 2016

Do you think ranking modules by the number of externally-used definitions (within a single scc) can help?

@JukkaL
Collaborator
JukkaL commented Sep 29, 2016

Tweaking priority based on the number of names imported from other modules might help. The kinds of things imported might also be a reasonable heuristic (importing a class would have a higher priority than importing a non-decorated function).

There are two things that can be helpful:

  1. Generate a relatively stable order for modules within a cycle, as changes in order can result in unexpected errors in seemingly unrelated code, usually due to types that haven't been inferred.
  2. Type check modules that define things used in a lot of places earlier (or that define things that are hard to infer types for, such as some decorated functions).

We've been focusing on (1), but (2) might be interesting as well.

@gvanrossum
Member

Abandoning this.

@gvanrossum gvanrossum closed this Oct 6, 2016
@gvanrossum
Member
gvanrossum commented Oct 12, 2016 edited

FWIW, this would fix the specific example given in #2015. The reason is that that example specifically puts one import inside if MYPY. Without the hack in this PR, the dependencies between nodes of the cycle (part[1234]) all have the same priority, so they are processed in the order 4, 3, 2, 1. But with the added information from deprioritizing imports inside if MYPY, the priorities change and the processing order becomes 4, 2, 3, 1 (i.e. 2 and 3 are swapped) and that solves the problem.

@gvanrossum gvanrossum reopened this Oct 12, 2016
@gvanrossum gvanrossum changed the title from [WIP] Add a new, lower priority for imports inside "if MYPY" to [depends on #2264] Add a new, lower priority for imports inside "if MYPY" Oct 17, 2016
gvanrossum added some commits Sep 21, 2016
@gvanrossum gvanrossum Add a new, lower priority for imports inside "if MYPY" fc2f577
@gvanrossum gvanrossum Update docstrings/comments in response to review. 8bab47b
@gvanrossum gvanrossum Add unittest by @elazarg a078583
@gvanrossum gvanrossum Tweak import_priority; gives slightly fewer errors
9a3308d
@gvanrossum gvanrossum changed the title from [depends on #2264] Add a new, lower priority for imports inside "if MYPY" to Add a new, lower priority for imports inside "if MYPY" Oct 24, 2016
@gvanrossum
Member

@JukkaL, this is now once again ready to review/merge! (It was ready before, except it reordered some import cycle in repo "C". That's now been taken care of by #2264.)

@gvanrossum gvanrossum Silence lint
bb26391
@JukkaL
Collaborator
JukkaL commented Oct 24, 2016

Ok, I'll probably look at this this tomorrow.

@JukkaL
Collaborator
JukkaL commented Oct 25, 2016

Oops, lost track of this. Will continue tomorrow.

mypy/build.py
+ return PRI_LOW
+ if imp.is_mypy_only:
+ # Inside "if MYPY" or "if typing.TYPE_CHECKING"
+ return max(PRI_MED, toplevel_priority)
@JukkaL
JukkaL Oct 26, 2016 Collaborator

What if we'd make the priority of mypy-only imports even lower, since they can never affect the runtime module initialization order? I feel that more priority levels is generally better, as there will be fewer ties when sorting cycles and thus we should get a more stable ordering. So my suggestion is to create a new priority level for mypy-only imports, which would be either between PRI_LOW and PRI_MED or even lower than PRI_LOW.

@JukkaL
JukkaL Oct 26, 2016 Collaborator

(And by lower I meant numerically higher...)

@@ -1313,3 +1313,27 @@ pass
[file b]
pass
[out]
+
+[case testTypeCheckPrio]
@JukkaL
JukkaL Oct 26, 2016 Collaborator

Does this test case fail without the changes in this PR?

@elazarg
elazarg Oct 26, 2016 Contributor

Yes

@gvanrossum gvanrossum Add separate PRI_MYPY priority, below PRI_LOW
dc1f8c2
@gvanrossum
Member

Pushed a new version. (BTW testTypeCheckPrio is almost straight from #2015 and/or #2016.)

@gvanrossum gvanrossum Silence lint
fa707f6
@codecov-io

Current coverage is 83.17% (diff: 97.36%)

Merging #2167 into master will increase coverage by 0.01%

@@             master      #2167   diff @@
==========================================
  Files            72         72          
  Lines         19048      19068    +20   
  Methods           0          0          
  Messages          0          0          
  Branches       3920       3921     +1   
==========================================
+ Hits          15840      15860    +20   
- Misses         2612       2613     +1   
+ Partials        596        595     -1   

Powered by Codecov. Last update 68c6e96...fa707f6

@gvanrossum gvanrossum merged commit 1fb67d0 into master Oct 27, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@gvanrossum gvanrossum deleted the deprio branch Oct 27, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment