Pattern matching raises ParseError when using return type annotation

[ArneBachmann]

def h(a is int, b is int): return a + b  # compiles
def h(a is str, b is str) = a + b  # fine
def g(a is int, b is int) -> int: return a + b  # raises error
def g(a is str, b is str) -> str = a + b

[ArneBachmann]

Gave me a headache while trying to understand pattern matching...

[evhub]

@ArneBachmann Yeah, right now pattern-matching functions don't support type annotations. For just return type annotation, this is probably doable, though it'd be much harder to also support argument type annotation.

[penntaylor]

This issue bit me as well. I figured out a manual way to get fully working type annotations into pattern-matching functions, so in principle it can be done, but I have not had a chance to determine how to patch the coconut compiler to automate this.

First, consider the following re-write of @ArneBachmann's function g, using mypy's comment-style annotations:

def g(a is int, b is int):
    # type: (int, int) -> int
    return a + b

g(3, 2)      # should pass typecheck
g(3, "this") # should NOT pass typecheck

The coconut compiler currently strips out the # type: ... comment, leaving this code unityped. Both subsequent calls to g typecheck, even though the second one should obviously fail.

However, if I manually put the type annotation back into the generated python code, we start to get somewhere:

...
# Compiled Coconut: -----------------------------------------------------------

def g(*_coconut_match_to_args, **_coconut_match_to_kwargs):  # line 1
    # type: (int, int) -> int
    # ^^^^^^^ manually put that line in
    _coconut_match_check = False  # line 1
...

Running mypy directly over the altered foo.py file yields some internal coconut type issues from __coconut__.py that we can ignore, then emits this:

foo.py:31: error: "MatchError" has no attribute "pattern"
foo.py:32: error: "MatchError" has no attribute "value"
foo.py:38: error: Argument 2 to "g" has incompatible type "str"; expected "int"

The last line is the typecheck fail we're after. We just need to deal with the MatchError stuff to get rid of extraneous errors coming from coconut's internally-generated code. That turns out to be easy; all we have to do is toss in type: ignore comments on lines 31 and 32:

...
        _coconut_match_err.pattern = 'def g(a is int, b is int):'  # type: ignore # line 1
        _coconut_match_err.value = _coconut_match_to_args  # type: ignore # line 1
...

And now (again ignoring errors coming from __coconut__.py, mypy emits this:

foo.py:38: error: Argument 2 to "g" has incompatible type "str"; expected "int"

Exactly the behavior we want. The arguments are clearly being typechecked. It's easy enough to show the return type is being properly checked as well.

One critical thing about the type: ignore comments inserted on lines 31 and 32: they have to be in exactly that format; that is, the type annotation has to come first, followed by an octothorpe, followed by any other "comment" associated with that line (in this case the coconut line number).

[penntaylor]

Should mention that this won't generally work with functions using @addpattern decorators. In the specific case of using those decorators to define a set of functions over a sum type with a common base class, I think it would work to put a default case over the naked base class as the very last pattern definition, and give explicit type annotation for only that one. All others would get type: ignore. Doesn't really help with the more general use of @addpattern to mimic multiple dispatch. Perhaps that could be done by scanning over all the patterns associated with a function and building up an enormous Union type. Not sure though.

[penntaylor]

It appears that some typing.overload declarations prior to a group of @addpattern style functions almost works for the multiple dispatch case. Mypy catches badly typed calls to the functions, but also complains "The implementation for an overloaded function must come last" -- even when it is last.

[evhub]

@ArneBachmann @penntaylor The proposed plan for adding type annotations to pattern-matching functions is to use the new typedef function as per #399, which should hopefully solve the problem this issue represents. Thoughts?

[penntaylor]

@evhub #399 looks like it would neatly address the case of operating over a sum type. For the more general multiple dispatch case is the idea to use Union[x, Y, ...] for the argument types?

[evhub]

@penntaylor Hmm... good point. You could definitely just write a Union manually, though perhaps another option might be to have typedef behave like

def typedef(actual_func: S, typed_func: T) -> Union[T, S] = actual_func

instead (and make addpattern do the same), though I fear what will then happen is that if any of your functions are untyped, you'll end up with an Any in the union, which will make the whole thing useless.

[penntaylor]

@evhub I think the only general way to avoid multimethods' collapsing into loose Unions or untyped messes is to use @overload decorators, since mypy will then keep track of each the full signature of each variation. Of course, that still allows a multimethod to collapse into an untyped mess if one of the overloads is untyped or unityped, but at that point the developer is essentially requesting that behavior.

[evhub]

Pattern-matching functions still don't support return-type annotations, though now they display nice error messages when you try rather than just a ParseError.

[evhub]

This is now fully supported as of coconut-develop>=1.5.0-post_dev107.