A NoReturn type is needed

[o11c]

In order to perform branch analysis, it is necessary to know which calls will never return normally. Examples are sys.exit (which always returns via exception) and os.exit (which never returns).

NoReturn should drop out of Union and dominate Intersection.

To manually mark a branch as dead, use cast(NoReturn, expression) in an expression context, or raise in statement context.

This can also be used as the return type of __init__ or __new__ or the metaclass's __call__ to indicate types that can't be normally instantiated.

[JukkaL]

A simpler approach would be to just teach tools about common things that don't return such as exit, and require some form of annotation elsewhere, such as a raise statement:

def f() -> int:
    if foo:
       return 1
    else:
        function_that_does_not_return()
        raise NotReachable

This has the benefit of making things more explicit for the reader and getting a runtime error if the expectation doesn't hold. Functions that don't return are special beasts anyway, and maybe explicit is better than implicit in this case.

[gvanrossum]

FWIW, I've never liked functions that don't return. Other tools are also
confused by them (e.g. the auto-dedent functionality in editors).

On Thu, Oct 29, 2015 at 9:49 PM, Jukka Lehtosalo notifications@github.com
wrote:

A simpler approach would be to just teach tools about common things that
don't return such as exit, and require some form of annotation elsewhere,
such as a raise statement:

def f() -> int:
if foo:
return 1
else:
function_that_does_not_return()
raise NotReachable

This has the benefit of making things more explicit for the reader and
getting a runtime error if the expectation doesn't hold. Functions that
don't return are special beasts anyway, and maybe explicit is better than
implicit in this case.

—
Reply to this email directly or view it on GitHub
#165 (comment).

--Guido van Rossum (python.org/~guido)

[o11c]

Requiring raise works in simple cases, but consider:

@overload
def foo(arg: int) -> int: ...
@overload
def foo(arg: str) -> NoReturn: ...
# other overloads with different return types

def bar(arg: Union[int, str]):
    x = foo(arg) # x has type `int`

just teach tools about common things that don't return such as exit,

Needing to special-case like that is a sign of bad design.

And auto-dedent is never going to be perfect anyway, it's more important to have completeness.

[elazarg]

(I apologize for repeating things I've said on related discussions):
NoReturn is simply the empty Union (though in mypy there's a dedicated class UninhabitedType). It might need an alias for readability, but these concepts have exactly the same meaning, AFAIK.

[gvanrossum]

We're moving ahead with NoReturn as a mypy extension first, see python/mypy#2637. Once we're confident that it works we'll add it to typing.py -- this issue will be left open as a reminder for that.

Regardless of whether it's the same as Union[()], we should not spell it that way -- it's ugly, it's hard to search for, and it currently doesn't work anyways, so it would be subject to the same release-mechanical problems as NoReturn (probably slightly worse because you'd have to remember to write from mypy_extensions import Union).

[elazarg]

I don't think I have suggested to spell it Union[()], from the point of view of the user; I agree it's ugly. I just want to treat it as equivalent, regardless of syntactic difficulties, instead of having a new concept floating around in mypy (we have UninhabitedType already). So if it does not worth a new syntax, it can be aliased right inside mypy.

Unless there is a difference, in which case it's interesting to know about it (for example, if None is an element of the empty union, then they are not equivalent)

[gvanrossum]

@ddfisher ^^ Here's some feedback from @elazarg for the implementation of NoReturn for you.

[ddfisher]

None may or may not be a member of the empty Union in non-strict-Optional mypy -- I don't think it's currently clear either way (but this shouldn't be a big deal, because typing doesn't allow you to specify empty Unions).

I think there's a decent argument to be made that NoReturn should be the same as UninhabitedType. From a type-theoretical perspective, it certainly makes sense, but I'm not sure we actually want them to behave identically. Let me cogitate on this a bit.

[markshannon]

What is the point of this?
To specify that a function never returns (like sys.exit in a stub file) or to verify that a function never returns?

The former can be easily done with

def exit(code=1):
     raise SystemExit(1)

I don't really see the point of the latter.

[gvanrossum]

It is needed to specify that a function never returns, so that local code paths involving such a call can be analyzed correctly.

We considered the raise syntax you proposed, but (a) it could only work in stubs, and (b) this is already used in typeshed to indicate that a function may raise a given exception (presumably as a pytype extension).

[markshannon]

That's unfortunate.

Why not distinguish between "may raise" and "will raise"?

def may_raise(...):
    if random():
        raise ...

def will_raise():
     raise ...

I do worry that a lot of useful information that could be in the stub files is being thrown away.

[elazarg]

I think "no return" and "will raise" are different guarantees, and

def foo() -> NoReturn:
    do_some_stuff()

is very clear, and we can also verify that do_some_stuff() does not return.

What is the problem with this addition?

[markshannon]

There are three ways to not return.

1. Raising an exception
2. Non-termination
3. Segfaulting
   Lets ignore 3 as the interpreter shouldn't segfault.
   2 can be expressed easily:

def non_terminate():
    while True:
        pass

Should the above be marked as NoReturn? If not would be an error to call it

"What is the problem with this addition?"
The same as for any other addition.
Unless you can give a complete and sound sets of typing rules for it, including its interaction with all the other types, then how we do know it can work?

[markshannon]

FTR, we (Semmle) already do sophisticated control flow analysis, including pruning edges following calls to functions that do not return. No annotation is needed. Whether a function returns or not is purely a function of its control flow graph and the call graph. No type information is needed.

[elazarg]

It's the empty type. I think we know pretty well how it works.

[markshannon]

Given it is unnecessary, adding it merely adds extra work for those of us implementing checkers.
What sort of error does List[NoReturn] give (I assume that it is not OK), how does it impact subtyping of function types, etc, etc?

[elazarg]

There are two answers for this: 1. No matter how you spell it, if you allow expressing this specification then you run into the same questions. 2. I don't see what's wrong with the straight forward type-theoretical answers (only empty lists are allowed, NoReturn is a subtype of everything, etc.), plus whatever user friendly special casing you deem appropriate. In particular, NoReturn can be treated uniformly as the empty Union. I agree that a proof of concept will be helpful.

[markshannon]

IMO, pruning edges in the CFG is not really a type checking operation, and so NoReturn shouldn't really be a type.
My preference is for explicit stubs differentiating between "may raise", "will raise" and "never terminates".

That way the typing rules are kept simpler (CFG pruning should precede type checking) and the distinction between stubs and normal code is diminished, which again simplifies things.

[elazarg]

Since the only use of this type is as a return type, perhaps it can be defined not to work elsewhere; then you can treat it in the same way you'd treat non_terminate(), and other checkers can treat it as a type - this distinction will be an implementation detail, essentially.

[ddfisher]

I think NoReturn should be a type. So far we've maintained the invariant (in mypy at least -- not sure if this is discussed in the PEP) that everything you need to know about a function is contained in its type signature. This is important for the ability to do local-only inference and for specifying the behavior of too-dynamic-to-understand pieces of code. I think this is an important invariant to maintain.

From another lens, it seems reasonable to me that a user would want to be able to talk about functions that don't return as first-class citizens, and you can't do that unless you have some sort of NoReturn type.

[ilevkivskyi]

Since NoReturn is no more considered experimental, maybe it makes sense to move it from mypy_extensions to typing?

[gvanrossum]

Well, we haven't even had a single mypy release supporting it. So I don't know how "non-experimental" it is. In any case, the reason it's in mypy_extensions is mainly that we want to support it even on places that can't upgrade typing.py easily, e.g. existing Python 3.5.3 installations -- it'll be a long time before 3.5.4 will be released. (And ditto for 3.6.1 actually, I think it's too late to add it given that release schedule.)

[ilevkivskyi]

Well, we haven't even had a single mypy release supporting it. So I don't know how "non-experimental" it is.

I am referring here to what I have heard from @ddfisher in python/mypy#2920 (comment)

In any case, the reason it's in mypy_extensions is mainly that we want to support it even on places that can't upgrade typing.py easily

Yes, this is a good reason.

[gvanrossum]

I don't actually know what definition of "experimental" Jukka and David are using. Maybe it means a mypy feature that can be changed incompatibly or withdrawn in any future release? In any case, it's right that a mypy extension is not always a proposal for a typing feature (even though the current two, TypedDict and NoReturn, both are). I'm guessing also that the design of NoReturn at this point is uncontroversial so we can start updating the PEP and the next version of typing.py; but IIUC there are still many details for TypedDict that haven't been settled yet.

[JukkaL]

NoReturn is not experimental in the sense that it seems pretty uncontroversial and it has no major open issues. I still consider TypedDict quite experimental, however.

[gvanrossum]

So we also need changes to mypy that allow NoReturn to be imported from either typing or mypy_extensions.

[ilevkivskyi]

So we also need changes to mypy that allow NoReturn to be imported from either typing or mypy_extensions.

Here is a simple PR to allow this python/mypy#3041 (one line plus tests).
Later, we could gradually replace mypy_extensions with typing in tests and docs.