Temporary moratorium on literal constants (until the end of 2022) #7258
Comments
|
At least in the sort term we should revert in cases where literals cause problems and in the future only use literals where it's unlikely that the literal gets assigned to a mutable variable. In the long term we should rethink how type checkers handle literals. For example, FOO: Literal[1] = 1
BAR: Literal[2] = 2
x = FOO # `x` should probably have the type `int`, not `Literal[1]` |
|
Actually, at least in mypy the following already works: FOO: Final = 1
BAR: Final = 2
x = FOO
reveal_type(FOO) # type: Literal[1]
reveal_type(x) # type: intSo maybe we should allow |
|
Cc @rchen152 @erictraut @pradeep90 for feedback on using |
|
Also Cc @sobolevn as " |
|
In #7259 pytype currently fails with "ParseError: Default value for CRITICAL: $external$typing_extensions.Final can only be '...', got LITERAL(50)", so we'd at least need to fix that and of course flake8-pyi. |
This to me feels much more expressive than writing I worry, however, that there's not much point in annotating a variable as |
Using Y: Final = XSince a variable with a literal type is effectively a constant, using PEP 586 has some relevant discussion:
A common use case for explicit literal types is as function arguments, and using a X: Final = 5
f(X) # X treated as Literal[5] as neededSimilarly, this is fine: X: Final = 5
a: Literal[5, 6] = X |
|
Thanks for letting me know. I think this is a perfect example of "soft literal" type that I've mentioned several times in different discussions. So, basically we have two cases:
I know that @Akuli was interested in this as well. |
|
Just double-checking that I understand correctly: If so:
|
See python/typeshed#7258. While that issue proposes that Final constants be handled differently from Literal ones, we generally allow type mutations (unless a name is annotated), so it should be fine to just convert Final to Literal. PiperOrigin-RevId: 429642600
I'm not seeing any errors when I run mypy on these code samples. Are you saying that mypy infers that the type of Type stub files should be as unambiguous as possible. The statement So I would recommend against using the x: Literal[3] # x is a read/write variable that can be assigned only a value of 3
x: Final[Literal[3]] # x is a read-only variable that has a value of 3
x: Final = 3 # x is a read-only variable that most (all?) type checkers will infer to have the type Final[Literal[3]]Edit: reading this section of PEP 586 makes it pretty clear that |
|
Pyright follows these rules consistently:
Here's how these rules apply to the examples above: import logging
class C:
def __init__(self) -> None:
reveal_type(self.level) # int
self.level = logging.INFO
reveal_type(self.level) # Literal[20] (narrowed via assignment)
def enable_debug(self) -> None:
reveal_type(self.level) # int
self.level = logging.DEBUG
reveal_type(self.level) # Literal[10] (narrowed via assignment)
c = C()
reveal_type(c.level) # int
c.level = 1
reveal_type(c.level) # Literal[1] (narrowed via assignment)
c.level = "hi" # Error because inferred type of `C.level` is `int`FOO: Literal[1] = 1
x = FOO
reveal_type(x) # Literal[1] (narrowed via assignment)
x = 3 # This is not an error because the type of x was not declared
reveal_type(x) # Literal[3] (narrowed via assignment)
x = "hi" # This is not an error because the type of x was not declared
reveal_type(x) # Literal["hi"] (narrowed via assignment)@sobolevn, I don't understand what you mean by a "soft literal". If you declare a variable with the type |
Literal types are only supported for simple types such as I agree that this would a little inconsistent with how we define other things in stubs, but it would be consistent with how you'd define the variable in non-stub code. With A: Final = 'some-value
B: Final = AIf we'd use explicit literal types in non-stub code, this would look something like the following: A: Final[Literal['some-value']] = 'some-value
B: Final[Literal['some-value']] = AWe had to duplicate the literal value three times. Even though this is more explicit, I much prefer the shorter version. If we'd support
Mypy doesn't infer Literal[3] as the type, as discussed above, but something a little different, as implied in PEP 586 (but the relevant section in the text is not super clear, I have to admit). PEP 586 was based on the implementation in mypy, so if something can be interpreted in multiple ways, the mypy interpretation seems implicitly like a valid one (though not necessarily the only one!). Most PEPs can't explain every detail -- that's one reason why it's good to have a prototype/reference implementation, as it can "fill in the gaps".
Okay, that section apparently can be understood in multiple ways. Here the intention was that 'effectively literal' is different from just plain 'literal'. The key part is this: "Specifically, given a variable or attribute assignment of the form var: Final = value where value is a valid parameter for Literal[...], type checkers should understand that var may be used in any context that expects a Literal[value].". If this would be exactly the same as a literal type, the above passage would feel badly/confusingly written to me. Instead, it would be better written as e.g. " For something with a "real" literal type (i.e. not "effectively literal"), mypy treats literal types like any other type. It is propagated via type inference, etc. Mypy treats explicit literal types similarly to TypedDict types. The same constructor (e.g. string literal or dict expression) can generate values of multiple types, depending on the type context. Here's an example: def f(a: Literal['x', 'y']) -> None:
b = a # b has type Literal['x', 'y'], propagated from rhs which has an explicit literal type
c = 'x' # c has type str, since no explicit annotation
d: Literal['x', 'y'] = 'x' # d has type Literal['x', 'y']
def g(a: SomeTypedDict) -> None:
b = a # b has type SomeTypedDict
c = {'k': 5, ...} # c has type dict[str, ...], since no explicit annotation
d: SomeTypedDict = {'k': 5, ...} # d has type SomeTypedDictA type like Anyway, if different type checkers have interpreted PEP 586 differently, it seems to me that we should avoid using literal types in stubs in contexts where their semantics aren't consistent, at least until we can figure out a convention that works for all major tools. Clearly part of the reason for the confusion is that PEP 586 was kind of vague and open to multiple interpretations, some of which were even explicitly allowed in the PEP. Perhaps we could narrow the semantics down bit, at least for stubs. |
|
Another way to explain the mypy behavior is that explicit literal types are "sticky"/"strict" and don't get implicitly "weakened" into the underlying runtime type such as # can achieve many things that real enums can do, but require some extra type annotations
EnumLike = Literal['x', 'y', 'z', ...]In contrast, The type |
|
Thanks for the additional details and background information, @JukkaL. I think I understand what you mean by "soft" literals versus "real" literals, but that's an internal implementation detail of mypy that is not documented in any specification. Pyright doesn't use such a distinction, but it uses a variety of other heuristics to determine when a value should be interpreted as literal or widened. Incidentally, I find some of mypy's behaviors regarding literals very non-intuitive. Thanks to your explanation above, I now better understand the reason for these behaviors, but I still find them confusing as a user. For example, the following code sample type checks without errors in pyright. And indeed, there are no type safety issues here, but mypy emits three (what I would consider false positive) errors. from typing import Final, Literal
def requires_x(val: Literal["x"]): ...
a: Literal["x", "y"]
a = "x"
requires_x(a) # Mypy: OK
b: Literal["x", "y"] = "x"
requires_x(b) # Mypy: incompatible type
c: str
c = "x"
requires_x(c) # Mypy: incompatible type
d = "x"
requires_x(d) # Mypy: incompatible type
e: Final = "x"
requires_x(e) # Mypy: OK
Yeah, that makes sense. |
These are working as designed. The relevant rules are these: (1) We don't infer literal types for variables if there is no explicit Mypy type inference is quite conservative by design and doesn't even try to avoid these kinds of "false positives", since my earlier attempts with more powerful type inference (and experience with other languages with more complex type inference) resulted in friction from confusing error messages. I think that there is some similarity with the philosophy of Go, though clearly Go takes this much further. (2) The second feature allows type-safe upcasting in assignment to a more general type. This is sometimes a handy feature, and currently Python typing doesn't provide a clean way to do this, so mypy maintains this as an extension. If I want x: Sequence[str] = ['x', 'y'] # Use Sequence to discourage mutation
alias = x # alias also has type Sequence[str] to discourage mutationIt can also be used to infer a more general type for invariant containers (this is a legacy use case but changing this would break existing code): x: object = 's'
a = [x] # Inferred type is list[object], not list[str](I'm going to stop the discussion about the semantics of literal types here. This is going way off topic, sorry.) |
Is everybody else on board with this? To rephrase, if we agree with this, we'll stop adding literal types that don't behave consistently across type checkers (and can revert some existing uses). We can seek a long-term solution that works for everybody elsewhere (we can refer to this discussion). As far as I understand, using literal types in overloads poses no compatibility problems. The issue only affects module-level or class-level attributes with a literal type. Example: FOO: Literal[20]This would have to written like this: FOO: int # or Final[int] |
From a typeshed perspective, I am fine with temporarily stopping new literals and temporarily reverting literals that prove problematic for mypy. But in the long term having literals is a clear win for typeshed as it allows type checking "enum-like" arguments and fields more precisely. There could be other uses, like improved documentation, as well. I don't particular care whether we type those as |
srittau
changed the title
|
I wouldn't want to extend this moratorium past the end of the year (2022). I hope we'll find a solution until then. Otherwise I suggest that typeshed will continue to use the |
srittau
changed the title
|
Technically, it would be easy to change mypy to support |
See python#7258 for an extended discussion. In summary, for mypy these would be better written as follows, but this would not work with other type checkers: ``` CRITICAL: Final = 50 ```
See #7258 for an extended discussion. In summary, for mypy these would be better written as follows, but this would not work with other type checkers: ``` CRITICAL: Final = 50 ```
…7354) (#12239) Fixes some regressions. Context: python/typeshed#7258
|
As the author of #6610 who introduced these literals in question to typeshed, I figured I'd leave my thoughts on the matter here. It's kinda a shame I only stumbled upon this issue and discussion after 6 months since it started, but it's better late than never, especially since it's still open. Personally, I agree with @srittau about typeshed eventually committing to the Now, as far as the issue itself goes, the starting sentence of:
has immediately caught my attention, since I've personally provided a solution to this exact same problem that was described right under it, in the PR of mine. And I disagree with the:
statement, as this really isn't that complicated. In fact, it follows one of the "common issues" MyPy documentation describes. Now I know a Now, I know that MyPy isn't the only type checker typeshed provides the typing information for, but it'd only be a case of adding a similar explanation of this exact situation in other type checker documentations, in case it wouldn't be there already, possibly also mentioning that the same logic applies to I think that sufficiently solves the problem already. If desirable, the inference logic could be improved to try constructing a union of literals, if it detects an unannotated variable being assigned a literal, and a different literal later. |
|
I haven't reread the whole thread, so I might be missing something from the discussion. But from what I understand, we can't really use literals for enum- or flag-like constants, because when assigning a literal to a variable, at least in mypy the variable now has a literal type that can't be changed later: FLAG1: Literal[1]
FLAG2: Literal[2]
_Flag: TypeAlias = Literal[1, 2]
a = FLAG1 # type is now "Literal[1]"
a = FLAG2 # error
class Foo:
flag = FLAG1
class Bar:
flag = FLAG2 # errorWould typing such constants in typeshed like this work? _Flag: TypeAlias = Literal[1, 2]
X: Final[_Flag] = 1
Y: Final[_Flag] = 2As far as I understand, this would automatically get us "enum-like" type checking automatically. What would probably not work at the moment is code like this, although type checkers could probably make it work when they would special case the idiom above: def foo(x: Literal[1]): ... # we only accept a specific set of flags
foo(X) |
@srittau You can though, and I kinda explained this twice already, not sure why it's still unclear. Here's a practical explanation using your example: There is something to be said about how MyPy infers the type without an annotation, forcing only the first literal it gets assigned to, and not a This discussion also seems to argue about whether or not one should use typeshed/stdlib/logging/__init__.pyi Lines 298 to 305 in c626137 As far as the semantics go, either of the two translates to a literal value to me, so it's really whatever as long as it's usable as a literal in the end. The supposed problem arises from the fact that people would often assign those literals to a variable, which would infer the literal type, and then throw an error upon a reassignment attempt, like so: import logging
verbose = False # read from somewhere
log_level = logging.ERROR
if verbose:
log_level = logging.INFO # errorMy PR which added If this literal inference issue is really so bad to require the user to add an explicit annotation to, MyPy could change the inference logic to using |
|
Happy new year! With the end of 2022, what is the status on this? If I can specify the value of constants in the stubs I've contributed, I'd like to do it, and do it right :) |
|
My preference is that literal constants are only/mainly used in stubs when the value of the literal is documented or the value is otherwise semantically significant somehow. #9367 has some context. Otherwise we risk unnecessarily leaking implementation details in the stub, which is intended as a description of the (public) interface of a module. If we want to expose the value of a particular constant, I propose that we standardize on a # This would work identically in both a stub and the implementation
PY3: Final = TrueIt should be easy to add support for this, even If some type checkers don't support this currently in stubs. Mypy already supports this, and My reasoning is that it's better for the stub definition to be similar to the implementation. Writing the definition like this outside a stub would look out of place and redundant to me: PY3: Literal[True] = TrueI think that this is analogous to function definitions. We don't declare a function like this in a stub: func: Callable[[int], None]Instead the stub looks similar to the actual definition, even if this is sometimes more verbose than the prior option: def func(__x: int) -> None: ...I believe that adding type aliases that map to literal unions and using them as "fake enums" is a clever idea but ultimately too confusing for end users to be adopted as a best practice. I'm thinking of something like this, as suggested by @srittau above: _MyType = Literal[1, 2, 3]
CONST1: Final[_MyType] = 1
CONST2: Final[_MyType] = 2I discussed this in #9367, but here are my main arguments:
Using a real enum avoids all the issues and I think that we should recommend that libraries switch to enums instead (e.g. |
|
Is there anything left to do or discuss here? We seem to have been specifying literal constants for a while now. |
Indeed, the moratorium is over. |
I've encountered cases where the changes from
intto a literal type in stubs causes regressions for mypy users. Here's one (simplified) example:Here's another example:
Previously the types were
int, so no errors were generated. The code in the above examples doesn't look buggy to me, and the behavior of mypy also seems as expected.There could well be other problematic cases like this. Having more precise types is generally useful, but in the case of literal types, the regressions such as the above may make them not worth the potential false positives (which are hard to predict, since in most use cases literal types are fine). I wonder if there are concrete examples where the literal types bring concrete benefits that could offset the false positives.
There were several false positives like the above in a big internal codebase when using a recent typeshed. Working around the issues is simple by annotating the attributes as
int, but this could be quite non-obvious for non-expert users who aren't familiar with literal types (a somewhat advanced typing feature).Thoughts? I'm leaning towards preferring to only use literal types when there is a concrete use cases where they help, since the fallout from using them seems difficult to predict.
The text was updated successfully, but these errors were encountered: