Allow use of Required and NotRequired to make an existing typed dict total or optional

[CaselIT]

I think the ability of indicating that a typed dict requires all keys or that all are optional would be very useful.

For example to allow defining default values for a particular dict or to return a dict that has been completely assigned from a function

A simple example

class A(TypedDict):
    a: int
    b: int

ADefault = {'a': 1, 'b': 42}

def optionalA(v: NotRequired[A]) -> A:
    return ADefault | v

optionalA({'a': 11}) # ok since all keys of NotRequired[A] are optional

class X(TypedDict, total=False):
    x: str
    y: str

XDefault: X = {"x": "xv", "y": "yv"}

def withXDefault(v: X) -> Required[X]:
    return XDefault | v

withXDefault({"x": "foo"})["y"]  # ok since all keys of Required[X] are required

This idea is inspired by the analogous types in typescript Required and Partial that are in general very convenient.

I think it makes sense having the same behaviour of the typescript counterparts:

• NotRequired[X] is a no op, since X is defined as total=False. Same for Required[A].
• Required/NotRequired would include all keys of the typed dict and all its superclasses. This differs from how total works. Example:

  class TD1(TypedDict):
      a: int
  
  class TD2(TypedDict, total=False):
      b: int
  
  TDR = Required[TD2]  # b becomes required
  TDNR = NotRequired[TD2]  # a becomes not required

• Required/NotRequired used inline in the TypedDict are not taken into consideration, so the end result is the same independently of how the typed dict is defined

[erictraut]

@CaselIT, you might be interested in this discussion about an approach that would enable what you're proposing but in a more general way, similar to how TypeScript works.

[CaselIT]

Sure, that would achieve the same level of features, but just skimming that proposal it seems several years worth of peps.

I think the two could be compatible with each other, so if that other proposal is ever accepted these features could be re-defined using that syntax.

The main advantage of this limited change is that only type checkers would need to be changes, since the code above is already legal python, not needing additional types or features (if keeping NotRequired instead of Partial). The limitation seems completely arbitrary

Note that this issue comes from trying to use a typed dict with several keys (about 15) with a mix of required and not-required ones for an api. Trying to define default types for the optional key, add utilities to merge a typed dict with a partial one etc is currently an exercise in frustration if one has ever even tangentially used typescript (or not, it really isn't user friendly). I've ended up defining the same typed dict multiple times to have the various combination of required / not required

[erictraut]

I agree that the more general solution would require a more involved PEP, but your proposed solution would require a PEP too.

Your proposed solution would create an ambiguity in the type system because NotRequired and Required would have two contradictory meanings. You can see this in the following code:

class A(TypedDict):
    a: str

class B(TypedDict):
    # Does this mean `b` is a not-required field in B?
    # Or does it mean that `b` is required but the fields in `A` are not required?
    b: NotRequired[A]

Ambiguities like this will create problems in the type system. Even if the PEP clearly disambiguates the meaning in this particular usage, there will be others cases — either now or as new features are added — where this ambiguity will cause problems. For this reason, I think it's unlikely for this proposal to gain acceptance, at least in the form you've suggested.

[CaselIT]

I agree that the more general solution would require a more involved PEP, but your proposed solution would require a PEP too.

sure, I'm not saying otherwise, just that the scope is vastly different.

Regarding ambiguity, you are indeed right. Note that I've used the Required/NotRequired names since they already exist. Also since typechecker currently error when used outside the definition of a typed dict, I though it would not cause too much of an issue, but the typed dict key is indeed a problem

For NotRequired there is the option of using Partial as in typescript, I have no good suggestions of an alternative to Required.
Personally I think PEP 655 should not have been accepted with the given names, since it uses too general names for a too narrow scope. RequiredKey / NotRequredKey (or better OptionalKey) would have been much better. But that ship has sailed

[gresm]

This would be useful with "dict factories":
Let's have a class A:

class A(TypedDict):
    property1: str
    property2: str

Some default vales for it:

default_A: A = {"property1": "a", "property2": "b"}

And a naive implementation of factory:

def A_factory(**kwargs: Unpack[A]):
    ret = default_A.copy()
    ret.update(kwargs)
    return ret

The issues is that all of the properties are required by a type-checker, so actually no default values. Currently it is required to define a new class with all of the properties the same, but optional:

class B(TypedDict, total=False):
    property1: str
    property2: str


def A_factory(**kwargs: Unpack[B]):
    ...

With NotRequired, this no longer would be an issue. My first attempt to do such a thing was with inheritance:

class B(A, total=False):
    pass


class B(A, TypedDict, total=False):
    pass

But both of them didn't work, as total only works with keys that are defined in current class:

>>> class A(TypedDict):
...  a: str
... 
>>> class B(A, total=False):
...  b: str
... 
>>> A.__required_keys__
frozenset({'a'})
>>> B.__required_keys__
frozenset({'a'})
>>> A.__annotations__
{'a': <class 'str'>}
>>> B.__annotations__
{'a': <class 'str'>, 'b': <class 'str'>}

This behavior is of course reasonable and in other cases useful, the ability to mark upper class properties as optional/required in subclass would be also useful. The first example in which I just create a duplicate class works, but it gets messy quickly (keeping the second class up-to-date) and when using inheritance in the first class (A), you need to manually resolve the inheritance tree (explained why in the last example). NotRequired[A] could avoid it altogether, as it would mark every item optional:

def A_factory(**kwargs: Unpack[NotRequired[B]]):
    ...

But as already pointed out, NotRequired usage here can be ambiguous. If ever implemented, it will likely have a different name.

[CaselIT]

@gresm that's basically my use case

[gresm]

So it's not so uncommon. That's good to know.