From 4a49013d228228db910b57e4dcd64012cc6f2d0e Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 22 May 2023 14:08:12 -0500 Subject: [PATCH 01/10] Tweak for recent comments --- pep-0712.rst | 79 +++++++++++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 35 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 768c7d6fa65..cb76240dfa0 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -71,12 +71,14 @@ New ``converter`` parameter --------------------------- This specification introduces a new parameter named ``converter`` to the -:func:`dataclasses.field` function. When an ``__init__`` method is synthesized -by ``dataclass``-like semantics, if an argument is provided for the field, the -``dataclass`` object's attribute will be assigned the result of calling the -converter on the provided argument. If no argument is given and the field was -constructed with a default value, the ``dataclass`` object's attribute will be -assigned the result of calling the converter on the provided default. +:func:`dataclasses.field` function. If provided, it represents a single-argument +callable used to convert all values when assigning to the associated attribute. +The converted values include: + +* The value specified during object construction +* The value of ``default``, when used +* The return value of ``default_factory``, when used +* The value used for attribute assignment on the object Adding this parameter also implies the following changes: @@ -91,13 +93,14 @@ Example @dataclasses.dataclass class InventoryItem: - # `converter` as a type + # `converter` as a type (including a GenericAlias). id: int = dataclasses.field(converter=int) skus: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...]) - # `converter` as a callable + # `converter` as a callable. + vendor: str | None = dataclasses.field(converter=str_or_none)) names: tuple[str, ...] = dataclasses.field( converter=lambda names: tuple(map(str.lower, names)) - ) + ) # Note that lambdas are supported, but discouraged as they are untyped. # The default value is also converted; therefore the following is not a # type error. @@ -105,12 +108,29 @@ Example converter=pathlib.PurePosixPath, default="assets/unknown.png" ) - item1 = InventoryItem("1", [234, 765], ["PYTHON PLUSHIE", "FLUFFY SNAKE"]) + # Default value conversion extends to `default_factory`; + # therefore the following is also not a type error. + shelves: tuple = dataclasses.field( + converter=tuple, default_factory=list + ) + + item1 = InventoryItem( + "1", + None, + [234, 765], + ["PYTHON PLUSHIE", "FLUFFY SNAKE"] + ) # item1 would have the following values: # id=1 # skus=(234, 765) + # vendor=None # names=('python plushie', 'fluffy snake') # stock_image_path=pathlib.PurePosixPath("assets/unknown.png") + # shelves=() + + # Attribute assignment also participates in conversion. + item1.skus = [555] + # item1's `skus` attribute is now `(555,)`. Impact on typing @@ -124,12 +144,13 @@ In other words, the argument provided for the converter parameter must be compatible with ``Callable[[T], X]`` where ``T`` is the input type for the converter and ``X`` is the output type of the converter. -Type-checking the default value -''''''''''''''''''''''''''''''' +Type-checking ``default`` and ``default_factory`` +''''''''''''''''''''''''''''''''''''''''''''''''' -Because the ``default`` value is unconditionally converted using ``converter``, -if arguments for both ``converter`` and ``default`` are provided to -:func:`dataclasses.field`, the ``default`` argument's type should be checked +Because default values are unconditionally converted using ``converter``, if +an argument for ``converter`` is provided alongside either ``default`` or +``default_factory``, the type of the default (the ``default`` argument if +provided, otherwise the return value of ``default_factory``) should be checked using the type of the single argument to the ``converter`` callable. Converter return type @@ -141,23 +162,6 @@ a type that's more specialized (such as a converter returning a ``list[int]`` for a field annotated as ``list``, or a converter returning an ``int`` for a field annotated as ``int | str``). -Example -''''''' - -.. code-block:: python - - @dataclasses.dataclass - class Example: - my_int: int = dataclasses.field(converter=int) - my_tuple: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...]) - my_cheese: Cheese = dataclasses.field(converter=make_cheese) - - # Although the default value is of type `str` and the field is declared to - # be of type `pathlib.Path`, this is not a type error because the default - # value will be converted. - tmpdir: pathlib.Path = dataclasses.field(default="/tmp", converter=pathlib.Path) - - Backward Compatibility ====================== @@ -184,6 +188,8 @@ users of converters are likely to encounter. Such pitfalls include: * Needing to handle values that are already of the correct type. * Avoiding lambdas for converters, as the synthesized ``__init__`` parameter's type will become ``Any``. +* Forgetting to convert values in the bodies of user-defined ``__init__`` and + ``__setattr__``. Reference Implementation ======================== @@ -213,15 +219,18 @@ Not converting default values There are pros and cons with both converting and not converting default values. Leaving default values as-is allows type-checkers and dataclass authors to expect that the type of the default matches the type of the field. However, -converting default values has two large advantages: +converting default values has three large advantages: -1. Compatibility with attrs. Attrs unconditionally uses the converter to - convert the default value. +1. Consistency. Unconditionally converting all values that are assigned to the + attribute, involves fewer "special rules" that users must remember. 2. Simpler defaults. Allowing the default value to have the same type as user-provided values means dataclass authors get the same conveniences as their callers. +2. Compatibility with attrs. Attrs unconditionally uses the converter to + convert the default value. + Automatic conversion using the field's type ------------------------------------------- From f59132836ba2132d4372a94a10e677f51af00ee7 Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 22 May 2023 14:11:47 -0500 Subject: [PATCH 02/10] fix wording --- pep-0712.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index cb76240dfa0..fb75c0eb6bf 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -130,7 +130,7 @@ Example # Attribute assignment also participates in conversion. item1.skus = [555] - # item1's `skus` attribute is now `(555,)`. + # item1's skus attribute is now (555,). Impact on typing @@ -229,7 +229,7 @@ converting default values has three large advantages: their callers. 2. Compatibility with attrs. Attrs unconditionally uses the converter to - convert the default value. + convert default values. Automatic conversion using the field's type ------------------------------------------- From aec18ed8acd6715f816157b7fee421d7bfb6d575 Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 22 May 2023 15:06:58 -0500 Subject: [PATCH 03/10] Tweak the wording --- pep-0712.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index fb75c0eb6bf..65bd30e51d6 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -73,12 +73,11 @@ New ``converter`` parameter This specification introduces a new parameter named ``converter`` to the :func:`dataclasses.field` function. If provided, it represents a single-argument callable used to convert all values when assigning to the associated attribute. -The converted values include: -* The value specified during object construction -* The value of ``default``, when used -* The return value of ``default_factory``, when used -* The value used for attribute assignment on the object +For frozen dataclasses, the converter is only used inside a ``dataclass``-synthesized +``__init__`` when setting the attribute. For non-frozen dataclasses, the converter +is used for all attribute assignment (E.g. ``obj.attr = value``), which includes +assignment to default values. Adding this parameter also implies the following changes: From ca9bbd70da218cb7ae07c28408ac53a6d262ac8a Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 22 May 2023 15:12:06 -0500 Subject: [PATCH 04/10] tweak some wording --- pep-0712.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index 65bd30e51d6..146739f85bd 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -187,7 +187,12 @@ users of converters are likely to encounter. Such pitfalls include: * Needing to handle values that are already of the correct type. * Avoiding lambdas for converters, as the synthesized ``__init__`` parameter's type will become ``Any``. -* Forgetting to convert values in the bodies of user-defined ``__init__`` and +* Forgetting to convert values in the bodies of user-defined ``__init__`` in + frozen dataclasses. +* Forgetting to convert values in the bodies of user-defined ``__setattr__`` in + non-frozen dataclasses. + +and ``__setattr__``. Reference Implementation From cab662d91e3fdd6f201670c1b1f221d11799d329 Mon Sep 17 00:00:00 2001 From: Joshua Date: Wed, 14 Jun 2023 21:04:17 -0500 Subject: [PATCH 05/10] Add section on omitting annotation --- pep-0712.rst | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index 146739f85bd..fa89f6431c4 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -246,7 +246,27 @@ appear to be similar to this approach. This works well for fairly simple types, but leads to ambiguity in expected behavior for complex types such as generics. E.g. For ``tuple[int, ...]`` it is ambiguous if the converter is supposed to simply convert an iterable to a tuple, -or if it is additionally supposed to convert each element type to ``int``. +or if it is additionally supposed to convert each element type to ``int``. Or +for ``int | None``, which isn't callable. + +Deducing the attribute type form the return type of the converter +----------------------------------------------------------------- + +Another idea would be to allow the user to omit the attribute's type annotation +if providing a ``field`` with a ``converter`` argument. Although this would +reduce the common repetition this PEP introduces (e.g. ``x: str = field(converter=str)``), +it isn't clear how to best support this while maintaining the current dataclass +semantics (namely, that the attribute order is preserved for things like the +synthesized ``__init__``, or ``dataclasses.fields``). This is because there isn't +a way in Python (today) to get the annotation-only attributes interspersed with +un-annotated attributes in the order they were defined. + +A sentinel annotation could be applied (e.g. ``x: FromConverter = ...``), +however this breaks a fundamental assumption of type annotations. + +This PEP doesn't suggest it can't or shouldn't be done. Just that it isn't +included in this PEP. + References ========== From 234b3cd2cc9fbab236b27186e4a950b2331447b5 Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 15 Jun 2023 06:49:25 -0500 Subject: [PATCH 06/10] typo --- pep-0712.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index fa89f6431c4..8d165a2063c 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -249,7 +249,7 @@ ambiguous if the converter is supposed to simply convert an iterable to a tuple, or if it is additionally supposed to convert each element type to ``int``. Or for ``int | None``, which isn't callable. -Deducing the attribute type form the return type of the converter +Deducing the attribute type from the return type of the converter ----------------------------------------------------------------- Another idea would be to allow the user to omit the attribute's type annotation From 170eb3618cb61bc5c1bba221e789614fd45f5c6e Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 15 Jun 2023 09:39:37 -0500 Subject: [PATCH 07/10] assignment tidbit --- pep-0712.rst | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 8d165a2063c..b930ce546a8 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -20,7 +20,7 @@ Abstract several common dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django. -A common feature these libraries provide over the standard library +A common feature other libraries provide over the standard library implementation is the ability for the library to convert arguments given at initialization time into the types expected for each field using a user-provided conversion function. @@ -77,7 +77,10 @@ callable used to convert all values when assigning to the associated attribute. For frozen dataclasses, the converter is only used inside a ``dataclass``-synthesized ``__init__`` when setting the attribute. For non-frozen dataclasses, the converter is used for all attribute assignment (E.g. ``obj.attr = value``), which includes -assignment to default values. +assignment of default values. + +The converter is not used when reading attributes, as the attributes should already +have been converted. Adding this parameter also implies the following changes: @@ -232,7 +235,7 @@ converting default values has three large advantages: user-provided values means dataclass authors get the same conveniences as their callers. -2. Compatibility with attrs. Attrs unconditionally uses the converter to +3. Compatibility with attrs. Attrs unconditionally uses the converter to convert default values. Automatic conversion using the field's type @@ -264,6 +267,12 @@ un-annotated attributes in the order they were defined. A sentinel annotation could be applied (e.g. ``x: FromConverter = ...``), however this breaks a fundamental assumption of type annotations. +Lastly, this is feasible if *all* fields (including those without a converter) +were assigned to ``dataclasses.field``, which means the class' own namespace +holds the order, however this trades repetition of type+converter with +repetition of field assignment. The end result is no gain or loss of repetition, +but with the added complexity of dataclasses semantics. + This PEP doesn't suggest it can't or shouldn't be done. Just that it isn't included in this PEP. From 637404801588ef5a146725b15a3c5594d5d63a4b Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 15 Jun 2023 09:43:47 -0500 Subject: [PATCH 08/10] fix example --- pep-0712.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index b930ce546a8..d2d849714d8 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -118,17 +118,19 @@ Example item1 = InventoryItem( "1", - None, [234, 765], + None, ["PYTHON PLUSHIE", "FLUFFY SNAKE"] ) - # item1 would have the following values: - # id=1 - # skus=(234, 765) - # vendor=None - # names=('python plushie', 'fluffy snake') - # stock_image_path=pathlib.PurePosixPath("assets/unknown.png") - # shelves=() + # item1's repr would be (with added newlines for readability): + # InventoryItem( + # id=1, + # skus=(234, 765), + # vendor=None, + # names=('PYTHON PLUSHIE', 'FLUFFY SNAKE'), + # stock_image_path=PurePosixPath('assets/unknown.png'), + # shelves=() + # ) # Attribute assignment also participates in conversion. item1.skus = [555] From f9484753b817c6d4314e6f1e6dc06cc4b5dcccba Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 15 Jun 2023 10:19:04 -0500 Subject: [PATCH 09/10] Add downside section --- pep-0712.rst | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index d2d849714d8..2d76d290448 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -166,6 +166,18 @@ a type that's more specialized (such as a converter returning a ``list[int]`` for a field annotated as ``list``, or a converter returning an ``int`` for a field annotated as ``int | str``). +Indirection of allowable argument types +--------------------------------------- + +One downside introduced by this PEP is that knowing what argument types are +allowed in the dataclass' ``__init__`` and during attribute assignment is not +immediately obvious from reading the dataclass. The allowable types are defined +by the converter. + +This is true when reading code from source, however typing-related aides such +as ``typing.reveal_type`` and "IntelliSense" in an IDE should make it easy to know +exactly what types are allowed without having to read any source code. + Backward Compatibility ====================== @@ -197,9 +209,6 @@ users of converters are likely to encounter. Such pitfalls include: * Forgetting to convert values in the bodies of user-defined ``__setattr__`` in non-frozen dataclasses. -and - ``__setattr__``. - Reference Implementation ======================== From fd9697c6fd0dad4f992bc4ff4deaf4204a5fb4cc Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 15 Jun 2023 13:24:40 -0500 Subject: [PATCH 10/10] it ain't easy --- pep-0712.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 2d76d290448..cbf9f51b5c5 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -272,8 +272,8 @@ reduce the common repetition this PEP introduces (e.g. ``x: str = field(converte it isn't clear how to best support this while maintaining the current dataclass semantics (namely, that the attribute order is preserved for things like the synthesized ``__init__``, or ``dataclasses.fields``). This is because there isn't -a way in Python (today) to get the annotation-only attributes interspersed with -un-annotated attributes in the order they were defined. +an easy way in Python (today) to get the annotation-only attributes interspersed +with un-annotated attributes in the order they were defined. A sentinel annotation could be applied (e.g. ``x: FromConverter = ...``), however this breaks a fundamental assumption of type annotations.