Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

typing.NewType isn't supported #94

Closed
rhofour opened this issue Sep 20, 2020 · 8 comments
Closed

typing.NewType isn't supported #94

rhofour opened this issue Sep 20, 2020 · 8 comments

Comments

@rhofour
Copy link

rhofour commented Sep 20, 2020

  • cattrs version: 1.0.0
  • Python version: 3.8.5
  • Operating System: Fedora 32

Description

NewType seems to break structuring / destructuring.

Here's a minimal test case:

import unittest
from typing import NewType
import attr
import cattr

ConfigId = NewType('ConfigId', int)

@attr.s(auto_attribs=True)
class ClassWithNewTypeMember:
    configId: ConfigId

class TestAttrs(unittest.TestCase):
    def test_newType(self):

        event = ClassWithNewTypeMember(configId = ConfigId(0))
        unstructured = cattr.unstructure(event)
        structured = cattr.structure(unstructured, ClassWithNewTypeMember)

I would expect this test to pass, but instead I get the following:
E ValueError: Unsupported type: <function NewType.<locals>.new_type at 0x7fa0349398b0>. Register a structure hook for it.

The workaround is to manually register a structure hook like so:
cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

A destructuring hook isn't needed.

@letalvoj
Copy link

letalvoj commented May 19, 2021

According to the documentation https://docs.python.org/3/library/typing.html#newtype

The static type checker will treat the new type as if it were a subclass of the original type. This is useful in helping catch logical errors:

def get_user_name(user_id: UserId) -> str:
    ...

# typechecks
user_a = get_user_name(UserId(42351))

# does not typecheck; an int is not a UserId
user_b = get_user_name(-1)

Hence, as ConfigId is more specific than int, cattrs should not know how to derive an encoder for it, right?

@Tinche
Copy link
Member

Tinche commented May 19, 2021

We probably just need to teach cattrs to treat newtypes as their underlying types. I agree this would be useful.

@raabf
Copy link
Contributor

raabf commented Feb 4, 2022

The following can be also a workaround since it eliminates the NewType alias at runtime (although I personally prefer also to register a un-/structure hook for ID):

from typing import NewType, TYPE_CHECKING
ID = NewType('ID', str)

if not TYPE_CHECKING:
    ID = ID.__supertype__

@uskudnik
Copy link

uskudnik commented Feb 7, 2022

(Commenting here since #173 is closed)

@Tinche Unfortunately manually registering a hook for NewTypes doesn't work any more, based on my tracking down the culprit is the same (traceback below).

It seems to boil down to failing isinstance(ConfigId, type):

In [6]: isinstance(int, type)
Out[6]: True

In [7]: isinstance(ConfigId, type)
Out[7]: False

from typing import NewType
import cattr
 
ConfigId = NewType("ConfigId", int)
cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-3bdf89a2fc7e> in <module>
      3 
      4 ConfigId = NewType("ConfigId", int)
----> 5 cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

/usr/local/lib/python3.10/site-packages/cattr/converters.py in register_structure_hook(self, cl, func)
    267             self._structure_func.clear_cache()
    268         else:
--> 269             self._structure_func.register_cls_list([(cl, func)])
    270 
    271     def register_structure_hook_func(

/usr/local/lib/python3.10/site-packages/cattr/dispatch.py in register_cls_list(self, cls_and_handler, direct)
     55                 self._direct_dispatch[cls] = handler
     56             else:
---> 57                 self._single_dispatch.register(cls, handler)
     58                 self.clear_direct()
     59         self.dispatch.cache_clear()

/usr/local/lib/python3.10/functools.py in register(cls, func)
    854         else:
    855             if func is not None:
--> 856                 raise TypeError(
    857                     f"Invalid first argument to `register()`. "
    858                     f"{cls!r} is not a class."

TypeError: Invalid first argument to `register()`. __main__.ConfigId is not a class.

@uskudnik
Copy link

uskudnik commented Feb 7, 2022

Okay, might've found a solution by using register_structure_hook_func

import cattr
from typing import NewType

cattr.register_structure_hook_func(
    lambda t: t.__class__ is NewType, lambda d, t: t.__call__(d)
)

🤞 nothing 💥 too much 🙂

@raabf
Copy link
Contributor

raabf commented Feb 7, 2022

Before python 3.10, typing.NewType is a function which returns a function, hence isinstance does not work. In python 3.10 the symbol typing.NewType became an own class where you can do normal isinstance{obj, typing.NewType) checks (or as you noticed t.__class__ is NewType which does the same).

Something like isinstance(ConfigId, type) should not work on python 3.10 because ConfigId is then an instance of the typing.NewType class instead of an own generated function.

Heve a look at this is_new_type() function for a complete test regardless of python version.

@henryiii
Copy link
Contributor

henryiii commented Mar 11, 2022

I'm looking for a way to register a custom converter via a type; that is, I'd like to do a little post processing on fields marked with URL, but I don't need a custom type for the URL in the classes, just str is fine. I could define a field for each URL, but doing it via types is simpler and cleaner, and I don't really need a converter for setting an instance (since they are frozen, no real difference). It seems like NewType would have been a nice way to do it.

From what I can tell above (which is literally asking for the opposite of what I want to do - though I'm fine if it also works by default but is customizable), what I want was supported before 3.10 (though possibly by accident and not fully), but now it's broken?

@Tinche
Copy link
Member

Tinche commented May 5, 2022

NewTypes are in, and will be released with the 22.2.0 release!

@Tinche Tinche closed this as completed May 5, 2022
mergify bot pushed a commit to aws/jsii that referenced this issue Oct 3, 2022
…3 in /packages/@jsii/python-runtime (#3785)

Updates the requirements on [cattrs](https://github.com/python-attrs/cattrs) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/python-attrs/cattrs/blob/main/HISTORY.rst">cattrs's changelog</a>.</em></p>
<blockquote>
<h2>22.2.0 (2022-10-03)</h2>
<ul>
<li><em>Potentially breaking</em>: <code>cattrs.Converter</code> has been renamed to <code>cattrs.BaseConverter</code>, and <code>cattrs.GenConverter</code> to <code>cattrs.Converter</code>.
The <code>GenConverter</code> name is still available for backwards compatibility, but is deprecated.
If you were depending on functionality specific to the old <code>Converter</code>, change your import to <code>from cattrs import BaseConverter</code>.</li>
<li><code>NewTypes &lt;https://docs.python.org/3/library/typing.html#newtype&gt;</code>_ are now supported by the <code>cattrs.Converter</code>.
(<code>[#255](python-attrs/cattrs#255) &lt;https://github.com/python-attrs/cattrs/pull/255&gt;</code><em>, <code>[#94](python-attrs/cattrs#94) &lt;https://github.com/python-attrs/cattrs/issues/94&gt;</code></em>, <code>[#297](python-attrs/cattrs#297) &lt;https://github.com/python-attrs/cattrs/issues/297&gt;</code>_)</li>
<li><code>cattrs.Converter</code> and <code>cattrs.BaseConverter</code> can now copy themselves using the <code>copy</code> method.
(<code>[#284](python-attrs/cattrs#284) &lt;https://github.com/python-attrs/cattrs/pull/284&gt;</code>_)</li>
<li>Python 3.11 support.</li>
<li>cattrs now supports un/structuring <code>kw_only</code> fields on attrs classes into/from dictionaries.
(<code>[#247](python-attrs/cattrs#247) &lt;https://github.com/python-attrs/cattrs/pull/247&gt;</code>_)</li>
<li>PyPy support (and tests, using a minimal Hypothesis profile) restored.
(<code>[#253](python-attrs/cattrs#253) &lt;https://github.com/python-attrs/cattrs/issues/253&gt;</code>_)</li>
<li>Fix propagating the <code>detailed_validation</code> flag to mapping and counter structuring generators.</li>
<li>Fix <code>typing.Set</code> applying too broadly when used with the <code>GenConverter.unstruct_collection_overrides</code> parameter on Python versions below 3.9. Switch to <code>typing.AbstractSet</code> on those versions to restore the old behavior.
(<code>[#264](python-attrs/cattrs#264) &lt;https://github.com/python-attrs/cattrs/issues/264&gt;</code>_)</li>
<li>Uncap the required Python version, to avoid problems detailed in <a href="https://iscinumpy.dev/post/bound-version-constraints/#pinning-the-python-version-is-special">https://iscinumpy.dev/post/bound-version-constraints/#pinning-the-python-version-is-special</a>
(<code>[#275](python-attrs/cattrs#275) &lt;https://github.com/python-attrs/cattrs/issues/275&gt;</code>_)</li>
<li>Fix <code>Converter.register_structure_hook_factory</code> and <code>cattrs.gen.make_dict_unstructure_fn</code> type annotations.
(<code>[#281](python-attrs/cattrs#281) &lt;https://github.com/python-attrs/cattrs/issues/281&gt;</code>_)</li>
<li>Expose all error classes in the <code>cattr.errors</code> namespace. Note that it is deprecated, just use <code>cattrs.errors</code>.
(<code>[#252](python-attrs/cattrs#252) &lt;https://github.com/python-attrs/cattrs/issues/252&gt;</code>_)</li>
<li>Fix generating structuring functions for types with quotes in the name.
(<code>[#291](python-attrs/cattrs#291) &lt;https://github.com/python-attrs/cattrs/issues/291&gt;</code>_ <code>[#277](python-attrs/cattrs#277) &lt;https://github.com/python-attrs/cattrs/issues/277&gt;</code>_)</li>
<li>Fix usage of notes for the final version of <code>PEP 678 &lt;https://peps.python.org/pep-0678/&gt;</code><em>, supported since <code>exceptiongroup&gt;=1.0.0rc4</code>.
(<code>[#303](python-attrs/cattrs#303) &lt;303 &lt;https://github.com/python-attrs/cattrs/pull/303&gt;</code></em>)</li>
</ul>
<h2>22.1.0 (2022-04-03)</h2>
<ul>
<li>cattrs now uses the CalVer versioning convention.</li>
<li>cattrs now has a detailed validation mode, which is enabled by default. Learn more <code>here &lt;https://cattrs.readthedocs.io/en/latest/validation.html&gt;</code>_.
The old behavior can be restored by creating the converter with <code>detailed_validation=False</code>.</li>
<li><code>attrs</code> and dataclass structuring is now ~25% faster.</li>
<li>Fix an issue structuring bare <code>typing.List</code> s on Pythons lower than 3.9.
(<code>[#209](python-attrs/cattrs#209) &lt;https://github.com/python-attrs/cattrs/issues/209&gt;</code>_)</li>
<li>Fix structuring of non-parametrized containers like <code>list/dict/...</code> on Pythons lower than 3.9.
(<code>[#218](python-attrs/cattrs#218) &lt;https://github.com/python-attrs/cattrs/issues/218&gt;</code>_)</li>
<li>Fix structuring bare <code>typing.Tuple</code> on Pythons lower than 3.9.
(<code>[#218](python-attrs/cattrs#218) &lt;https://github.com/python-attrs/cattrs/issues/218&gt;</code>_)</li>
<li>Fix a wrong <code>AttributeError</code> of an missing <code>__parameters__</code> attribute. This could happen
when inheriting certain generic classes – for example <code>typing.*</code> classes are affected.
(<code>[#217](python-attrs/cattrs#217) &lt;https://github.com/python-attrs/cattrs/issues/217&gt;</code>_)</li>
<li>Fix structuring of <code>enum.Enum</code> instances in <code>typing.Literal</code> types.
(<code>[#231](python-attrs/cattrs#231) &lt;https://github.com/python-attrs/cattrs/pull/231&gt;</code>_)</li>
<li>Fix unstructuring all tuples - unannotated, variable-length, homogenous and heterogenous - to <code>list</code>.
(<code>[#226](python-attrs/cattrs#226) &lt;https://github.com/python-attrs/cattrs/issues/226&gt;</code>_)</li>
<li>For <code>forbid_extra_keys</code> raise custom <code>ForbiddenExtraKeyError</code> instead of generic <code>Exception</code>.
(<code>[#225](python-attrs/cattrs#225) &lt;https://github.com/python-attrs/cattrs/pull/225&gt;</code>_)</li>
<li>All preconf converters now support <code>loads</code> and <code>dumps</code> directly. See an example <code>here &lt;https://cattrs.readthedocs.io/en/latest/preconf.html&gt;</code>_.</li>
</ul>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/python-attrs/cattrs/commit/405f0291b958ae9eb45ee38febeb91fb65dd644f"><code>405f029</code></a> v22.2.0</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/89de04f57aa774d6abfb0ae62517dc8a8064e3c2"><code>89de04f</code></a> Fix some mor</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/0abbf271461babca203862f3581c20743f6118e0"><code>0abbf27</code></a> Fix tests</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/3b750439aec826a8fd20976ca113f30a27e75408"><code>3b75043</code></a> <strong>notes</strong> is list[str]</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/906b95cfef4903e2d6247abf0fdd72f7da21617a"><code>906b95c</code></a> Reorder HISTORY</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/ed7f86a0ccd9adeab895b9afac2dea69fa02bcff"><code>ed7f86a</code></a> Improve NewTypes (<a href="https://github-redirect.dependabot.com/python-attrs/cattrs/issues/310">#310</a>)</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/e7926599ad44e07d8325ae4072626e1a24705542"><code>e792659</code></a> Fix missing imports of preconf converters (<a href="https://github-redirect.dependabot.com/python-attrs/cattrs/issues/309">#309</a>)</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/e425d6378aa8dfc74bbdd9e152365e1b962fd8cf"><code>e425d63</code></a> Reorder HISTORY</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/cc56b2b873852a4e91b16706a39da00755c87759"><code>cc56b2b</code></a> Remove spurious type comment</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/cbd6f29d2c0ebc40805b3ca0d81accfc330eb10c"><code>cbd6f29</code></a> Reformat</li>
<li>Additional commits viewable in <a href="https://github.com/python-attrs/cattrs/compare/v1.8.0...v22.2.0">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants