Skip to content

Commit

Permalink
PEP 604-inconsistent type hint protection.
Browse files Browse the repository at this point in the history
This commit almost entirely reverts the prior commit -- which, on second
thought, was *not* entirely well-thought out. Now, @beartype explicitly
prohibits all **PEP 604-inconsistent type hints** (i.e., object
permissible as an item of a PEP 604-compliant new union whose
machine-readable representation is *not* the machine-readable
representation of this hint in new unions). PEP 604-inconsistent type
hints induce non-deterministic behaviour in @beartype and *must* thus be
explicitly detected and prohibited. This includes those produced by the
third-party `nptyping` package, for which @beartype now raises
cautionary PEP 604-specific exceptions resembling:

```
beartype.roar.BeartypeDecorHintPep604Exception: Type hint
NDArray[Shape['N, N'], Float] inconsistent with respect to repr()
strings. Since @beartype requires consistency between type hints and
repr() strings, this hint is unsupported by @beartype. Consider
reporting this issue to the third-party developer implementing this
hint: e.g.,
	>>> repr(NDArray[Shape['N, N'], Float])
	NDArray[Shape['N, N'], Float]  # <-- this is fine
	>>> repr(NDArray[Shape['N, N'], Float] | int)
	nptyping.ndarray.NDArray | int  # <-- *THIS IS REALLY SUPER BAD*

	# Ideally, that output should instead resemble:
	>>> repr(NDArray[Shape['N, N'], Float] | int)
	NDArray[Shape['N, N'], Float] | int  # <-- what @beartype wants!
```

(*Descrying inscriptions on inscrutable lutes!*)
  • Loading branch information
leycec committed Nov 16, 2023
1 parent e5a3915 commit a469322
Show file tree
Hide file tree
Showing 11 changed files with 354 additions and 194 deletions.
30 changes: 22 additions & 8 deletions beartype/_cave/_cavefast.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,10 @@
EnumMeta as _EnumMeta,
)
from io import IOBase as _IOBase
from typing import Any
from typing import (
Any,
Tuple as _TupleTyping,
)

# Note that:
#
Expand Down Expand Up @@ -910,7 +913,7 @@ class or instance attributes).

# ....................{ TYPES ~ hint }....................
# Define this type as either...
HintGenericSubscriptedType: Any = (
HintGenericSubscriptedType: type = (
# If the active Python interpreter targets at least Python >= 3.9 and thus
# supports PEP 585, this type;
type(list[str]) # type: ignore[misc]
Expand All @@ -920,18 +923,17 @@ class or instance attributes).
)
'''
C-based type of all subscripted generics if the active Python interpreter
targets Python >= 3.9 *or* :class:`UnavailableType` otherwise.
targets Python >= 3.9 *or* :class:`.UnavailableType` otherwise.
Subscripted generics include:
This type is a version-agnostic generalization of the standard
:class:`types.GenericAlias` type available only under Python >= 3.9. Subscripted
generics include:
* :pep:`585`-compliant **builtin type hints** (i.e., C-based type hints
instantiated by subscripting either a concrete builtin container class like
:class:`list` or :class:`tuple` *or* an abstract base class (ABC) declared by
the :mod:`collections.abc` submodule like :class:`collections.abc.Iterable`
or :class:`collections.abc.Sequence`). Since *all* :pep:`585`-compliant
builtin type hints are classes, this C-based type is the class of those
classes and thus effectively itself a metaclass. It's probably best not to
think about that.
or :class:`collections.abc.Sequence`).
* :pep:`484`-compliant **subscripted generics** (i.e., user-defined classes
subclassing one or more :pep:`484`-compliant type hints subsequently
subscripted by one or more PEP-compliant type hints).
Expand All @@ -955,6 +957,18 @@ class or instance attributes).
detecting :pep:`585`-compliant generic type hints.
'''


HintPep604Types: _TupleTyping[type, ...] = (type, HintGenericSubscriptedType)
'''
Tuple of all :pep:`604`-compliant **new union item types** (i.e., types of all
objects permissible as the items of new unions), including:
* The C-based type of all types (e.g., the type of the first item in the new
union ``list | None``).
* The C-based type of all subscripted generics (e.g., the type of the first item
in the new union ``list[dict[str, int]] | None``).
'''

# ....................{ TYPES ~ scalar }....................
StrType = str # Well, isn't that special.
'''
Expand Down
70 changes: 2 additions & 68 deletions beartype/_check/convert/convcoerce.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,11 @@
from beartype._data.func.datafuncarg import ARG_NAME_RETURN
from beartype._data.func.datafunc import METHOD_NAMES_DUNDER_BINARY
from beartype._check.checkcall import BeartypeCall
from beartype._check.forward.fwdhint import resolve_hint
from beartype._check.forward.fwdmain import resolve_hint
from beartype._util.cache.map.utilmapbig import CacheUnboundedStrong
from beartype._util.hint.utilhinttest import is_hint_uncached
from beartype._util.hint.pep.utilpepget import get_hint_pep_args
from beartype._util.hint.pep.proposal.pep484.utilpep484union import (
make_hint_pep484_union)
from beartype._util.hint.pep.proposal.utilpep604 import is_hint_pep604

# ....................{ COERCERS ~ root }....................
#FIXME: Document mypy-specific coercion in the docstring as well, please.
Expand Down Expand Up @@ -392,70 +390,6 @@ def coerce_hint_any(hint: object) -> Any:
'''

# ..................{ NON-SELF-CACHING }..................
# If this hint is PEP 604-compliant new union (e.g., "int | str"), this hint
# is *NOT* self-caching (e.g., "int | str is not int | str") and *MUST* thus
# be explicitly cached here.
#
# Ideally, new unions would *NOT* require explicit caching here but could
# instead simply be implicitly cached below by the existing "elif
# is_hint_uncached(hint):" conditional. Indeed, that is exactly how new
# unions were once cached. Tragically, that approach no longer suffices.
# Why? Because poorly implemented third-party packages like "nptyping"
# dynamically generate in-memory classes that all share the same
# fully-qualified names despite being fundamentally different classes: e.g.,
# $ python3.12
# >>> from nptyping import Float64, NDArray, Shape
# >>> foo = NDArray[Shape["N, N"], Float64]
# >>> bar = NDArray[Shape["*"], Float64]
#
# >>> foo == bar
# False # <-- this is sane
# >>> foo.__name__
# NDArray # <-- this is insane
# >>> bar.__name__
# NDArray # <-- still insane after all these years
#
# >>> foo | None == bar | None
# False # <-- this is sane
# >>> repr(foo | None)
# NDArray | None # <-- big yikes
# >>> repr(bar | None)
# NDArray | None # <-- yikes intensifies
#
# Alternately, it could be argued that the implementation of the
# types.UnionType.__repr__() dunder method underlying the repr() strings
# printed above is at fault. Ideally, that method should embed the repr()
# strings of its child hints in the string it returns; instead, that method
# merely embeds the classnames of its child hints in the string it returns.
# Since the implementation of this method is unlikely to improve, however,
# the burden remains on third-party authors to correctly name classes.
#
# In either case, reducing new unions on the overly simplistic basis of
# their repr() strings fails in the general case of poorly implemented
# third-party packages that violate Python standards.
#
# Instead, this new approach transforms PEP 604-compliant new unions to
# equivalent PEP 484-compliant old unions (e.g., from "int | str" to
# "typing.Union[int, str]"). While substantially slower than the old
# approach, the new approach is substantially more resilient against bad
# behaviour in third-party packages. Ultimately, worky >>>>> speed.
if is_hint_pep604(hint):
# Tuple of the two or more child type hints subscripting this new union.
hint_args = get_hint_pep_args(hint)

# Reduce this hint the equivalent PEP 484-compliant old union. Why?
# Because old unions implicitly self-cache, whereas new unions do *NOT*:
# >>> int | str is int | str
# False
#
# >>> from typing import Union
# >>> Union[int, str] is Union[int, str]
# True
#
# Note that this factory function is memoized and thus optimized.
hint = make_hint_pep484_union(hint_args)
# Else, this hint is *NOT* a PEP 604-compliant new union.
#
# If this hint is *NOT* self-caching, this hint *MUST* thus be explicitly
# cached here. Failing to do so would disable subsequent memoization,
# reducing decoration- and call-time efficiency when decorating callables
Expand All @@ -467,7 +401,7 @@ def coerce_hint_any(hint: object) -> Any:
# * Else, one or more prior copies of this hint have already been passed to
# this function. In this case, replace this subsequent copy by the first
# copy of this hint originally passed to a prior call of this function.
elif is_hint_uncached(hint):
if is_hint_uncached(hint):
# print(f'Self-caching type hint {repr(hint)}...')
return _HINT_REPR_TO_SINGLETON.cache_or_get_cached_value(
key=repr(hint), value=hint)
Expand Down
76 changes: 38 additions & 38 deletions beartype/_check/convert/convreduce.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,21 +157,25 @@ class variable or method annotated by this hint *or* :data:`None`).
return hint

# ....................{ PRIVATE ~ reducers }....................
@callable_cached
def _reduce_hint_cached(
def _reduce_hint_uncached(
hint: Any,
conf: BeartypeConf,
cls_stack: TypeStack,
arg_name: Optional[str],
exception_prefix: str,
) -> object:
'''
Lower-level **context-free type hint** (i.e., type hint *not* contextually
dependent on the kind of class, attribute, callable parameter, or callable
return annotated by this hint) efficiently reduced (i.e., converted) from
the passed higher-level context-free type hint if this hint is reducible
*or* this hint as is otherwise (i.e., if this hint is irreducible).
Lower-level **contextual type hint** (i.e., type hint contextually dependent
on the kind of class, attribute, callable parameter, or callable return
annotated by this hint) inefficiently reduced (i.e., converted) from the
passed higher-level context-free type hint if this hint is reducible *or*
this hint as is otherwise (i.e., if this hint is irreducible).
This reducer is memoized for efficiency. Thankfully, this reducer is
responsible for reducing *most* (but not all) type hints.
This reducer *cannot* be meaningfully memoized, since multiple passed
parameters (e.g., ``arg_name``, ``cls_stack``) are typically isolated to a
handful of callables across the codebase currently being decorated by
:mod:`beartype`. Thankfully, this reducer is responsible for reducing only a
small subset of type hints requiring these problematic parameters.
Parameters
----------
Expand All @@ -180,6 +184,16 @@ def _reduce_hint_cached(
conf : BeartypeConf
**Beartype configuration** (i.e., self-caching dataclass encapsulating
all settings configuring type-checking for the passed object).
cls_stack : TypeStack
**Type stack** (i.e., either tuple of zero or more arbitrary types *or*
:data:`None`). See also the :func:`.beartype_object` decorator.
arg_name : Optional[str]
Either:
* If this hint annotates a parameter of some callable, the name of that
parameter.
* If this hint annotates the return of some callable, ``"return"``.
* Else, :data:`None`.
exception_prefix : str
Substring prefixing exception messages raised by this function.
Expand All @@ -199,41 +213,41 @@ def _reduce_hint_cached(
# Callable reducing this hint if a callable reducing hints of this sign was
# previously registered *OR* "None" otherwise (i.e., if *NO* such callable
# was registered, in which case this hint is preserved as is).
hint_reducer = _HINT_SIGN_TO_REDUCE_HINT_CACHED.get(hint_sign)
hint_reducer = _HINT_SIGN_TO_REDUCE_HINT_UNCACHED.get(hint_sign)

# If a callable reducing hints of this sign was previously registered,
# reduce this hint to another hint via this callable.
if hint_reducer is not None:
# print(f'Reducing hint {repr(hint)} to...')
hint = hint_reducer( # type: ignore[call-arg]
hint=hint, # pyright: ignore[reportGeneralTypeIssues]
conf=conf,
cls_stack=cls_stack,
arg_name=arg_name,
exception_prefix=exception_prefix,
)
# print(f'...{repr(hint)}.')
# Else, *NO* such callable was registered. Preserve this hint as is, you!

# Return this possibly reduced hint.
return hint


def _reduce_hint_uncached(
@callable_cached
def _reduce_hint_cached(
hint: Any,
conf: BeartypeConf,
cls_stack: TypeStack,
arg_name: Optional[str],
exception_prefix: str,
) -> object:
'''
Lower-level **contextual type hint** (i.e., type hint contextually dependent
on the kind of class, attribute, callable parameter, or callable return
annotated by this hint) inefficiently reduced (i.e., converted) from the
passed higher-level context-free type hint if this hint is reducible *or*
this hint as is otherwise (i.e., if this hint is irreducible).
Lower-level **context-free type hint** (i.e., type hint *not* contextually
dependent on the kind of class, attribute, callable parameter, or callable
return annotated by this hint) efficiently reduced (i.e., converted) from
the passed higher-level context-free type hint if this hint is reducible
*or* this hint as is otherwise (i.e., if this hint is irreducible).
This reducer *cannot* be meaningfully memoized, since multiple passed
parameters (e.g., ``arg_name``, ``cls_stack``) are typically isolated to a
handful of callables across the codebase currently being decorated by
:mod:`beartype`. Thankfully, this reducer is responsible for reducing only a
small subset of type hints requiring these problematic parameters.
This reducer is memoized for efficiency. Thankfully, this reducer is
responsible for reducing *most* (but not all) type hints.
Parameters
----------
Expand All @@ -242,16 +256,6 @@ def _reduce_hint_uncached(
conf : BeartypeConf
**Beartype configuration** (i.e., self-caching dataclass encapsulating
all settings configuring type-checking for the passed object).
cls_stack : TypeStack
**Type stack** (i.e., either tuple of zero or more arbitrary types *or*
:data:`None`). See also the :func:`.beartype_object` decorator.
arg_name : Optional[str]
Either:
* If this hint annotates a parameter of some callable, the name of that
parameter.
* If this hint annotates the return of some callable, ``"return"``.
* Else, :data:`None`.
exception_prefix : str
Substring prefixing exception messages raised by this function.
Expand All @@ -271,20 +275,16 @@ def _reduce_hint_uncached(
# Callable reducing this hint if a callable reducing hints of this sign was
# previously registered *OR* "None" otherwise (i.e., if *NO* such callable
# was registered, in which case this hint is preserved as is).
hint_reducer = _HINT_SIGN_TO_REDUCE_HINT_UNCACHED.get(hint_sign)
hint_reducer = _HINT_SIGN_TO_REDUCE_HINT_CACHED.get(hint_sign)

# If a callable reducing hints of this sign was previously registered,
# reduce this hint to another hint via this callable.
if hint_reducer is not None:
# print(f'Reducing hint {repr(hint)} to...')
hint = hint_reducer( # type: ignore[call-arg]
hint=hint, # pyright: ignore[reportGeneralTypeIssues]
conf=conf,
cls_stack=cls_stack,
arg_name=arg_name,
exception_prefix=exception_prefix,
)
# print(f'...{repr(hint)}.')
# Else, *NO* such callable was registered. Preserve this hint as is, you!

# Return this possibly reduced hint.
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion beartype/_check/forward/fwdscope.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def __init__(self, scope_dict: LexicalScope, scope_name: str) -> None:
provide both. Why? Because this forward scope is principally
intended to be passed as the second and last parameter to the
:func:`eval` builtin, called by the
:func:`beartype._check.forward.fwdhint.resolve_hint` function. For
:func:`beartype._check.forward.fwdmain.resolve_hint` function. For
unknown reasons, :func:`eval` only calls the :meth:`__missing__`
dunder method of this forward scope when passed only two parameters
(i.e., when passed only a global scope); :func:`eval` does *not*
Expand Down
17 changes: 9 additions & 8 deletions beartype/_util/hint/pep/proposal/utilpep585.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ def is_hint_pep585_generic(hint: object) -> bool:
# ....................{ TESTERS ~ doc }....................
# Docstring for this function regardless of implementation details.
is_hint_pep585_builtin.__doc__ = '''
``True`` only if the passed object is a C-based :pep:`585`-compliant
:data:`True` only if the passed object is a C-based :pep:`585`-compliant
**builtin type hint** (i.e., C-based type hint instantiated by subscripting
either a concrete builtin container class like :class:`list` or
:class:`tuple` *or* an abstract base class (ABC) declared by the
Expand All @@ -177,10 +177,11 @@ def is_hint_pep585_generic(hint: object) -> bool:
----------
**This test returns false for** :pep:`585`-compliant **generics,** which
fail to satisfy the same API as all other :pep:`585`-compliant type hints.
Why? Because :pep:`560`-type erasure erases this API on :pep:`585`-compliant
generics immediately after those generics are declared, preventing their
subsequent detection as :pep:`585`-compliant. Instead, :pep:`585`-compliant
generics are only detectable by calling either:
Why? Because :pep:`560`-type erasure erases the low-level superclass
detected by this tester on :pep:`585`-compliant generics immediately after
those generics are declared, preventing their subsequent detection as
:pep:`585`-compliant. Instead, :pep:`585`-compliant generics are only
detectable by calling either:
* The high-level PEP-agnostic
:func:`beartype._util.hint.pep.utilpeptest.is_hint_pep484585_generic`
Expand All @@ -195,12 +196,12 @@ def is_hint_pep585_generic(hint: object) -> bool:
Returns
----------
bool
``True`` only if this object is a :pep:`585`-compliant type hint.
:data:`True` only if this object is a :pep:`585`-compliant type hint.
'''


is_hint_pep585_generic.__doc__ = '''
``True`` only if the passed object is a :pep:`585`-compliant **generic**
:data:`True` only if the passed object is a :pep:`585`-compliant **generic**
(i.e., object that may *not* actually be a class originally subclassing at
least one subscripted :pep:`585`-compliant pseudo-superclass).
Expand All @@ -214,7 +215,7 @@ def is_hint_pep585_generic(hint: object) -> bool:
Returns
----------
bool
``True`` only if this object is a :pep:`585`-compliant generic.
:data:`True` only if this object is a :pep:`585`-compliant generic.
'''

# ....................{ GETTERS }....................
Expand Down

0 comments on commit a469322

Please sign in to comment.