from_type fails on nested annotated type with mysterious error

[td-anne]

I have the following code:

FiniteFloat = Annotated[float, IsFinite]


@given(st.from_type(FiniteFloat))
def test_annotated_type(thing):
    assert thing == thing

The result of running this test is:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

annotated_type = typing.Annotated[float, typing.Annotated[~_NumericType, Predicate(func=<built-in function isfinite>)]]

    def find_annotated_strategy(annotated_type):
        metadata = getattr(annotated_type, "__metadata__", ())
    
        if any(is_annotated_type(arg) for arg in metadata):
            # We are in the case where one of the metadata argument
            # is itself an annotated type. Although supported at runtime,
            # This shouldn't be allowed: we prefer to raise here
>           raise ResolutionFailed(
                f"Failed to resolve strategy for the following Annotated type: {annotated_type}."
                "Arguments to the Annotated type cannot be Annotated."
            )
E           hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[float, typing.Annotated[~_NumericType, Predicate(func=<built-in function isfinite>)]].Arguments to the Annotated type cannot be Annotated.

../../Library/Caches/pypoetry/virtualenvs/da-dropper-wire-data-l5CQQYIb-py3.11/lib/python3.11/site-packages/hypothesis/strategies/_internal/types.py:310: ResolutionFailed

Presumably I have annotated this type incorrectly, but I can't tell how. There is no obvious nesting in the way I wrote this. Can hypothesis clarify what is actually wrong here? Or is this an issue for annotated-types (annotated-types/annotated-types#61)?

The Python documentation at https://docs.python.org/3/library/typing.html#typing.Annotated appears to say that annotations that are nested should be automatically flattened, but this would be nesting on the first argument; I don't even understand how nesting on the second element arises.

The hypothesis test suite does not include annotated float types or the IsFinite condition, but the format appears to match what is used for integers here:

hypothesis/hypothesis-python/tests/test_annotated_types.py

Line 67 in 21fd917

@pytest.mark.parametrize(

[td-anne]

As an additional data point, the following works as expected:

FiniteFloat = Annotated[float, Predicate(math.isfinite)]


@given(thing=st.from_type(FiniteFloat))
def test_annotated_type(thing):
    assert thing == thing

This is peculiar as annotated-types claims that IsFinite is defined to be Predicate(math.isfinite).

[JonathanPlasse]

Annotated[float, IsFinite] is not a type, if st.from_type accepts annotation like this its type signature should be updated.
Here is a similar issue raised on Pyright microsoft/pyright#7238.

from typing import Annotated

assert isinstance(float, type)
assert not isinstance(Annotated[float, "some annotation"], type)

[td-anne]

hypothesis/hypothesis-python/tests/test_annotated_types.py

Line 78 in 21fd917

def test_annotated_type_int(annotated_type, expected_strategy_repr):

This test suggests that this is the intended use of from_type; perhaps the name is simply inaccurate?

[td-anne]

In fact, it looks like the problem is that annotated-types tries to express a constraint on the kinds of type that can be annotated with IsFinite and hypothesis can't cope with this:

_NumericType = TypeVar('_NumericType', bound=Union[SupportsFloat, SupportsIndex])
IsFinite = Annotated[_NumericType, Predicate(math.isfinite)]

annotated-types also does this for its string predicates (e.g. IsLower), but hypothesis contains no tests of these either. And indeed these fail too:

AsciiString = Annotated[str, IsAscii]


@given(thing=st.from_type(AsciiString))
def test_annotated_type(thing):
    assert thing == thing

[td-anne]

It looks like the intended usage is as IsAscii[str] (https://github.com/annotated-types/annotated-types/blob/main/annotated_types/test_cases.py#L125) but this fails with the same error:

AsciiString = Annotated[str, IsAscii[str]]


@given(thing=st.from_type(AsciiString))
def test_annotated_type(thing):
    assert thing == thing

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

annotated_type = typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]]

    def find_annotated_strategy(annotated_type):
        metadata = getattr(annotated_type, "__metadata__", ())
    
        if any(is_annotated_type(arg) for arg in metadata):
            # We are in the case where one of the metadata argument
            # is itself an annotated type. Although supported at runtime,
            # This shouldn't be allowed: we prefer to raise here
>           raise ResolutionFailed(
                f"Failed to resolve strategy for the following Annotated type: {annotated_type}."
                "Arguments to the Annotated type cannot be Annotated."
            )
E           hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]].Arguments to the Annotated type cannot be Annotated.

../../Library/Caches/pypoetry/virtualenvs/da-dropper-wire-data-l5CQQYIb-py3.11/lib/python3.11/site-packages/hypothesis/strategies/_internal/types.py:310: ResolutionFailed

[td-anne]

Aha! It seems things like IsFinite are not meant to be used in annotations at all, they are meant to be (parametrized) types on their own: annotated-types/annotated-types#61 (comment)

So maybe hypothesis is doing everything right. It would be nice to have tests of these in the test suite, though.

[Zac-HD]

I think the core problem here is that the error message is confusing, and we should improve it! e.g. you saw

• hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[float, typing.Annotated[~_NumericType, Predicate(func=<built-in function isfinite>)]].Arguments to the Annotated type cannot be Annotated.
• hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]].Arguments to the Annotated type cannot be Annotated.

but "Arguments to the Annotated type cannot be Annotated" only helps if you already know the problem. Some ideas for how to improve this, in both Hypothesis and annotated-types:

• Fix the incorrect annotated-types README; IsFinite and other types are mis-described 😱 More examples would be helpful annotated-types/annotated-types#61 (comment)
• Improve the repr of Predicate (func=<method 'isascii' of 'str' objects> is... suboptimal)
• Improve the error message Hypothesis gives in this case.
  • General template: {the type} is invalid because {explanation}
  • Replace typing.Annotated with Annotated for brevity; it's obviously implied
  • Explain what a valid type would look like, e.g. nesting Annotated is allowed for the first (type) argument, but not for later (metadata) arguments.. Try to show what a valid type would look like in a "did you mean..." message, accounting for possibly identical/subclass/typevar inner types. Check in the annotated_types namespace and use the declared name if it's one of those objects. e.g.:
    • Annotated[float, IsFinite] is invalid because nesting Annotated is allowed for the first (type) argument, but not for later (metadata) arguments. Did you mean: Annotated[IsFinite[Float]? (OK, so we'll want to unwrap a no-other-metadata Annotated and actually just say IsFinite[float])
    • (imagining that you defined your own IsAscii type) Annotated[str, Annotated[str, Predicate(func=str.isascii)], something_else] is invalid because <snip>. Did you mean: Annotated[str, Predicate(func=str.isascii), something_else]?