Way to communicate more information between libraries

[samuelcolvin]

I'm trying to thinking about a way for pydantic to communicate extra field information to hypothesis which is:

1. reusable by other libraries - e.g. doesn't use hypothesis types
2. doesn't require any understanding of pydantic internals - e.g. not based on pydantic core schema
3. can be extended without further integration discussion - e.g. is a proper protocol, not a list of types

For reference here is the hypothesis plugin from pydantic V1. The types we'd like to support with this system are as follows:

• EmailStr - a string that is a valid email address, could crudely be just a regex
• NameEmail - email and name in the name <email> format, can be a regex + email
• PyObject (now ImportString) - string that can represent anything, the hypothesis plugin used a random attribute of math
• Color - could be a regex
• luhn valid card number - currently generated by trial and error
• IPvAnyAddress - either IPv4 or IPv6
• JsonWrapper - a JSON string, possibly with a type hint

Everything else I think is covered by using Annotated and things already defined by annotated-types.

Note in V2 some of the above are implemented as arguments to Annotated (albeit with an alias), some are legitimate custom types.

One idea I thought of was to use JSON Schema - you provide a method or property on either type or argument to
Annotated which returned some JSON Schema. But looking at the above list, I don't think JSON Schema would help with many.

Therefore here's my proposal:

annotated-types defines a new property or method on types and arguments to Annotated which returns the following
pieces of information (could be a tuple, a dict with specific keys, or a dataclass defined herein):

• documentation_example - a canonical example of the datatype as might be shown in documentation, e.g. john@example.com
• random_example - a random varying example of the datatype, e.g. ad90cj-i3ljlkd@dk33w4poedd.co.uk
• type_code - a string that libraries can use to identify the type (e.g. email), and thereby decide to do more powerful things - e.g. hypothesis has a better strategy for generating random email addresses than pydantic will. Would be None if no type_code exists for a field.

The idea is that annotated-types defines:

• the above data structure
• a list of agreed type_codes, starting with the ones from above

Example usage

EmailStr: would emit ExtraTypeInfo('email', 'john@example.com', 'ad90cj-i3ljlkd@dk33w4poedd.co.uk').

hypothesis would ignore the crude random example and use it's own strategy for email addresses since it recognises the email type code.

---

Color: would emit ExtraTypeInfo('color', '#ff0000', '#00ff00'), if hypothesis doesn't recognize color it could
fallback to using the random example generated by pydantic.

---

If a user wanted their own UKPostCode type (alias of Annotated[str, UKPostCodeMetadata]), then UKPostCodeMetadata could emit ExtraTypeInfo(None, 'W1A 1AA', 'sp119dg'), None for type code since no type code exists for uk post codes, hypothesis would use just use the random example 'sp119dg'.

---

In theory another tool could use this data (e.g. for generating documentation) with no knowledge of hypothesis or pydantic.

[samuelcolvin]

I'd love to get this agreed in principle fairly quickly so we can release V1 with at least a strategy in place (pun intended) to solve this, and ideally the pydantic side written.

[lig]

Should random_example be an iterable maybe? Just to allow adding some more randomness to it 🤷🏻‍♂️

[samuelcolvin]

yes good idea. I was thinking (but forgot to add) that the library would call the function multiple times to generate different examples. But an iterator, would be best as it could either be a simple list, or an infinite generator.

[adriangb]

Hmm, this seems like a lot of stuff that is very specific to Pydantic and Hypothesis. I would rather this package remain more neutral. Could this instead be something that Hypothesis provides? I haven't written a Hypothesis plugin but maybe something like:

from hypothesis.plugin import Analyzer, MetadataProtocol, HypothesisMetadata

class PydanticHypothesisMetadataTransfer(MetadataProtocol):
    def gather(self, type: Any) -> HypothesisMetadata | None:
        # do internal stuff

Analyzer(gather_metadata=PydanticHypothesisMetadataTransfer)

And if PydanticHypothesisMetadataTransfer.gather returns a HypothesisMetadata then Hypothesis can stop recursing/analyzing the type. If it doesn't, then Hypothesis does its thing.

Essentially, Hypothesis would have to define what metadata it needs from a type and Pydantic would have to figure out how to transform it's json schema or other already done analysis into Hypothesis' metadata.

[samuelcolvin]

I think this could already be done with a hypothesis plugin, I just thought a general solution would be more powerful.

I don't think the the idea of:

1. an agreed unique string with describes a type, or
2. an example of the data

Are that unique, but I understand if this is too much of a step for annotated-types.

[Zac-HD]

The random_example is an antipattern for Hypothesis, because it breaks shrinking, so we need a way to generate from strategies in the first place. I think there are two parts here:

1. For custom types, have a Hypothesis plugin which registers a strategy for that type. luhn valid card numbers, IPvAnyAddress, and JsonWrapper might be in this category?
2. For Annotated[T, ...] types, we can use a neat deferred-resolution hack using Hypothesis resolution of st.from_type(Annotated[T, ..., some_strategy, ...]) to some_strategy and the GroupedMetadata annotation:

   @dataclass
   class MaybeStrategy(GroupedMetadata):
       def __iter__(self) -> Iterator[object]:
           if hypothesis := sys.modules.get("hypothesis"):
               yield st.builds()  # or whatever you want here

   This is basically @adriangb's idea but without needing any new features 😁
   (ok, we might want to document that iterating a GroupedMetadata might produce non-BaseMetadata objects)

and I think that those two probably cover all the motivations in this issue.

[samuelcolvin]

Mostly sounds good.

Does that mean all arguments to Annotated that want to work with hypothesis have to inherit from GroupedMetadata? I guess that's a price worth paying in this case.

The if hypothesis := sys.modules.get("hypothesis") looks a bit brittle, e.g. what happens if you want to expand a MaybeStrategy to get the normal stuff, but happen to have imported hypothesis?

Do you think hypothesis could set a contextvar or similar so there's a safer way to detect that I'm being expanded by hypothesis?

[Zac-HD]

Does that mean all arguments to Annotated that want to work with hypothesis have to inherit from GroupedMetadata?

Yep, price seems pretty low though.

The if hypothesis := sys.modules.get("hypothesis") looks a bit brittle, e.g. what happens if you want to expand a MaybeStrategy to get the normal stuff, but happen to have imported hypothesis?

Then you also get a SearchStrategy object in the annotations, which the rest of your code presumably ignores.

Do you think hypothesis could set a contextvar or similar so there's a safer way to detect that I'm being expanded by hypothesis?

Unfortunately this would miss some cases... I think the sys.modules approach is the best option available.

[adriangb]

Does that mean all arguments to Annotated that want to work with hypothesis have to inherit from GroupedMetadata?

Yep, price seems pretty low though.

Speaking of: #36

[lig]

@Zac-HD Sadly this doesn't seem to work

@given(data=st.data())
def test_hypothesis_uuid_version_pure(data):
    from hypothesis import strategies as st

    @dataclass
    class UuidVersion(GroupedMetadata):
        uuid_version: Literal[1, 3, 4, 5]

        def __iter__(self) -> Iterator[object]:  # type: ignore
            if 'hypothesis' in sys.modules:
                from hypothesis import strategies as st

                yield st.uuids(version=self.uuid_version)

    # this seem to have no effect and works just the same as `st.from_type(UUID)` 
    value = data.draw(st.from_type(Annotated[UUID, UuidVersion(1)]))

    assert isinstance(value, UUID)
    assert value.version == 1

Using hypothesis 6.75.3

I think I've misunderstood the idea. I don't see Hypothesis iterating specifically over GroupedMetadata. Is it supposed to be done on a plugin side?

[Zac-HD]

Ah, HypothesisWorks/hypothesis#3356 just hasn't been implemented yet 😅

Hypothesis runs strictly on a "no roadmap, features happen when someone volunteers to write them" basis, I've been kept busy by work and bugfixes and my phd and other features, and nobody else has taken it on yet.

[andylamp]

@Zac-HD, please do not apologize - your work in open source is a gift, not an expectation.