Replies: 9 comments 5 replies
-
Protocol BlessingWe want to be able to tell beartype (and e.g. downstream plugins) what precisely a thing's "type" is, and make sure that's known relative to all other types. the IdeaSo the idea is that, wherever possible, a subset of our stuff can be explicitly "blessed" to have it's type known statically by beartype, which let's beartype.door do all kinds of awesome! example napkin codefrom beartype import beartype
from beartype import typing as bt
from beartype.door import TypeHint as TH
from abc import abstractmethod
import types
CT = bt.TypeVar("ClassType")
@bt.runtime_checkable
class BearTyped(bt.Protocol, bt.Generic[CT]):
"""in the name of our unholy bearfathers, I dub thee as _typed_!"""
@classmethod
@abstractmethod
def __beartype__(cls: bt.Type[CT]) -> TH | bt.Sequence[TH]:
...
def beartypeable(myclass: bt.Type[CT]) -> BearTyped[CT]:
"""nasty monkey-patching, do not use"""
@classmethod
def __beartype__(cls: bt.Type[CT]) -> TH:
"""Default janky behavior"""
return TH(cls)
if not hasattr(myclass, "__beartype__"):
setattr(myclass, "__beartype__", __beartype__) # yes?
return myclass
# class MoreBearable(myclass, BearTyped): # no
# def __beartype__(cls)->TH:
# return TH(cls)
if __name__ == "__main__":
@beartypeable
class Fake(bt.NamedTuple):
name: str
kind: str
velocity: float
fake_data = ("n1", "bob", 2.3)
fake: BearTyped = Fake(*fake_data)
FakeLike = bt.Tuple[str, str, float] # let the shennanigans commence
print(
"\nbaselines\n",
TH(bt.Tuple[str, str, float]).is_bearable(fake), # works, Tuple >= NamedTuple
TH(Fake).is_bearable((fake_data)), # Doesn't, NamedTuple <= Tuple
TH(Fake) <= TH(FakeLike), # works, Fake is a specialization of FakeLike
"\n\nshennanigans\n",
TH(bt.Tuple[str, str, float]) >= Fake.__beartype__(), # should work, right?
TH(bt.Tuple[str, str, float]) <= Fake.__beartype__(),
TH(BearTyped).is_bearable(fake),
fake.__beartype__() is TH(Fake),
Fake.__beartype__() is TH(Fake),
)baselines
True False False
shennanigans
False False True True TrueassessmentNotice how the tuple never checks as a supertype of the namedtuple? even though it's the same thing inside? That's the generics issue. In practice, I would like the beartype thing to return a sequence of typehints when I want it to behave like a product type. why? Check back in our tags/adt section for more! 😛 : |
Beta Was this translation helpful? Give feedback.
-
Gah!!!!!!!! You may have uncovered a squishy The Truth: Nobody KnowsSo, >>> from beartype.door import TypeHint
>>> TypeHint(tuple[str, str, float]) >= TypeHint(Fake)
True # <-- this currently returns "False"; this is possibly badSemantically, that is. Pragmatically, official documentation for the
Of course, types are valid annotations. From this, we can conclude that The Truth: Mypy KnowsSo, the world-shattering question that will shape the course of future QA for millennia to come then becomes: what should
Dare we defy The more important question, I suppose, is what do static type checkers think about all this shadow madness? Does mypy agree with the ...it's time to incite the Anti-mypy Uprising of 2022! 🔥 💥 |
Beta Was this translation helpful? Give feedback.
-
|
Ah cool! Ok I'll come back to these, but I think it is a bug then. Not in general, just for this tuple case. I knew NamedTuple was a subtype (and subhint) of tuple, and we happen to know that immutable containers, as a rule, are by default covariant in their parameters. So if the parameters are subtypes, then the types should be as well. This is why we use Sequence rather than List and stuff, so we know covariance is applicable for container types at runtime! |
Beta Was this translation helpful? Give feedback.
-
Yes. Yes, of course. We know that! Surely. Clearly. I mean, who doesn't. Right? ...tugs collar fitfully Okay, okay. I admit. I actually had no idea. @beartype currently just ignores the Great Covariance-contravariance Divide by uniformly type-checking everything covariantly. Thankfully, that seems to mostly be what everyone wants anyway. The creaking silence and ominous lack of complaints justifies our shoddy laziness. But that's a poor excuse for ostensibly doing the wrong thing. @beartype should really be stricter and more standards-compliant about this sort of thing... especially where type variables are concerned. If you see @beartype improperly type-check something covariantly where @beartype should instead be type-checking that thing invariantly or contravariantly, please submit an issue exposing our shame for all to see. In the meanwhile, let's open up a new issue disclosing the above badness for all to see. |
Beta Was this translation helpful? Give feedback.
-
|
I mean, this is all pretty new to me tbh, just a really weirdly engaging rabbit hole for some reason. Eventually I will absolutely say something stupid and being a total n00b at this will be my excuse! (Haha, take that! Not-so-smart now, am I??) I did go ahead and add some other "bug" examples showing generic variance on that issue, so that we can check. I guess this means something like a |
Beta Was this translation helpful? Give feedback.
-
|
I don't feel like writing out an example of the global registry option (but someone else feel free to say what the merits would be! I don't really care, for the sake of argument), but nonetheless we either maintain Why?? Runtime Optimization of complex type checksThis can let us do some pretty stupid stuff, pretty fast: deeply nesting? why not?If I know a priori that some type Either instances or soon-to-be
OK, that's the setup. Now... what do we know?
There we are!This is how we can get tagged unions a la pydantic, or fancy nested product types like stacks of But... how?The way I've written this, I'm imagining a preprocessing step _inside of gasp Ok, this may be a bad idea, since that devilry is so optimized, it's actually insane that it works. But what I'm hoping is that for consenting types, the type checker can skip the need to actually validate everything in a generic, opting instead to explicitly trust whatever we told it our intent was until we get to the type stack's foundation, when the check comes due. So, in that glorious irony, we fix the stack of nominal typing with a teensy-but-foundational layer of structural typing to get us going. Bad, no-good, unoptimized PoC as follows: from beartype import beartype
from beartype import typing as bt
from beartype.door import TypeHint as TH
from beartype.door import UnionTypeHint, is_bearable
from abc import abstractmethod
import types
CT = bt.TypeVar("ClassType")
@bt.runtime_checkable
class BearTyped(bt.Protocol, bt.Generic[CT]):
@classmethod
@abstractmethod
def __beartype__(cls: bt.Type[CT]) -> TH | bt.Sequence[TH]:
...
def beartypeable(compiletime=None):
"""nasty monkey-patching, do not use"""
def decorator(myclass: bt.Type[CT]) -> BearTyped[CT]:
@classmethod
def __beartype__(
cls: bt.Type[CT], compiletime: bt.Optional[TH] = compiletime
) -> TH:
"""Default janky behavior"""
if compiletime and isinstance(cls, type):
return compiletime
else:
return TH(cls)
if not hasattr(myclass, "__beartype__"):
setattr(myclass, "__beartype__", __beartype__)
return myclass
return decorator
def wrapped_is_bearable( # forgive me, for I have copy-pasted
# Mandatory flexible parameters.
obj: object,
hint,
):
print("\n--CALLING WRAPPER--\n\n")
if TH(hint) <= TH(BearTyped):
myhint = hint.__beartype__()
print(f"my hint says {hint}, but really that means a: \n\t{myhint}")
else:
myhint = TH(hint)
if isinstance(obj, BearTyped):
# forgive the use of isinstance
print(f"\n\n I see it now!\n This hint is no mere {type(obj)}")
mybeartype = obj.__beartype__()
print(f"but a blessed {mybeartype}!\n My third eye has opened!\n\n")
# oh look more isinstance
if isinstance(myhint, UnionTypeHint): # optimizeable overhead
print(f"checking divergent timelines...{mybeartype.hint}")
if mybeartype in myhint: # verbatim matches one branch
return True
else:
possibles = [wrapped_is_bearable(obj, t.hint) for t in myhint]
# going recursive, see if any won
return any(possibles)
# so this is an ugly hack approximating prod type
if myhint <= TH(tuple):
print(f"checking convergent timelines...{mybeartype.hint}")
if mybeartype <= TH(tuple): # really should be...
# well, except multiple inheritance
# ok, and protocol overlapping... 0_o
requirements = [
wrapped_is_bearable(x.hint, t.hint)
for x, t in zip(mybeartype, myhint)
]
else:
requirements = [
wrapped_is_bearable(mybeartype.hint, t.hint) for t in myhint
]
print(
"\nthese succeeded:\n\t ", list(zip(mybeartype, myhint, requirements))
)
return all(requirements)
else:
print("currently on the one true timeline")
return myhint >= mybeartype
else:
print(f"Oh, this \n\t{obj}\n is a boring _normal_ thing...")
return myhint.is_bearable(obj)
if __name__ == "__main__":
FakeLike = bt.Tuple[str, str, float] # let the shennanigans commence
@beartypeable(compiletime=TH(FakeLike))
class Faker(bt.NamedTuple):
name: str
kind: str
velocity: float
FakeLike = bt.Tuple[str, str, float] # let the shennanigans commence
# this kinda __beartype__ expansion thing has to be automated...
@beartypeable(compiletime=TH(bt.Tuple[int, Faker.__beartype__().hint]))
class FakeEst(bt.NamedTuple):
id: int
fakestuff: Fakerthen: print(wrapped_is_bearable((1, ("hi", "bob", 2.8)), FakeEst))❯ python -m grabble.beartypeable
--CALLING WRAPPER--
my hint says <class '__main__.FakeEst'>, but really that means a:
TypeHint(tuple[int, tuple[str, str, float]])
Oh, this
(1, ('hi', 'bob', 2.8))
is a boring _normal_ thing...
True print(
wrapped_is_bearable(
FakeEst(1, Faker("hi", "bob", 2.8)), bt.Tuple[int, FakeLike]
)
)❯ python -m grabble.beartypeable
--CALLING WRAPPER--
I see it now!
This hint is no mere <class '__main__.FakeEst'>
but a blessed TypeHint(tuple[int, tuple[str, str, float]])!
My third eye has opened!
checking convergent timelines...tuple[int, tuple[str, str, float]]
--CALLING WRAPPER--
Oh, this
<class 'int'>
is a boring _normal_ thing...
--CALLING WRAPPER--
Oh, this
tuple[str, str, float]
is a boring _normal_ thing...
these succeeded:
[(TypeHint(<class 'int'>), TypeHint(<class 'int'>), False), (TypeHint(tuple[str, str, float]), TypeHint(tuple[str, str, float]), False)]
FalseEDIT: did some more work to make things... more sensible. by a slim margin. Still, the second test is failing, since for some reason those equivalent types are not compatible 🤔 Time for sleep now though. |
Beta Was this translation helpful? Give feedback.
-
|
ZOMG. My piddly brain succumbs to your galactic brain. I barely grok the forest that you're boldly clearing a path through. So let's instead hyper-focus on the trees, which is my only forte on GitHub. I do trees like nobody's business. It's pretty much all I do, really. Thank the Asperger's Syndrome for small favours, everybody.
Let's... not do global if we can help it. Please tell me we can help it, because I barely understand what's going on above. Local is the way in all things except video games and @beartype, which are both admittedly impossible to do local. I live in the mouldy woods mostly inhabited by elderly retirees and rabid wild turkeys. What would locally sourced video games even look like here!?!?
I like what I'm hearing.
I love what I'm hearing. Gimme that pretty stupidly fast stuff, stat!
My head is already blowing up. So, let's pretend this is the only use case. Immutability is the way. Functional languages and JAX tell us this. Some people are even listening. I might be one of them.
Okay. This is fascinating. Flame-breathing dragons to my left; many-headed hydras to my right. First, let's admit that @beartype (probably) cannot internally call a pure-Python callable resembling Calling pure-Python callables incurs considerable overhead. Internally, @beartype avoids that overhead by instead inlining raw In theory, that's hopefully still not an obstacle to the runtime typing revolution you're envisioning. But it kinda sounds like it might be. The usual way of augmenting raw In practice, I'm unsure where that leaves us. Let's pretend we can still do everything you've outlined in a
Superficially, that's beginning to smell disastrously inefficient. @beartype guarantees negligible <1µs type-checking overhead. Anything that comprises that is strictly verboten, sadly. Ludicrous speed is (pretty much) the only reason anyone uses @beartype. Yes. A great sadness is welling up in my soul. Matters would improve considerably if we only care about # This conditional is great! @beartype can do this at decoration time,
# in which case speed is largely ignorable. Raise your hands like you
# just don't care, everybody!
if TH(hint) <= TH(BearTyped):
myhint = hint.__beartype__()
print(f"my hint says {hint}, but really that means a: \n\t{myhint}")
else:
myhint = TH(hint)
# This conditional frightens me, which is less great. @beartype *CANNOT*
# do this at decoration time but must instead perform this test at
# *EACH* type-checking level of *EACH* nested container in *EACH*
# type-checking call. This is compounded by the fact that `BearTyped` is
# a protocol, which are notoriously slow to type-check. The
# "beartype.typing.Protocol" superclass is actually an undocumented
# macro-optimization that subclasses and overrides the notoriously slow
# "typing.Protocol" superclass with considerably faster behaviour, but...
# Still. It's runtime protocols. We really should *NOT* be testing every
# single thing to be type-checked against a runtime protocol. Even
# optimizing this into a much faster, mostly equivalent
# "hasattr(obj, '__beartype__')" call would impose considerable
# slo-mo across the board for general @beartype users -- who will then
# have my GitHub avatar on a pike that they parade on their front lawn
# and then burn in ignominy on Leycec Fawkes Day.
if isinstance(obj, BearTyped):The window of progress is closing. Gah! I can feel the fetid breath of Python inefficiency breathing down my ice-burned neck even as I type these broken words. I'm spent now. Please tell me it's not all for naught! You've finalized such a phenomenal doctoral-level thesis on "Dynamic Product Types: The @beartype Way" here. Squandering the genius you so proudly flaunt above would be the baddest thing I've done all month (followed only by playing twelve continuous hours of Stellaris last Sunday night, which I'm less proud of than you might believe). I too want to destroy Pydantic (...in the nicest way possible, of course) with a competing @beartype: why you so rabidly obsessed with runtime complexity, bro. 😮💨 |
Beta Was this translation helpful? Give feedback.
-
|
Gah! I thought for two real-world seconds and realized belatedly that this... if isinstance(obj, BearTyped):...probably can also be relegated to decoration time, maybe? Specifically, we would perform that check in the private def _beartype_type(...):
...
if issubclass(cls, BearTyped):Presumably, this would also necessitate refactoring the Would operating on the class rather than the instance of that class demolish your Steven Hawking-tier QA dissertation? Or is the fragile house of our hopes and dreams now thoroughly crushed beyond repair and useful commentary? |
Beta Was this translation helpful? Give feedback.
-
So I actually am super glad you asked this, I struggled the longest on this when I was cooking things up. _Notice the My thinking was that, with deeply-nested generics, I want to spend time comparing This is why my proof-of-concept wrapper function is recursive in |
Beta Was this translation helpful? Give feedback.
-
Yes, yes, YES!!! I feel warm and tingly inside. Other questions raise their sinuous snake-like muzzles, however:
If that works (...it won't), we're in the clear for fun with dynamic code generation at @beartype decoration time. My dim suspicion is that most of the magic in the # If the current type hint annotating the current callable being
# decorated by @beartype is a "beartyped thing" (i.e., the type of
# this type hint defines a __beartyped__() dunder method), then
# just generate type-checking code efficiently deferring to the
# beartype.door.TypeHint.is_subhint() method.
if is_beartyped_thing(hint):
# "beartype.door.TypeHint" object or sequence of such objects
# describing this type hint.
hint_wrapper = hint.__beartype__()
#FIXME: This is clearly insufficient. "{{pith}}" is the
#current object being type-checked (e.g., parameter,
#return value, something passed to is_bearable()). That
#will pretty much *NEVER* be a "TypeHint" object. So,
#we'll need to test "is_beartyped_thing({{pith}})" on
#that object as well and, if true, call
#"{{pith}}.__beartyped__()" (*or something something*) on
#that object before passing it to `is_superhint()` below.
#WHO EVEN KNOWS WHAT'S GOING ON HERE ANYMORE?!?!?!?!?!?!?
# Pretend you didn't see substrings like "{{hint_wrapper}}"
# or "{{pith}}" here. That's just to remind us to generate
# code operating on the expected objects at type-check time.
return '{{hint_wrapper}}.is_bearable({{pith}})'ELI5 This Like I Don't Know What I'm Doing...because I kinda don't. I think what we're doing here is defining an extensible plugin architecture enabling decisive and courageous end users to override standard The only issue then is that said end users probably want some means of telling the I'm suspecting we now need to either:
More importantly, @beartype will now need to publicly expose signs. Signs are how @beartype internally categorizes PEP-compliant type hints. For example:
Signs are currently sequestered away in a private submodule of @beartype. We'll need to publish that. This is all starting to get intensely sweaty – but these are fascinating discussions worth discussing. Somehow, someway, @beartype absolutely does need sane plugin architecture. This may be the start of something expensive but also beautiful. Let's see if anybody does anything. I can barely even release |
Beta Was this translation helpful? Give feedback.
-
|
Wam. Bam. Plugin API begins with a whimper at feature request #191, where we delineate the first of many shambling steps to be taken towards a full-featured @beartype plugin API. I did this so that I wouldn't forget anything. I'm already forgetting everything, though. Stay inside my head, you half-formed and vaguely incoherent conceptual metaphors! 🧠 💭 💥 We begin with signs. You'd think that would be mundane and trivial. I thought so, too. Then I wrote everything up. I am now listless and half-dead at 2:18AM in the morning – which only goes to show that you can't trust anything in this world. |
Beta Was this translation helpful? Give feedback.
-
|
All specced out. Please see feature request #193 for a very cursory draft discussion of the proposed These long-winded late-night diatribes are for you. Who else is gonna read this stuff!?!? |
Beta Was this translation helpful? Give feedback.
-
This idea was bad, and I should feel badSo I realized after name-dropping Metaclasses exist, and But seriously, I think basically all of the instance-vs-type shenanigans I was doing above are totally done cleaner by Phantom. The one issue was that they have to do a type-check the old TypeGuard way, meaning one-per-container-element (i.e. slow). So to answer your question
Yes. Well, kinda. I think I wanted a way
and ... the way to get there, for my brain, was to provide a mechanism for users to override type checking in general. Whether they do the smart thing and rely on beartype.DOOR, or do insane, singularity-causing eldritch horrors was on them. Since I see you've already launched an all-out-war on things, would you mind taking a look at the horrifying fork I've started over at Kermode Types that was meant to really dig into how phantom-types works and what beartype could to to speed it up? And mostly, how does the new plugin system make that easier? |
Beta Was this translation helpful? Give feedback.
-
|
Glargh! I'd forgotten all about the lucid mysteries of Further evidence we're on the non-ideal timeline.
Ah-ha! I can answer this... I think.
Static type-checking is mostly a non-goal for us. I mean, it's vital that we not break static type-checking too badly – but @beartype generally doesn't go out of its way to support static type-checking until users begin raising pitch forks and wildly storming the castle gates. Runtime type-checking, of course, is the goal – but there's more than five hundred billion ways to skin that QA onion.
...not do what Instead, whatever way can most effectively support beartype validators is the way. Beartype validators minimize (and, in many cases, entirely eliminate) the number of pure-Python calls needed to perform a single type-check. This is why our syntactic sugar for the proposed
I also have no answers – only more questions. If @beartype's prospective New Plugin System of Wonders can't help us do things easier and/or faster, then it's bad and I should feel bad. 😮💨 |
Beta Was this translation helpful? Give feedback.
-
Running off of #187, where some limitations of
typingare causing fundamental issues that prevent possible functionality from existing. Thought I would move the discussion here, since we started to stray off into plugin/protocol territory, which would impact a heck of a lot more than just contravariant generics. I'm going to add ideas to this as I have time, but for now, a checklist for what we might discuss:Mechanisms (that could exploit
TypeHintfor great good:tm:)__beartype__attribute to override checks in complex typesImpacts (i.e. why the heck are we talking about this)
I will post some initial code sketches for these, I just wanted to get things started before I take a break for lunch!
Beta Was this translation helpful? Give feedback.
All reactions