Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Instance apartness guards #56
This proposal tackles the thorny topic of Overlapping instances, for both type classes and Type Families/Associated types, by annotating instance heads with type-level apartness Guards. Type-level disequality predicates appear in Sulzmann & Stuckey 2002; in the type-level ‘case selection’ in HList 2004; and in various guises in Haskell cafe discussions in following years. This proposal builds on the apartness testing implemented as part of the Closed Type Families work.
To assess this proposal against SPJ's 2010 desiderata (message also linked from 'Motivation', at "well-known") at foot of message:
('FDL' is an imagined Haskell with Functional Dependencies, overlapping instances and L = local type constraints, such as inside existentials or GADTs. For historical context note that as at 2010 Haskell had Type Functions but no overlapping/Closed Type Functions; superclass constraints with Type Functions and equality constraints
To explain some of the motivation around requiring overlapping instances in the same module:
Yes providing a generic instance was the style at the time. (I'd say that style was already in decline by 2010, precisely because of the importing/overlapping difficulties.) Notice that the generic instance could only be using bare type vars in the instance head:
So the instance body could know nothing about the types of
Contrast that HList's catch-all instances at least know there's a constructor
Under this proposal, you still can't write generic 'catch-all' instances like that; unless you know all the possible type constructors that you intend it to overlap; then you can explicitly push the not-so-generic instance apart from them.
Does anybody still write in that catch-all generic instance style? What are the use cases?
I guess type inference (instance selection) needs to provide an evidence term for the selection. For apartness guards, that means evidence that
That could look like
For a guard like
Another possible solution for the awkward equation in the gnarly example. This treats the arguments symmetrically:
Not all the comparands are whole parameters from the head. For 4a we could go
I don't see the reason why the inequality needs funny syntax. Can't it be a normal constraint?
That should make the semantics quite obvious.
@MarLinn Ironically, if we kept the current instance selection rules (which I suppose are the ones you're calling "quite obvious"), your code would never work.
An instance is first selected - by looking only at the instance head, and then its constraints are added. If the constraints fail to satisfy, that's a type error. There is no backtracking and another instance is not selected.
So your proposed syntax either 1) introduces backtracking into instance resolution 2) introduces semantics that are anything but obvious (during instance resolution we have to scan for apartness constraints in potentially nested tuple of constraints).
In fact, you can define a
The whole idea of this proposal is to make use of such inequality during instance selection.
@mniip I guess I was arguing on a different level of semantics. Not precise semantics of evaluation but intuitive semantics.
Checking constraints is a necessary step anyway. So the simplest implementation would be to keep options around and just defer the pruning to later. That's not precisely backtracking, but along the line. It also doesn't sound very performant yet. But it has the benefit of adding more value to existing syntax instead of introducing new syntax just because of some implementation details.
Or approached from another side: After this proposal, there will be two possible types of constraints to add on the right-hand side. No equality constraints yet, no type family applications yet, but it appears like it's only a matter of time until they're proposed. Before long, left-hand-side and right-hand-side constraints will probably converge. Now it might be that this separation still makes sense then, and that there are qualitative differences between intra-selection and post-selection constraints that I don't see or understand right now. Quite possibly. In this case, it might be worth exploring what we know about these differences right now and at least discuss how this proposal fits into that current knowledge in an appendix. Not for me, but as documentation for those that will want to add more of them.
Thanks @mniip , you're quite correct.
I'm very carefully not describing these guards as "constraints", because that would indeed be confusing. If you want a different word: "restraints" -- that is, they restrain the instance from applying.
Yes they have. It's a common newbie's mistake. So I've deliberately used syntax to the right of the instance head, to look as much as possible like guards at the term level. If you think that's still too confusing, by all means suggest alternative syntax.
Type improvement/type inference is hardly a "detail", IMO. It's why we have FunDeps and Type Families and equality constraints
The merit of this proposal is it doesn't change any of the underlying semantics. It's chiefly providing prettier/more readable syntax for what you can write already. (And in a significant number of cases, you can avoid adding post-resolution constraints altogther. I'm sure you'll see that as a Good Thing. ;-)
If you want to change the semantics of pre-/post- instance selection, that would be a very different subject for a proposal. I note that none of the Alternatives considered are proposing anything like that.
Well, Type Family equations don't have (left-hand-side) constraints, so no. And guards are not "constraints", so no again. Guards can only be type comparisons, no class applications, so no a third time. (Compare that at the term level, I can put arbitrarily complex expressions in guards: just the same sort of expressions as I can put to the right of the
(I hear howls of anguish coming from GHC HQ.)
This is a long proposal (which I encouraged @AntC2 to write), and I'm surprised it hasn't attracted more commentary. Perhaps a summary is in order:
This proposal introduces instance guards, a new form of type constraint that can be used during instance selection (that is, before committing to an instance). This applies to all kinds of instances: of classes, type families, and data families. Only one form of guard is currently proposed: an apartness constraint. So, if we have
then this instance is chosen only when
Because it's hard to see how to marry instance guards with overlapping instances, these features cannot be used in concert. To avoid this possibility, classes whose instances (may) have guards are required to be marked with the
Note that, due to the way apartness is defined, this new feature does not induce backtracking into the solver and can be implemented straightforwardly, in my opinion.
Thank you @goldfirere for the excellent (brief!) summary.
A detail that might have escaped: guards can help in applying the FunDep consistency conditions, by 'pushing apart' the argument side of FunDeps that would otherwise overlap and thereby give inconsistent result sides.
GHC's FunDep consistency checking is currently rather lax. And for good reason: there's lots of code exploiting that; I've written plenty myself. With guards there's an opportunity to tighten that check. I haven't writen that into the proposal (yet). I'd appreciate feedback ...
As a result of the laxness, there's some corner cases of FunDeps you can write that cannot be equivalently expressed using Type Families and Equality constraints, not even with Closed Type Families. Trac #10675 is an example. Richard's comment is the example is "bogus". I agree. But it's difficult to pin-point the bogusness and 'fix' the consistency check without breaking (legitimate?) code that exploits the laxness. So #10675 remains unfixed.
Currently the FunDep consistency check goes:
Arguably that last check should be:
Trac #10675's example is:
These instances do not overlap. Nevertheless the argument sides of the FunDep
(which should use only the first instance) gets an inferred signature
With the tighter check on FunDep consistency, as above, those instances must be rejected: the result sides after substitution are not identical
But under those tighter rules, code like this (relied on by many type-level hackers) would also fail the check:
The argument sides of the FunDep unify, with substitution
With the tighter consistency check and Instance Guards, that relied-on code goes:
(Firstly note this is easier to read: the result
That's a a mega-proposal. Well written and thorough. But a bid daunting.
In some ways it's less modular than overlapping instances. Eg today I can say
If we put all this in, is there anything else we can take out?
Can I say
I don't have a clear verdict.
Thank you, and fair comment. I've been (over-)thinking this since at least 2012 alternative to Closed Type Families/see "Disequality constraints"
That's the danger point with overlaps: I put
EDIT: Errk! I might have misunderstood Simon's point. He said
Yes you must think of those instances together. So exactly the same as closed instances. I think the two approaches are very similar within a single module. What I can't do with closed instances is put all
I think I could spec a mechanical translation from closed instances to apartness guards. (That's why the proposal works through an example in detail. As I understand from Richard, CTFs under the hood are using a similar mechanism to guards, to try to help progress.) But less hope of a translation for overlapping type class instances (esp Incoherent ones).
For Type Families, I agree the equations are easier to read (each on a line). But you should also look aesthetically at Instance Chains (or Morris+Eisenberg Closed Classes). Now the instance heads are spread out and the bodies/method overloads are mixed in with the instance logic. To understand each instance, you have to read through the chain to understand what pattern of types this applies for. Horrible!
Yes (long term plan): take out Closed Type Families (or translate them to apartness guards); take out
Oh, and I think we have a better basis for Injective Type Families (that is, where result with one arg determines the other arg).
It would have been better to put in this proposal before all that other stuff -- circa 1998. There might have been a different ''History of Haskell: being guarded with class'' ;-).
(I'm a little miffed at your "all this": there's no new type-level technology; the mechanisms are only unification and substitution and apartness checks; you can achieve this today in GHC, as does HList, it would be just very fiddly and verbose. In fact you could achieve it in 2004. The type theory was sketched in 2002, Sulzmann+Stuckey/Chameleon.)
Yes you can say
Thank you very much for taking the time to review.
Please don't be miffed. It's good work -- just has a lot of implications.
I wasn't clear enough. Today with overlapping instances I can declare
in a library module, and then override it in a client module, written separately, later, by a different author
With guards I have to anticipate all the other instances, writing (I suppose)
Not only is that tiresome and anti-modular, it may not even be possible if the clients haven't been written when the library is designed. So I anticipate that there will be pressure to keep overlapping instances. Is that right or have I misunderstood?
I think you can!
and similarly for module B. We simply delegate to a specialised class. Now we can put all the instances for
but then we realised that instead we can just make specialised closed families
I agree that is horrible. But writing guards is also tricky; that's why we have closed type families and top-to-bottom matching for term-level functions. If closed type families work well for the type-family case, I wonder if it'd be worth looking for a closed-instance syntax that was less horrible. eg
I'm just thinking aloud here. Having named instances could be useful in other ways (selective export).
I for one would find it helpful to have these impressively long list of points enumerated, with more info about each to show how the problem is solved. Killing many birds with one stone makes a proposal stronger.
Um, those instances don't overlap. Did you mean
What you describe is how the mtl library used to work. Except it didn't work, so it was rewritten to not use overlaps like that. The problem with writing a 'catch-all' instance in the library module is it leads to all sorts of surprising incoherence/silently changing behaviour from imports.
I'm not sure you've picked a strong example:
Anyway, overlapping instances do tend to be tiresome and anti-modular. Closed instances don't address this requirement either. So adding guards to the catch-all instance in the library has the same burden as adding instances within the closed set.
Yes I am anticipating in the short term, people will want to continue to use overlapping instances. In the longer term they can piecemeal (class-by-class) change over, replacing the overlap pragmas with guards -- only for the instances that do actually overlap.
Yeah OK. (And RIchard made a similar point.) Now that I've seen it coded out, I'm even more put off by how verbose and non-perspicuous it is.
Yes I recognise that proposal. It came from me. I continue to be disappointed it never happened. (But I prefer apartness guards.)
Maybe simpler for the compiler-writers. You haven't convinced me it's simpler for the instance-writers.
Firstly, I think it's much easier to write the logic for an instance+guard stand-alone. More importantly the reader can understand it stand-alone. Secondly, the compiler can validate it is genuinely apart from any other instance. With a sequence of instances (especially if they're widely spread out with method overloads in-between), the logic is not so clear. And I think it's prone for the reader to mistake which patterns get selected by which instance decls. I'm saying this based on looking at some of the examples in the 'Instance Chains' paper. (Admittedly that has a lot of other fancy conditioning, like instance-level
With respect, I don't think you do exactly have top-to-bottom matching: In order to make progress with solving, CTF resolution tries to look for 'coincident overlap' by considering all of the equations, not necessarily in top-to-bottom sequence. From Richard's description, I think it's ended up working a lot more like apartness guards. My mental model is that all instances are 'competing' to be selected. Ordered equations is one way to prefer an instance out of several that might apply. Overlapping instances is another way (and not a very good one). The trouble with both those ways is the compiler has to entertain several instances that might apply. The advantage of apartness guards is the compiler knows the instance definitions are non-overlapping (after taking guards into account), so it can keep improving the types until at most one instance applies. Importantly, this doesn't involve search or backtracking.
Yes they do work well from the equation-writer's point of view. Not so much under the hood. IMO it's confusing having two mechanisms: both stand-alone TF instances, associated type instances within stand-alone class instances and closed instances. I'd prefer to write more of my type instances as associated types; but currently that clashes horribly with overlaps.
Aargh! No never introduce named/scoped instances. It's a total can of worms. (People claim it was implicit in Kaes' or Wadler's original designs 1988; I think that's a mis-reading.)
There is a legitimate reason currently for wanting selective export: I've a 'private' instance that overlaps instances my clients want to write. Yes that's one of the (many) awkwardnesses with overlaps. Solve it the proper way with instance guards. So we have global coherence and global non-overlap (after taking guards into account).
OK I'm happy to do that. And I can quote the ticket numbers. Are you sure you want the proposal to be even longer?
I was thinking of instances of the following form:
I'm not proposing any change to how GHC decides whether two type (scheme)s are unifiable or apart. If there's bogusness now, that presumably affects ordinary instance selection:
If #32 is going to fix the bogusness, I'll still follow however GHC decides: apartness guards will follow the #32 rules. Because instance selection must be consistent with the behaviour for classes and instances that don't use apartness guards.
EDIT: I don't need to express an opinion on how
No, that's not correct.
Making them apart in the future is not a breaking change currently (at least that's what Richard says), but it will become a breaking change if we introduce apartness guards (as I demonstrated by the example above).
I agree with @int-index here. With the feature proposed here, making more types apart is a potentially breaking change. However, this is such a corner case that I'm not worried about it one way or another.
As to @AntC2's quandary: I agree that it's unclear how to proceed here. Simon's "mega-proposal" comments was, I believe, more about the length of the text, not the difficulty of the implementation. I don't think the implementation would be all that hard. (I wish I could say otherwise, but I agree with the "less than a handful of [busy] people could implement this" -- we need more type-checker grease-monkeys!)
I recommend submitting to the committee for a decision. If accepted, yes, this may wallow about for a while waiting for an implementor, but that's just the way of things. If you wish to submit, do be sure to mention @ nomeata, the committee secretary.
Can the proposal please include a BNF for all the proposed changes to the source language syntax? E.g it looks as if the guards are always of form
I'm unclear about the precise rules for how to handle fundeps when we have instance guards. (And if you say that GHC does not have precise rules for how fundeps are handled anyway, your'd be right. But perhaps we can take the opportunity to fix that problem.)
Yes, you shouldn't take me too seriously on this. It is a brand-new bit of technology that has to attach to every instance decl. But that should not be too hard.
I dislike that it overlaps so much with the functionality of closed type families. But desugaring a closed type family into bunch of open family instances with suitable guards is (I think, not routine).
Earlier you wrote
I'd love to see this vision articulated explicitly in the proposal. It becomes much more attractive if it allows us to simplify the language by (say) deprecating or removing features. Making that explicit as a medium term goal might elicit responses from people who are relying on those features and think, for some reason, that apartness guards will not solve their problem.
I'm not arguing strongly against. But it's a relatively big feature in a part of the type checker that is already complicated.
I'm thinking that if @simonpj's reaction is there's a lot to read, other Committee members will just give up: so much text; then it must be complicated/too violent a change. Whereas ...
What (I think) I'm doing is finding a way to express the logic that's already in effect for overlapping instances, and in effect for closed type families; and explains ghc's spurious 'instances inconsistent with fundeps' rejections. A new formalism, maybe; but not new technology.
I'm happy to articulate more motivations in the proposal. I think in fact most already is. But you've not seen it because there's just too much darned text.
I'm not expecting anyone will abandon closed type families, or overlaps, or
OK will do. Guards are often of the form
Here's an example where the
If there's some reason why
No change to how fundeps generate the 'result' types, once an instance is selected. (Yes I know that's not an answer.) There is a change to validation for
Ah, so now you want a conference paper. Where can I find a sponsor? ;-)
Turns out the rule (or at least a possible rule) has been hiding in plain sight since 2000:
This more or less follows Mark P Jones' original paper; and coincides with the Relational Database predicate logic characterisation for Functional Dependencies. So all we need to tackle is how to get GHC to validate a set of instances with this cycle in superclass constraints; and quite possibly cycles in the instance constraints.
I think GHC does have precise rules. Partly documented in the 'FDs via CHRs' paper, supplemented with comments in the code that you've been diligent enough to copy on to various Trac tickets -- for example the 'bogus consistency check'.
Which bits do you think are not precise enough? I must admit that once you switch on
Fair question. Let me try to be more specific.
With fundeps, if we have
then when we have a constraint
How does that rule change when we add instance guards?
We coverage and consistency rules for fundeps. How do they change?
Better than specifying changes (as diffs from a baseline that may vary between readers) would be specifying the new rules.
No change. As I said.
No change. But because the consistency rule is different [below], it is less likely you'll need to write a 'catch-all' overlapping instance which needs
The check for consistency (pairwise between instances):
Again, see the bigger example I linked to in earlier comment.
It's the "apply
Ok good. Can you just write the new rules in the "Proposed specification" section? Then my questions are answered explicitly by the proposal rather then in the comment stream.
(Adding section numbers would be super helpful; a one-line change.)
ref @MarLinn's comment way back when, and @mniip's reply: even when a disequality constraint (not guard) would be enough to block an instance, you can't always write one. Consider this (pathological) case from the User Guide on
Even though the instance for
Let's say class
Assuming type operator
But you can't put a wildcard
You can get away (maybe) by switching on
I find an apartness guard more to the point
@AntC2, I am scheduled to ping on the status of this?
I feel like this is this kind of language innovation that would be well suited for an academically reviewed paper that comes with an example implementation that thoroughly explores the design, and can afterwards go through the GHC proposal process. But the effort that takes is probably too much for anyone who does not write papers for a living…
But if you think your proposal is done and in a good shape, you can certainly submit it to the committee!
Yes, or a project by someone under (say) @goldfirere's direction that uses the idea to implement Closed Type Families and/or Overlapping Instances within a module; we expose the surface syntax later.
I'm happy to help anybody/anything that will get a bit of traction.
This comment is bugging me:
I feel an algorithm is possible (and would also be useful for making further progress with Injective TFs and to turn 'Closed Type Classes' (possible future) into open class instances). Here's a stab. I'll use the language and notation of the Closed Type Families paper.
The basic idea
Supposing a guard is needed, how to generate it?
From section 3.5 Optimized matching of the paper
From the wiki
from the paper section 2 (adapted)
For the second equation, wrapping the instance heads as tuples and unifying gives
Two further wrinkles for the algorithm (to be demo'd in the final example below)
Those two points suggest the algorithm should precompute by running down the equations looking for those conditions before generating guards.
Final example is the gnarly one from the proposal. Using the algorithm generates guards that are symmetric, rather pleasingly:
The guards for 4a are obtained from cancelling down:
Withdrawn; and withdrawing.
This proposal has sat around over a year. There's been some polite comment but no uptake. Addressing Functional Dependencies and instance overlaps and their interplay just doesn't fit with the direction of travel for GHC. Indeed the difficulties and disfluencies in GHC's behaviour wrt those extensions -- that were apparent from over a dozen years ago -- have received no love.
Then I might as well start afresh from that time, with a better-disciplined Haskell. And I'm finding it surprisingly easy to engineer.
@nomeata I presume that closing this doesn't lose it entirely from the proposals process(?) I'm happy for and would support somebody to take up the ideas later.
If I'm frank with myself: GHC-Haskell is no longer the language through which I became enthusastic about Functional Programming. Still the records system is an embarrassment. (How many attempts have there been to make progress?) Meanwhile GHC has become bloated with abstruse features. Most of those features are of small benefit to me. Each seems to have come with significant downsides in added complexity and impenetrable error messages -- even when I didn't think I was using that extension.
Now that I've discovered it's not so difficult to hack (some) compiler, I'm going back to a simpler Haskell vintage ~2006.