Rethinking the default class hierarchy

[Robert-Aron293]

No description provided.

[mdparker]

Thanks!. A few points:

• Please refrain from using a number in any commits. This DIP has not yet been assigned a number.
• The header is missing. Please add it and set the status to "Draft".
• The Table of Contents is missing.
• The link to Edi's talk should go in a "Reference" section near the bottom.

Please refer to the template.

[Robert-Aron293]

Done. Thanks!

[dukc]

Reconsider the naming of the interfaces. Three of these are verbs (Hash can be alternatively seen as a substantive), one is an adjective. I think adjectives and substantives are okay but verbs, not so much.

[ntrel]

For those I suggest something like: Stringable, Hashable, Equatable.

2 inconsistencies:

• Here Ordered has method opCmp but the definition below has cmp.
• Here Equals has method opEquals but the definition below has equals.

[edi33416]

Yes, those are typos.
The methods should have the name opCmp and opEquals

[dukc]

You are right that introduction of ProtoObject will not cause breakage. However, adding attributes to Object:s member functions will. Any classes doing, for example, an unsafe comparison, will break.

Unless you meant that the Object will not implement the four interfaces you're suggesting to add?

[Robert-Aron293]

Object will not implement the four interfaces. It won't be changed

[dukc]

Ah okay, that clarifies a lot. Though, did the DIP say it to did I just read too quickly?

[dukc]

Are the nothrow and @nogc attributes worth it here? Sure you can implement most comparisons easily enough with those attributes, but there are likely classes around that reliy on exceptions and the gc for this. The migration would be painful for such code. You rarely use classes without the gc anyway.

[edi33416]

Checking for equality should not throw, nor use the GC. I don’t see a problem here. Am I missing something?

[dukc]

I was assuming object would implement Equals but if it does not, it's not so much a problem. Old code can simply not implement Equals. I guess at least the nothrow is justified then.

[dukc]

Same concerns as for the equals attributes.

[edi33416]

Computing the hash of something (even Object) should not throw, nor use the GC. I don’t see a problem here. Am I missing something?

[dukc]

Since old code won't break unlike how I assumed, yeah I guess nothrow is justified. I still do question @nogc though. An ideal hash function can work in stack only, but it may sometimes be easier to use the heap for the hash calculations. Requiring no garbage collection might set the bar a bit too high.

[moon-chilled]

Perhaps, if (Ordered o2 = cast(Ordered)p2) return -o2.cmp(p1) ?

[moon-chilled]

Why not 'bool'?

[Imperatorn]

Yes, please document why int or change too bool

[ntrel]

Object.opEquals returns bool:
https://dlang.org/phobos/object.html#.Object.opEquals

[edi33416]

Yes, this will be bool

[ntrel]

I think Ordered should inherit from Equals.

[aermicioi]

I think in this case Ordered should be renamed to Comparable then. Comparable means that something can be compared to something. It can be less, equal or bigger, while Ordered in my understanding, is something that is before something, at same level, or after it.

[ntrel]

I have made some minor tweaks here: https://github.com/Robert-Aron293/DIPs/pull/1

[rikkimax]

The names chosen may conflict with existing libraries or user code.

It may be needed to prefix the names of the interfaces into something like IProtoHash.

[JohanEngelen]

The DIP needs to contain a discussion about the performance impact of the interfaces proposed (e.g. Equals). It may add significant overhead, what immediately comes to mind:

• data layout and size of class object (especially in the case of several interfaces, offset of user data and cache line size.
• virtual calls through interfaces, and the cost of dyncasting in the default implementation shown. How opaque is this to a (theoretical) optimizer?

[ntrel]

This DIP might need to explain why Java and C# have a generic interface whereas this proposes a non-generic interface Ordered and runtime type checks.

[andralex]

Java's Object.equals method predates generics. They later introduced https://docs.oracle.com/javase/8/docs/api/java/lang/Comparable.html, which is a good prior art to cite as evidence backing up this DIP.

[adamdruppe]

Your abstract justification is virtually all factually false and moreover, even if it were true, ProtoObject does nothing to change that. Fixing most these issues is trivially easy with existing Object, even without breaking changes. (And even if you were to do the breaking changes, they'd probably be trivial to fix.)

Every class defined in the D language has Object as the root ancestor.

Not true. There's COM and C++ classes that don't.

Object defines four methods: toString, toHash, opCmp and opEquals; at a fist glance

"fist" typo

As a consequence, these methods make it difficult to use Object with qualifiers or in code with properties such as @nogc, pure, or @safe. We propose the introduction of a new class, ProtoObject, as the root class and ancestor of Object. ProtoObject defines no method and requires the user to implement the desired behaviour through interfaces

Technically, those properties don't work with Object per se, but since you can add them to other interfaces or child classes (remember, you can always tighten restrictions not present in the base), their presence in Object are irrelevant. And since ProtoObject also requires you to add interfaces or child classes, it doesn't actually change anything.

The rationale is much better, actually getting into it. Though I still don't see the need for the new thing. Like it could just remove factory() and the mutex from the current Object. Fixing this breakage in user code would be utterly trivial, you just add the explicit opt in during the deprecation period.

The toString thing is already done in the Throwable class in today's druntime. Today's writeln already uses it if it can. The gc one is easily implemented in terms of the delegate one. It is fair to say the problem of having two is if you override one but not the other, the result can change just based on which function calls it. There's also existing problems with the attributes being dependent on the passed delegate, which is currently difficult to express in terms of an interface (you can with overloads but it is a pain and then you've got even more override potential trouble.) I see no toString in any list of the ProtoObject interfaces, so I guess that just conveniently sidesteps it. It is true, you can just DBI it, but then you have the troubles of that like a seemingly minor typo leads to it just mysteriously not working.

But regardless, if you're sidestepping it anyway, you haven't really gained much over the status quo.

opCmp and opEquals missing const scope is legit, not much we can do about that (except just.... add it and let people's overrides catch up). opHash... meh, status quo doesn't really hurt anything, but the fact it must agree with opEquals is a legit pitfall (though the default implementation of opHash could handle this).

factory indeed sucks, I'd be in favor of removing it entirely. Keeping it there but having a workaround to avoid it doesn't gain as much as just killing it.

Well, that's my objection to this thing existing at all. There's better approaches. But if we are going forward with it anyway, here's some stuff to improve the dip by itself:

Because of the hidden __mutex member, and the fact that the D programming language supports function attributes, the design of Object is susceptible to the Fragile Base Class Problem: this states that a small and seemingly unrelated change in the Base class can lead to bugs and breakages in the Derived classes.

I don't see how. Nothing in current Object is comparable to the problems described in the link. Note that current Object has no state aside from that mutex, and none of the interface methods affect that mutex.

Perhaps an example would be illustrative.

This reconfiguration makes ProtoObject, not Object, the ultimate root of all classes.

ALL classes? Or just extern(D) classes? What about COM and extern(C++)? The reason they aren't compatible right now is a vtable slot incompatibility (D things put TypeInfo in slot 1, they do not), so I imagine they would continue to be separate.

const @nogc nothrow pure @safe scope

I'm inclined to agree with the others that this is probably over restrictive (personally, I find the lack of these in current Object to be a feature, not a bug. You can always add more later, but you cannot take away what is already restricted.) But in practice it'd probably be fine anyway.

int __cmp(ProtoObject p1, ProtoObject p2)

Why does this function exist? Same on __equals. And the existence of these functions is what actually causes the pain of equality and whatnot in @safe functions today - NOT actually the definitions in Object.

I'll go ahead and tell you: it is to inject a null check. That's it. It is so obj == null will return true instead of segfaulting on the virtual call to opEquals. We didn't used to have this, n00blars complained, object.opEquals (notice the lowercase - this is on the module object, not the class Object) got added. But then @safe is dropped there. That's the root problem with the attribute thing and if these functions weren't there, you'd be fine!

An alternative is to template the call to __cmp, so it uses the concrete static type passed. This avoids the dynamic casts and respects the interface given, attributes and all. I think another saoc participant is looking at this (though I don't know if they're doing these functions in particular).

See, this is why I'm against this whole ProtoObject push. It is based on a flawed analysis from the beginning. But anyway back to assuming it is the way.

Consider the following:

class Safe {
        override bool opEquals(Object c) const @nogc @safe nothrow pure {
                return this is c;
        }
}

void main() @safe nothrow pure {
        Safe s = new Safe();
        Safe b = s;

        assert(s == b);
}

You get a spew of errors. But notice the root:

demo.d(11): Error: pure function D main cannot call impure function object.opEquals

The module, not the class. Change it to:

        assert(s.opEquals(b));

And now it complies without trouble.

The covariant parameter thing is a legit problem still - I had to take Object instead of const Safe like I wanted to.... or did I? See if I just removed the override keyword, I could take const Safe. It'd be treated as an overload instead of an override, but then the explicit call to opEquals on Safe in particular WOULD work, since it goes through the new interface, not the old one in Object.

If we had a template top-level opEquals:

bool opEquals(C)(C c1, C c2) {
        if(c1 is null || c2 is null)
                return c1 is c2;
        return c1.opEquals(c2);
}

Well well, we have null checks, we can overload (not override) on the const Safe parameter, and whether overloading or overriding, the attributes are now inferred back out to the usage point:

assert(opEquals(s, b)); // works in nogc etc

What happens if you compared to Object? Well, then the template would use the common base type - Object - and be unable to infer the attributes. But that's fine, if you want the more strict interface, just use the more strict interface in your static type; pass Derived instead of Base. Then it all just works.

So the real problem is object.opEquals, NOT Object.opEquals.

If we're switching to ProtoObject.... maybe it should actually fix the real cause while you're at it. Change those module-level functions to take the types actually passed to it without the intermediate implicit cast. This eliminates the disgusting internal dynamic cast, allows user-defined attributes to be added as-needed, maintains the convenience null check at the boundary, and just as the cherry on top, works with normal Object-derived things too.

On a similar vein, I think defining the Ordered interface to take ProtoObject is just silly. Obviously you aren't ordered in relation to any other random object. So why force people to pretend they are and then do an internal runtime cast? Make the thing you're ordered against as part of the interface.

interface Ordered(What) {
   int opCmp(const scope What  w);
}

Then you say

class MyNumber : Ordered!MyNumber {
   int n;
   int opCmp(const scope MyNumber w) {
        return this.n - w.n;
  }
}

Much nicer to implement, and the static type is now meaningful - you can't compare it against a HttpClient so it won't even accept that through the static type system.

I have a hard time thinking of a case where you'd want arrays of "comparable" or "orderable" objects that cannot actually be compared or ordered against each other since they're different things. But I suppose I might be wrong here and you do want to sort a ProtoObject[] or something. I guess it could group like objects together or something. idk, but my feeling is this is of little value. You mention later that the Java and C# ones are generics too, which obviously their implementation of generics is different than D's templates (and their type erasure is legit nice in a lot of ways, I'm a bit jealous), the type system still treats it more similarly to a D template.

And with the template, you might be ordered against something that isn't a class at all, like Ordered!int. This interface permits that for virtual interoperability. Though I'm again at a bit of a loss as to when you'd actually use that through the interface.

Implementations of Ordered, Equals and Hash must agree with each other.

Is there any way to enforce, or at least encourage that through design? If not, so be it, but it would be nice. But again this seems to not actually have significant improvements over the status quo.

[edi33416]

The names chosen may conflict with existing libraries or user code.

It may be needed to prefix the names of the interfaces into something like IProtoHash.

We have a PR that adds such utilities and so far there aren't any name clashes.

[rikkimax]

The names chosen may conflict with existing libraries or user code.
It may be needed to prefix the names of the interfaces into something like IProtoHash.

We have a PR that adds such utilities and so far there aren't any name clashes.

What does that PR have to do with object.d. A module that gets imported into every other module implicitly?

[edi33416]

hat does that PR have to do with object.d. A module that gets imported into every other module implicitly?

Nothing. You're right. Disregard my previous comment

[adamdruppe]

The interfaces can easily be defined in core.whatever anyway. Makes migration just like a tiny wee bit more obnoxious but I'd be ok with it.

[aermicioi]

Why not make this interface a templated one?
Smth like this:

interface Ordered(T = typeof(this)) {
  ... int cmp(.... T rhs);
}

The __cmp hook can also be a template that will accept only types that extend Ordered iface, of same type.

This would make possible to declare a ordering relationship between two different types, if needed.

[aermicioi]

In case when the interface is not known at compile time, the __cmp hook can try cast the object to a generic version of ordered: Ordered!ProtoObject, and if implemented, then do the comparisons

[aermicioi]

Same question for Equals interface.

[aermicioi]

It might be worth investigating java's builder api for comparators: https://docs.oracle.com/javase/8/docs/api/java/util/Comparator.html as an alternative, that offers fluent style api for declaring how to compare the objects.

[pbackus]

I don't understand why these four interfaces (Stringify, Hash, Ordered, Equals) are necessary in the first place. As far as I can tell from the DIP, the only thing they are used for is implementing druntime functions like __cmp, __equals, and toHash—but a much simpler approach would be to do what @adr suggests and implement these functions as templates.

[rikkimax]

I don't understand why these four interfaces (Stringify, Hash, Ordered, Equals) are necessary in the first place. As far as I can tell from the DIP, the only thing they are used for is implementing druntime functions like __cmp, __equals, and toHash—but a much simpler approach would be to do what @adr suggests and implement these functions as templates.

The problem with Adam's approach is: a child class cannot implement these methods, pass around the parent of said class and then try to do any of these operations. At least without using something like TypeInfo to store function pointers.

[adamdruppe]

The parent doesn't implement the interface, so passing it is (correctly) a static type error.

If you're using it, you need to use the base that actually implements the interface.

[benjones]

Might be worth mentioning swift protocols which are similar to rust's traits approach.

It would be nice if the mixins took alias parameters rather than strings if possible

[Robert-Aron293]

Not true. There's COM and C++ classes that don't.

Yes, I was referring to the D classes.

The rationale is much better, actually getting into it. Though I still don't see the need for the new thing. Like it could just remove factory() and the mutex from the current Object. Fixing this breakage in user code would be utterly trivial, you just add the explicit opt in during the deprecation period.

Introducing ProtoObject provides the users an alternative, while also avoiding the breakage in user code.

ALL classes? Or just extern(D) classes? What about COM and extern(C++)? The reason they aren't compatible right now is a vtable slot incompatibility (D things put TypeInfo in slot 1, they do not), so I imagine they would continue to be separate.

extern(D) classes

int __cmp(ProtoObject p1, ProtoObject p2)

Why does this function exist? Same on __equals. And the existence of these functions is what actually causes the pain of equality and whatnot in @safe functions today - NOT actually the definitions in Object.

I'll go ahead and tell you: it is to inject a null check. That's it. It is so obj == null will return true instead of segfaulting on the virtual call to opEquals. We didn't used to have this, n00blars complained, object.opEquals (notice the lowercase - this is on the module object, not the class Object) got added. But then @safe is dropped there. That's the root problem with the attribute thing and if these functions weren't there, you'd be fine!

An alternative is to template the call to __cmp, so it uses the concrete static type passed. This avoids the dynamic casts and respects the interface given, attributes and all. I think another saoc participant is looking at this (though I don't know if they're doing these functions in particular).

See, this is why I'm against this whole ProtoObject push. It is based on a flawed analysis from the beginning. But anyway back to assuming it is the way.

Consider the following:

class Safe {
        override bool opEquals(Object c) const @nogc @safe nothrow pure {
                return this is c;
        }
}

void main() @safe nothrow pure {
        Safe s = new Safe();
        Safe b = s;

        assert(s == b);
}

You get a spew of errors. But notice the root:

demo.d(11): Error: pure function D main cannot call impure function object.opEquals

The module, not the class. Change it to:

        assert(s.opEquals(b));

And now it complies without trouble.

The covariant parameter thing is a legit problem still - I had to take Object instead of const Safe like I wanted to.... or did I? See if I just removed the override keyword, I could take const Safe. It'd be treated as an overload instead of an override, but then the explicit call to opEquals on Safe in particular WOULD work, since it goes through the new interface, not the old one in Object.

If we had a template top-level opEquals:

bool opEquals(C)(C c1, C c2) {
        if(c1 is null || c2 is null)
                return c1 is c2;
        return c1.opEquals(c2);
}

Well well, we have null checks, we can overload (not override) on the const Safe parameter, and whether overloading or overriding, the attributes are now inferred back out to the usage point:

assert(opEquals(s, b)); // works in nogc etc

What happens if you compared to Object? Well, then the template would use the common base type - Object - and be unable to infer the attributes. But that's fine, if you want the more strict interface, just use the more strict interface in your static type; pass Derived instead of Base. Then it all just works.

So the real problem is object.opEquals, NOT Object.opEquals.

If we're switching to ProtoObject.... maybe it should actually fix the real cause while you're at it. Change those module-level functions to take the types actually passed to it without the intermediate implicit cast. This eliminates the disgusting internal dynamic cast, allows user-defined attributes to be added as-needed, maintains the convenience null check at the boundary, and just as the cherry on top, works with normal Object-derived things too.

Few points here:

1. yes, object.opEquals should use templates
2. ProtoObject provides an alternatives. You can have a chain of classes that define only what you use. For example, if I want a type that is not comparable, then when I try to compare two objects of this type I want to get an error at compile, not at runtime.
3. yes, all functions(equals, cmp, hash, etc.) should use templates
4. ProtoObject does not impose interfaces. We will implement some default interface, but the users aren't forced to use them. The user can define his own.

I'm inclined to agree with the others that this is probably over restrictive (personally, I find the lack of these in current Object to be a feature, not a bug. You can always add more later, but you cannot take away what is already restricted.) But in practice it'd probably be fine anyway.

Users can use the interfaces provided in druntime or define other interfaces similar to those. If you want one that is less restrictive, you can just define it.

On a similar vein, I think defining the Ordered interface to take ProtoObject is just silly. Obviously you aren't ordered in relation to any other random object. So why force people to pretend they are and then do an internal runtime cast? Make the thing you're ordered against as part of the interface.

Interfaces take ProtoObject because we don't want to force the users to use our interfaces. They can use ours or define other, according to their needs.

Implementations of Ordered, Equals and Hash must agree with each other.

Is there any way to enforce, or at least encourage that through design? If not, so be it, but it would be nice. But again this seems to not actually have significant improvements over the status quo.

We were thinking of something similar to this:

traits(compiles, {

T t1 = T.init;

T t2 = T.init;

assert((t1 == t2) && (t1.toHash == t2.toHash) && ((t1 < t2) == (t2 < t1)))

})

[Robert-Aron293]

@mdparker the DIP is ready for review

[pbackus]

The implementations of the global __cmp and __equals functions currently given in the DIP are incorrect.

int __cmp(ProtoObject p1, ProtoObject p2)
{
    if (p1 is p2) return 0;
    Ordered o1 = cast(Ordered) p1;
    if (o1 is null) return -1; // we can't compare
    return o1.cmp(p2);
}

Returning -1 for incomparable arguments is incorrect, because a return value of -1 means that p1 < p2. The only acceptable behavior when p1 and p2 are incomparable is to (a) fail at compile time (preferable) or (b) throw an exception.

bool __equals(ProtoObject p1, ProtoObject p2)
{
    if (p1 is p2) return true;
    Equals o1 = cast(Equals) p1;
    Equals o2 = cast(Equals) p2;
    if (o1 is null || o2 is null) return false;
    return o1.equals(o2) && o2.equals(o1);
}

Returning false when casting one or both of the objects to Equals fails is incorrect, because a return value of false means that p1 != p2. Again, when p1 and p2 are incomparable, this function must either fail at compile time or throw an exception.

[pbackus]

yes, all functions(equals, cmp, hash, etc.) should use templates

If the intent is for these functions to use templates, then the DIP should be revised to reflect that. Currently, __cmp and __equals are specified in the DIP as non-template functions.

[adamdruppe]

On Fri, Nov 12, 2021 at 07:44:56AM -0800, Robert-Aron293 wrote: Yes, I was referring to the D classes.

The DIP text should be precise.

Introducing `ProtoObject` provides the users an alternative, while also avoiding the breakage in user code.

My big problem with this is that it trades long term pain for short term convenience. That's a bad deal.

1. yes, object.opEquals should use templates

We should probably just go ahead and make that change. It doesn't even change the language.

3. yes, all functions(equals, cmp, hash, etc.) should use templates

Those other functions probably simply shouldn't even exist. You can just call the method directly as needed - no need to do those ridiculous dynamic casts at all.

Interfaces take `ProtoObject` because we don't want to force the users to use our interfaces. They can use ours or define other, according to their needs.

This gives up static type safety and provides no actual generic benefit (you still cast to your interface, not to the user interfaces, so they cannot actually define other things according to their needs). Deleting these functions entirely would be more useful. Then the user would define their own functions that work in their own interface.

[12345swordy]

Deleting these functions entirely would be more useful. Then the user would define their own functions that work in their own interface.

Your solution will break existing code. Which apparently @andralex here wants to avoid.

[adamdruppe]

On Sun, Nov 14, 2021 at 12:50:45PM -0800, 12345swordy wrote: Your solution will break existing code.

If removing the functions is not acceptable (and frankly, there's no rational reason why not, since the migration is *utterly* trivial and a one time investment for greater long term returns, whereas this proposal avoids the small current cost in favor of a recurring long term cost), you can also just leave them alone. Their presence doesn't actually break anything.

[Bolpat]

@pbackus wrote:

The implementations of the global __cmp and __equals functions currently given in the DIP are incorrect.
...

Can we please admit that not all types that are ordered have a (total) linear order? If we want to take this seriously, please let's not misstep on the first meter. Ordering is difficult. What is intuitively an order can be a very, very general construct.
Take sets of floating point values for example. There's no standard name to refer to their most natural order. "Partial-partial preorder" sounds funny, but matches (my) mathematical intuition quite well. A preorder is a transitive and reflexive relation, so far, so easy. However, floats' <= is not fully reflexive: !(float.nan <= float.nan); nobody here invented it, but it's here and has this property, whether you like it or not. So, that's where one "partial" comes from: a <= a under the condition that a <= b or b <= a for some b. This makes a == b (defined by a <= b && b <= a) a partial equivalence relation which is what it should be. It is not an equivalence relation because of NaN and it is not a (partial) equality because a == b does not imply f(a) == f(b) as +0.0 and -0.0 behave slightly differently and yet +0.0 == -0.0.
Float values are linearly ordered insofar as: a <= b || b <= a under the condition that a <= a and b <= b. So "linear-partial preorder" kinda makes sense. But sets of things are not linearly ordered even if the things are, so sets of floats are partially ordered (versus linearly ordered). So, replace "linear" by "partial" and there's where you get "partial-partial preorder" from.

TL;DR: Don't assume orderings are nice and ugly orderings are few and niche.

[andralex]

This is a Romanianism I often use too: repeated references to "it". English only tolerates short-range references to "it", and seldom twice in the same sentence. In addiotion, here the two uses of "it" refer to distinct things: first time it's about the compilation, second time it's about the function if I understand correctly.

[ntrel]

I've now fixed this and 2 other minor fixes from your comments in https://github.com/Robert-Aron293/DIPs/pull/1

[andralex]

Was brought here by a rant that was since deleted, only to find @adamdruppe 's :). I agree with his comment that DIPs need to be precise ("every statement in the DIP should be defensible in a court of law in front of a lawyer who'd make one million dollars if they ruined you, representing your vengeful ex"). A simple corollary is, whatever needs explanation and qualification, don't put it here, paste it in the DIP instead.

As to what changes are acceptable and what aren't, my take is that if anything easier was possible (such as removing methods of Object), it would have been done by now because it's easiest implementation-wise. This DIP is immediately actionable and breaks a long-overdue stalemate. Anything and everything against it arguing X would be easier should mind the simple "but why hasn't X been done in the past 15 years?" comeback.

[adamdruppe]

Well, it is obvious that not many people have even correctly diagnosed the problem - this DIP (and all the associated dconf talks over the years) certainly fail to do so.

I'll fix it myself over the Christmas break perhaps.