Static-oriented syntax for invoking instance methods vs Simple static methods

[radeusgd]

The context

The PR #3764 introduced a calling convention that allows to write Foo.method (Mk_Foo 123) as a synonym of (Mk_Foo 123).method (just to note, the status of the static/instance calls before this PR was also unclear, I'm not suggesting to simply revert the PR).

It is not exactly clear why this was introduced, quoting the PR:

because Rust

While such a convention definitely has some advantages in some contexts, in the context of Enso it is causing us quite a bit of trouble and I'd like us to discuss if we can remove it in favour of having separate instance and static methods.

The proposal

I would propose to remove the convention mentioned above and instead allow to define static methods in parallel to instance methods. Which method is called would be decided based on if the left-hand-side is an instance of an Atom or if it is a type - i.e. the method namespace of an instance and a type would be completely independent.

So I would suggest to allow the following code to work:

type Foo
    Value (x : Integer)

    bar self = x+1000
    bar = 123

main =
    a = Foo.Value 42
    b = Foo

    IO.println a.bar
    IO.println b.bar

And print:

1042
123

Currently, it would fail with Method overloads are not supported: Foo.bar is defined multiple times in this module..

The motivation

The existing issues

The current calling convention is a source of multiple issues.

The major issue is centered around problems with to_text (and in the past ==).

Currently, to_text is actually defined in a way that is inconsistent to make it work - according to the rules Some_Type.to_text should be a closure that will take a My_Type and then return a text representation. But to make life easier, it returns "Some_Type" by default - so this calling convention is not respected.

However, if I override to_text on a type, it goes back to a 'default behaviour' and indeed if I do:

from Standard.Base import all
type T1
    Value x
type T2
    Value y
    to_text self = "T2{"+self.y.to_text+"}"

main =
    IO.println (T1.Value 23).to_text
    IO.println T1.to_text
    IO.println (T2.Value 23).to_text
    IO.println T2.to_text

it will yield:

(T1.Value 23)
T1
T2{23}
T2.type.to_text[filename.enso:6-18]

So we can see that it goes back to the standard convention and T2.to_text actually returns a closure that is waiting for the self argument. This is inconsistent with T1.

All this was mentioned in the following issues:

• Error.to_text causing problems Quick-fix for Error.to_text CCE #3357
• IO.println breaking IO.println: Type error: expected str to be Text, but got Function #5961
• the general description of the issue The same method is dispatched for instances of a type and the Atom representing the type #1538 and related Static method calls should not be dispatched to instance methods #5251

Less "type safety"

When I write an instance method:

type My_Type
    foo self = ...

I expect the self inside of foo to have type My_Type. However, with this current convention, I can happily call My_Type.foo 42 and get self : Integer there, causing behaviour that is rather surprising.

Current convention is not used anywhere

To the best of my knowledge, the current convention is not used anywhere in our codebase. Moreover, we are trying to design our libraries and workflows with the Component Browser in mind. It is oriented around executing methods on instances, i.e. it will always do instance.method convention - the CB will not generate Type.method instance calls. So this calling convention is not used and not likely to be used in the future.

Alternative Solutions

I think switching to a simpler calling convention is the most robust fix here and will ensure that there are no inconsistencies.

The biggest motivation though is:
We need to_text on types to work correctly, as we rely on it both in our Test suite as well in some places where displaying error messages to the users.

The trouble is mostly for functions that are expected to be defined both on type and instance. That is to_text and its friends (pretty, to_display_text), maybe to_json and == (as we want to be able to compare types for equality, like if type == Integer then ...).

We could offer a fix just for this limited set of functions, where any override like defining custom to_text will only apply to instances, whereas when dispatched on types this would always call-in to the default Type implementation (as I don't believe we ever need ability to override these methods on a type itself, not on instance). This would ensure that stuff like My_Type.to_text will return a Text "My_Type" even if to_text for the instances of this type has been overridden.

This is the least that we need to make to_text work correctly.

However, to make this work - we are still overriding the calling convention and making to_text (and friends) behave differently - as now I cannot do My_Type.to_text my_instance anymore, since it will evaluate to "My_Type" my_instance resulting in Not_Invocable error.

So we'd be making the language inconsistent with some functions not abiding to this calling convention. That's why IMO the best solution is to be consistent and drop this convention altogether.

[radeusgd]

cc: @jdunkerley @JaroslavTulach @hubertp @Akirathan @wdanilo @kustosz
I will appreciate your opinions on this.

[wdanilo]

Radek, I think the current implementation is correct, as it allows accessing methods from type without passing explicit self type and without creating a lambda like (.foo). From the language perspective, it unifies scopes nicely and I like it. However, I'd like to better understand the problems you describe. Can we have a call on Tuesday about it, please?

[radeusgd]

Sure, I'm happy to chat. I think it may be worth including @JaroslavTulach in the call too (and possibly others as well).

[radeusgd]

I'm not sure if I understand what use-case I may have to 'access methods from type' that is not covered by the lambda?

[JaroslavTulach]

So we can see that it goes back to the standard convention and T2.to_text actually returns a closure that is waiting for the self argument. This is inconsistent with T1.

Yup. That's a problem. I've been trying to make my head around the current behavior of to_text for a while. After a lot of debugging I got to the same conclusion. Calling to_text on a type returns a function that awaits self argument.

I expect the self inside of foo to have type My_Type. However, with this current convention, I can happily call My_Type.foo 42 and get self : Integer there, causing behaviour that is rather surprising.

Once I wrote a blog post another assembly language. With the current syntax we can happily call Enso another assembly language!

The trouble is mostly for functions that are expected to be defined both on type and instance.

Function defined on type and on instance must be defined on Any, right? Then it gets inherited into both....

From the language perspective, it unifies scopes nicely

...and object oriented paradigm sucks. Bringing inheritance into a functional language is the simplest way to ask for troubles!

Brainstorming - Would: If the method is defined on Any, then invoke it with the type as self solve anything? Except removing ability to invoke such methods statically?

Current convention is not used anywhere

It is heavily used in

• Consolidate Vector and Array methods #6218

and it works there quite nicely.

[radeusgd]

Current convention is not used anywhere

It is heavily used in

• Consolidate Vector and Array methods #6218

and it works there quite nicely.

Oh, I missed that - well it wasn't used in the existing codebase as this was not merged yet 😉

At the same time - this is exactly abusing the system to call on type X a method defined on type Y. Which exactly supports your assembly language argument (that I agree 100% with). So I'm not sure if we use this as a justification to keep it. Once any typechecking is added to the language, this will certainly become a problematic hack.

---

Brainstorming - Would: If the method is defined on Any, then invoke it with the type as self solve anything? Except removing ability to invoke such methods statically?

Could you expand on the idea of functions defined on Any having this special static behaviour? Because it sounds interesting, but it's not clear to me what exactly you mean there. Could you give some example of how this should behave on some example types?

[JaroslavTulach]

• this is exactly abusing the system to call on type X a method defined on type Y

The alternative was to move all the Vector methods into a special Vector_Helper and make them static. Both Vector and Array would then delegate to these methods. However @hubertp "insisted" on using this static syntax and calling into Vector methods - Hubert had to fix a bug in January to make it work, so he wanted me to use his bugfix ;-)

[hubertp]

@JaroslavTulach Yeah, that was one nice use-case for being able to call instance methods statically. Could potentially prevent a lot of code duplication but I'm also fine if you chose a different path.
There are obviously other ways to do it so I'm not insisting that this feature is a must-have.

[JaroslavTulach]

idea of functions defined on Any having this special static behaviour?

Right now Type.to_text is treated as a static reference to to_text. However that's not the only interpretation. The other one is based on the observation that everything in Enso inherits from Any. As to_text method is defined on Any, the Type.to_text could also mean invocation of to_text with self=Type. Such invocation would return "Type" text. Similarly evaluating Any.to_text would return "Any" text.

However, if the above interpretation is accepted, then there would be no syntax for invoking Any.to_text statically - no way to get to_text function without applied self argument.

[radeusgd]

Ok, so essentially the idea is that for a Type we invoke to_text statically, never looking at any 'overrides' that instances of this Type may provide, but just using the default Any.to_text implementation, right?

This is essentially what I meant in the Alternative solutions section above:

We could offer a fix just for this limited set of functions, where any override like defining custom to_text will only apply to instances, whereas when dispatched on types this would always call-in to the default Type implementation (as I don't believe we ever need ability to override these methods on a type itself, not on instance). This would ensure that stuff like My_Type.to_text will return a Text "My_Type" even if to_text for the instances of this type has been overridden

I agree it can be a relatively simple solution to the immediate issues we are facing. We need to consider that we would need to apply it to to_display_text and possibly some other (essentially all methods that we expect can be called on a type).

I don't like this solution because it makes method dispatch irregular: Type.foo is sometimes a static reference to an instance method and other times it is a call to a method on the type - depending on what foo is. This makes the language less regular which IMO makes it harder to learn - because you essentially need to know by heart or constantly look in the docs which methods have which behaviour. But I agree it's still better than the current state of things.

[hubertp]

to_text and to_display_text are a bit special though because we have a bunch fallback mechanisms like here or here, to name a few, to make it produce reasonable output on various occasions.

[radeusgd]

to_text and to_display_text are a bit special though because we have a bunch fallback mechanisms like here or here, to name a few, to make it produce reasonable output on various occasions.

Yes, but these are the irregularities of the language that I wanted to discuss - is making to_text behave differently (i.e. disallowing the Type.to_text instance calling convention that we currently allow for all other methods, so that Type.to_text may return a reasonable Text instead of a function) the way to go? Maybe.

If so, we have a bug because, becasue these fallback mechanisms stop working if you override to_text:

from Standard.Base import all

type My_Type
    Value x
    to_text self = "{My_Type: "+x.to_text+"}"

main =
    IO.println My_Type.to_text

will currently print an unapplied function.

---

I think such special handling is one OK way to solve this (we would then need to document it better as I'm not aware of this intent being documented anywhere and fix the bugs like the one above). But it does introduce irregularity (which the user needs to learn which methods are magical and which are not) which is not great (but maybe indeed good enough).

[JaroslavTulach]

Observation

If an instance method x is defined on Any and overriden in SomeType, then SomeType.x has dual meaning. It may either mean:

• invocation of instance method x on self=SomeType as that method is inherited from Any
• static invocation of x where self needs to be provided as first parameter

We need to decide what SomeType.x means!

Test Case

from Standard.Base import all

Any.x self = "Any:" + self.to_text

type My_Type
    Value f1

type With_X
    Value f1

    x self = "With_X:" + self.to_text

main =
    IO.println <| My_Type.Value 33 . x # prints Any:(My_Type.Value 33)
    IO.println <| With_X.Value 44 . x # prints With_X:(With_X.Value 44)
    IO.println <| Any.x # Any.type.x[y.enso:4-24]
    IO.println <| My_Type.x # prints Any:My_Type
    IO.println <| With_X.x # prints With_X.type.x[y.enso:12-32]
    IO.println <| Any.x (My_Type.Value 22) # prints Any:(My_Type.Value 22)
    IO.println <| Any.x (With_X.Value 22) # prints Any:(With_X.Value 22)

Please note that Any.x is the static call and My_Type.x is the instance call!

Solution

If a method x is defined on Any, then the only "static call" syntax is Any.x. Every other form is instance invocation. E.g. My_Type.x invokes the instance with self=My_Type (that's actually the current behavior). The same way With_X.x should be the instance invocation (that has to change).

TL;DR

The answer to initial question is: SomeType.x invokes instance method with self=SomeType for any method x defined on Any.

Thumbs up, if this is OKish. It is a minimal change so, we can turn it into a bug and implement it.

[somebody1234]

prior art for static member access syntax, if it matters:

	"static"	instance
Rust	A::b	a.b
C++	A::b	a.b
Lua	A.b()	a:b()
JavaScript	A.b	A.prototype.b
Kotlin	(A)::b	a::b
Kotlin	A.CompanionNameHere::b	a::b

most of these aren't even close to 1:1 analogies, but it might still be helpful

tl;dr one option would be to use My_Type::to_text to reference the "static" method (in other words, the to_text member of the namespace My_Type) - MyType.to_text will call the instance method Any::to_text (i'm assuming all types are instances of Any and nothing else?)

[wdanilo]

We try to keep Enso simple, as Python. C++ / Rust / etc are not good benchmarks for our syntax. I think that if we would need to introduce a special syntax for static methods, it might be wiser to remove the idea of static methods altogether. However, this needs longer discussion and I'm working on it. I will update this thread in a couple of days.

[radeusgd]

it might be wiser to remove the idea of static methods altogether

Just please keep in mind that a lot of our APIs rely on static methods and so removing them would require a significant redesign of our libraries. If we were to remove static methods, before we do so, we should analyse how we need to rewrite our current usages of static methods, like Vector.new, Table.from_rows, Map.from_vector and many many more.

Removing static methods may make the language simpler, but the cost of making the core libraries less simple may outweigh the benefits.

[somebody1234]

fwiw i think there's a difference between being syntactically simple and being conceptually simple - A::b may not be an ideal solution, but (imo) overloading a.b isn't a great idea either - especially since users may encounter surprising behavior like in the linked issues

[wdanilo]

@radeusgd I don't think removing static methods would be a good choice here! I only told that I think that it would be better choice than having special syntax for it, from the user perspective. I feel that both of these choices are bad, but if we would be forced to chose one of those, I think making the syntax more complex is the worse one. Anyway, these are just early thoughts and before we make any decision we will deeply talk about them and about ideas, no worries.

@somebody1234 being conceptually simple is of course crucial, so that's why we are talking about this issue. And yes, we need to solve it. However, such changes are affecting way more things than it seems initially. For example, changing this syntax would drastically change how you would define type classes in Enso (when we implement them) and it affects long-term language usability a lot.

[radeusgd]

Got one more, very practical example when the current state of affairs fails: #6273 - if a Vector contains a type whose to_text was overridden, calling to_text on that vector will fail with a Type error.

[radeusgd]

And another one: #6297

[JaroslavTulach]

Related long term open issue:

• Equality on types with Any as LHS does not work #5612