[vim9class] dunder methods and system integration

[errael]

Here are some points from @yegappan

1. The Vim built-in types like List, Dict and Blob support functions like len(), string(), empty(), index(), etc. For a user defined type using classes, we need to support these builtin functions.
2. We also need to support functions like map(), filter(), reduce(), etc.
3. There are also built-in operators (e.g comparison) that are supported for built-in types.
   We need to extend these to classes.
4. The dunder methods enable us to do the above and are designed to be called only by the Vim core and not directly by the script/method/user.
5. The dunder method signature will be validated during compilation.

Dunder methods are first addressed in #13238. Their usage will evolve, use this discussion to weigh in.

[errael]

Here's some additional things I hope find their way into vim9class. @ychin mentioned that

can implement dunder methods first, and then have interfaces [later]

It's probably important to know if the above is a correct statement

The Vim built-in types like List, Dict and Blob support functions like len(), string(), empty(), index(), etc. For a user defined type using classes, we need to support these builtin functions.

These types are primarily containers; in contrast, some of the builtin functions are applicable to any type. Some of the builtin functions throw an exception if not used with the proper type, for example len(), index(). Others, for example empty(), can be used with any type.

vim9script programmers might want to think about the container like things as in many strongly typed languages; as interfaces or traits.

Here are some things in particular that might be useful

• system defined dunder interfaces, e.g. interface __Container, made up of dunder methods
• extending dunder interfaces and/or including dunder methods in an interface

With these the following becomes possible

• programmer can extend, and so insure compliance, as pointed out by @ychin
• define functions that only accept containers as arguments (like some current builtins)
• test if an object satisfies the container contract using instanceof().

It would be nice if some builtin types, e.g. list or dict, acted like class/object, for example as though they implement __Container; and could do some_list_var->instanceof(__Container). Or pass a list to def F(container: __Container)

[dkearns]

I don't understand what drove this particular implementation but it seems to me the natural approach is a common base class and interfaces. I'd strongly prefer it wasn't added as another unnecessary mechanism.

The dunder methods ... are designed to be called only by the Vim core and not directly by the script/method/user.

Why are they designed in this manner?

[yegappan]

I don't understand what drove this particular implementation but it seems to me the natural approach is a common base class and interfaces. I'd strongly prefer it wasn't added as another unnecessary mechanism.

Are you referring to using a base class like "Object" for all the objects and adding these methods to the base class? The base class for all
the objects will be "Object"? From a user perspective, how will it make a difference whether these methods
are part of a base class or simply reserved methods? In both cases, the method signature will be enforced.

The dunder methods ... are designed to be called only by the Vim core and not directly by the script/method/user.

Why are they designed in this manner?

To differentiate these methods from methods that can be used by the user. These methods will also
use a name starting with a lower-case letter. The signature of these methods will be validated when
parsing the class.

[errael]

Are you referring to ...

It does feel a little confusing.

IIUC, the main issue is getting vim9class to interoperate with existing vim builtins and objects. Containers, like list and dict and the builtins they work with, are an important part of that. Two proposals (ignoring naming conventions and under bars) are

1. Use system reserved methods, that a class may define, to enable selective interoperability.
2. Use system reserved interfaces, that a class may implement, to advertise and provide high level interoperability.

It's been pointed out that reserved methods could be done with reserved interfaces coming later.
The advantages of reserved interfaces, at least to me, seem overwhelming as discussed in various threads.

So these questions seem most important

• Is reserved interfaces coming later?
• Is it worth releasing reserved methods if reserved interfaces is on the horizon?

Something like "common base class" is an implementation issue.

[ychin]

FWIW I find dunder methods kind of ugly and I don't love them. I was just pointing out that implementing them doesn't preclude using a interface but I wouldn't particularly vote on using them. I feel that if you look at other languages they all have other ways around this issue.

[yegappan]

If we go with reserved interfaces, how many interfaces will we create? One interface for all the container methods (named Container) like len(), empty(), index(), etc.? Which interface will have the "string()" method? Which interface will have the "lock()/unlock()" like methods?

If the double underscores used for the method names is the problem, then we can remove them and add support for method names starting with a lower case letter only for these reserved methods (as the user defined method names will start with an upper case letter).

[errael]

Here's my take, I'm sure you and others will chime in with preferences, suggestions... It could be a big job, best case there a smallish foundation that could be built upon as things evolve. I'm just throwing a bunch of stuff out there. This is not a proposal, just stuff to consider.

If we go with reserved interfaces, how many interfaces will we create? One interface for all the container methods (named Container) like len(), empty(), index(), etc.?

Yeah, those are the questions. I'd thing one interface for most containers, maybe __Dict extends __Container with some extra magic methods? Would __Container have everything that a list needs?

For a more fine grained approach, can always have something like

interface __Len
    def __len(): number
endinterface

interface __Empty
    def __empty(): number
endinterface

interface __Container extends __Len, __Empty
endinterface

Worst case, if after it's in use, if you need some extra methods, and you want to be compatible, you can define

interface __Container2 extends __Container
# the extra stuff we really need
endinterface

It's kind of a hack, but it does the job.

Which interface will have the "string()" method?

The same one that has __equals()? I guess that's what @dkearns meant by common base class. For example,

interface __CommonBaseInterfaceOrIsThisClassMaybeCalledObjectButIsAutomaticallyAndAlwaysThere
    def __string(): string
    def __equals(other: any): bool
endinterface

Which interface will have the "lock()/unlock()" like methods?

Maybe

interface __Lockable
    def __lock()
    def __Unlock()
    def __Islocked(): bool
endinterface

And if you want to support locking variables, under the class' control, maybe need something like

interface __LockableVariables extends __Lockable
    def __lock(name: string)
    def __Unlock(name: string)
    def __Islocked(name: string): bool
endinterface

I don't like having the string there for variable locking, hopefully there's a better way.

If the double underscores used for the method names is the problem, then we can remove them and add support for method names starting with a lower case letter only for these reserved methods (as the user defined method names will start with an upper case letter).

For me, I have no problem with the '__', that's just syntax.

__ is as good as anything to indicate magic name that is reserved by the system. And it's short and easily identified, both pluses for me. It was the concept of magic methods, rather than magic interfaces, that I had a problem with.

Any spelling/naming-conventions/prefix/capitalization is OK by me.

[dundargoc]

Petition to change "dunder methods" to "dundar methods".

[yegappan]

😄. Dunder methods == Double underscore methods.

[dkearns]

The way things are going you're going to have to rename yourself tundargoc soon.

[dkearns]

I don't understand what drove this particular implementation but it seems to me the natural approach is a common base class and interfaces. I'd strongly prefer it wasn't added as another unnecessary mechanism.

Are you referring to using a base class like "Object" for all the objects and adding these methods to the base class? The base class for all the objects will be "Object"?

Yes. That's how most similar languages work.

From a user perspective, how will it make a difference whether these methods are part of a base class or simply reserved methods? In both cases, the method signature will be enforced.

Simplicity, elegance, type-safety, normalcy and familiarity.

The language already provides all the features needed to implement this. Why add extra compiler magic? Add features common to all classes to a parent base class, any, and provide plain modules with classes/interfaces for anything else.

The dunder methods ... are designed to be called only by the Vim core and not directly by the script/method/user.

Why are they designed in this manner?

To differentiate these methods from methods that can be used by the user. These methods will also use a name starting with a lower-case letter. The signature of these methods will be validated when parsing the class.

I don't understand why they should be differentiated but even if you want to mark them as special I see no reason why they can't otherwise be used like anything else.

[errael]

why they [dunder methods] should be differentiated

They are used by vim builtins, len()/index()/...; is that a meaningful distinction?

[dkearns]

I would be inclined to see the builtins as client code, like any other.

[errael]

In the original post, there is

1. The Vim built-in types like List, Dict and Blob support functions like len(), string(), empty(), index(), etc.  
   For a user defined type using classes, we need to support these builtin functions.

2. We also need to support functions like map(), filter(), reduce(), etc.

3. There are also built-in operators (e.g comparison) that are supported for built-in types.
   We need to extend these to classes.

When you say

I would be inclined to see the builtins as client code, like any other

Are you suggesting that there may be no need for interoperabilty?

[dkearns]

I meant that I don't see anything special about the builtins being the calling code.

It's still unclear to me how integrated these vim9class features are supposed to be in the language and runtime. My questions are not rhetorical. :)

If the idea is just to have Vim9 script (with user classes) then maybe the Python-like model is reasonable. However, I think the magic should really be limited to methods in that case.

Rather than have special methods that are otherwise unusable,

class A
  def __len(): number
    return 42
  enddef
  def Length(): number
    return len(this)
  enddef
  def __string(): string
    return "Forty Two"
  enddef
endclass

an annotation of some sort would be better.

class A
  @builtin(len)
  def Length(): number
    return 42
  enddef
  @builtin(string)
  def _ToString(): string
    return "Forty Two"
  enddef
endclass

[errael]

I don't have a particular use case in mind; I'm not sure how useful this is. But I'd like to be able to do something like (I don't care about the syntax)

class C extends __List
endclass

def TakesBuiltinListOrObjectList(l: list)
enddef

As well as, use a class object with the builtins that operate on a list.
The interface model seems to fit this usage better.
I wonder if a builtin list could be extended?
I don't see a strong reason to have magic interfaces, or magic methods, in vim9.1. (of course that all depends on timing/schedule).