[RFC] Interfaces in Nim

[dom96]

I've recently had a discussion about this with @Araq, so this is more of a reminder for myself and him, but comments are still welcome.

Araq's ultimate goal for Nim is to have a simple language with a powerful macro system. As such, interfaces in Nim should be implemented using macros.

Goals:

• Compatibility with the "fake" interface used in the streams module

Potential further research:

• Look into some of the many OOP/Interface packages to see how they have been implemented. Perhaps with the idea to adopt them into the stdlib in mind.

[andreaferretti]

I have one, https://github.com/andreaferretti/interfaced to which I have not contributed much, if anything. Actual work done by @krux02 and @RSDuck. Example of usage:

import interfaced

type
  Human = ref object
    name: string
  Dog = ref object

proc makeNoise(human: Human): string =
  "Hello, my name is " & human.name

proc legs(human: Human): int = 2

proc greet(human: Human, other: string): string =
  "Nice to meet you, " & other

proc makeNoise(dog: Dog): string = "Woof! Woof!"

proc legs(dog: Dog): int = 4

proc greet(dog: Dog, other: string): string = "Woof! Woooof... wof!?"

createInterface(Animal):
  proc makeNoise(this: Animal): string
  proc legs(this: Animal): int
  proc greet(this: Animal, other: string): string
    
proc interact(animal: Animal) =
  echo animal.makeNoise
  echo animal.greet("James Bond")

proc interactAll(animals: varargs[Animal, toAnimal]) =
  for animal in animals:
    animal.interact()

when isMainModule:
  var
    me = Human(name: "Andrea")
    bau = Dog()

  for animal in @[me.toAnimal, bau.toAnimal]:
    echo "Number of legs: ", legs(animal)

  interactAll(me, bau)

A nice addition (which is not there right now) would be to link interfaces to concepts (somehow derive one from the other)

[zielmicha]

Existing OOP/interface macros:

• https://github.com/jyapayne/nim-extensions/blob/master/oop_macro.nim
• https://github.com/flaviut/nim-by-example/blob/master/content/content/oop_macro.md (the same thing?)
• https://github.com/zielmicha/collections.nim/blob/master/collections/iface.nim
• https://gist.github.com/PhilipWitte/f4f0c18f7f6436ee87dc3c506343d859

I have tried to implement elegant interfaces using macros. (https://github.com/zielmicha/collections.nim/blob/master/collections/iface.nim). It works, but requires interface definition outside of type section:

type Duck = distinct Interface
interfaceMethods Duck:
  quack(number: int): string

Unless I'm missing something, macro system has a limitation - you couldn't emit code in type section.
For example, this won't work:

template bar(): untyped =
  proc fooFunc(): int =
    return 5

  int

type Foo1 = bar()

let a: Foo1 = fooFunc()

[zielmicha]

What about planned vtref? Are they cancelled?

[Araq]

What about planned vtref? Are they cancelled?

Given the current status of concepts, they are too fragile to base yet another complex feature on top of them. They are delayed, as far as I'm concerned. Probably @zah will chim in and disagree. :-)

[andreaferretti]

Maybe it would be useful to take a simple example of usage and express it using the different libraries in order to compare them

[krux02]

@andreaferretti whay are there ref types? I made sure that you don't need ref types for the interfaces.

Aparently the inteface libary has been spoiled with ref types, I would like to point you to the original version that I wrote in the forum that does not need any ref types at all:
https://forum.nim-lang.org/t/2422

[RSDuck]

I changed it to ref types, because otherwise there would be the risk of a dangling pointer. It could probably be done better, but since being ref type is also required for dynamic dispatch via method, I thought it wouldn't be that bad.

[krux02]

@RSDuck probably we should discuss this somewher else, but it is "that bad". I desigend the interfaces carefully, so that they can be used without any GC active. I am very careful about this, I don't want any ref types in my rendering thread. With your changes I can't use interfaces at all anymore in my rendering thread. And can you elaborate on "dangeling pointers"? Yes I use pointers internally to implement it, but so does C backend of Nim. The interface object is not supposed to keep anything "alive" it is supposed to be like a pointer.

[andreaferretti]

The issue tracker of interfaced can be a good place to discuss this

[Araq]

I think we can and should give the interface macro a parameter to either use ref or ptr under the hood.

[krux02]

Araq: well I don't think that's necessary. The macro can check if it is based on a ref type or anything else. My original implementation just didn't care for ref types, but I don't think it is necessary to have an additional parameter.

[Araq]

Maybe, but whether it produces a converter or not should also be optional.

[zah]

These library-based approaches will become unnecessary once the VTable types are ready, but otherwise I'm not against them especially if they will give us some experience.

Some aspects of the VTable types are already implemented. In particular @Araq hasn't merged yet one of my patches enabling the "Converter Concepts" feature, which can be used here as well to get a bit easier to use interface for the interface-based procs :) (i.e. you can have a converter concept automatically calling toAnimal at the call-sites). I'll rebase the patch soon.

[Menduist]

I posted a $1000 bounty for this issue: https://app.bountysource.com/issues/68171483-rfc-interfaces-in-nim

This is the feature I miss the most for nim, but it doesn't seem to get much traction, so I'm creating some :)
(Obviously, for a native solution, the library-based one seems too limited)

[daiyam]

Interfaces can almost be implemented with concept but there are several issues:

• varargs don't like concepts
• @[x, y] don't detect the common concept

type
  Human = ref object
    name: string
  Dog = ref object

proc makeNoise(human: Human): string =
  "Hello, my name is " & human.name

proc legs(human: Human): int = 2

proc greet(human: Human, other: string): string =
  "Nice to meet you, " & other

proc makeNoise(dog: Dog): string = "Woof! Woof!"

proc legs(dog: Dog): int = 4

proc greet(dog: Dog, other: string): string = "Woof! Woooof... wof!?"

type
  Animal = concept x
    x.makeNoise() is string
    x.legs() is int
    x.greet(string) is string

assert(Human is Animal, "Human is Animal")
assert(Dog is Animal, "Dog is Animal")

proc interact(animal: Animal) =
  echo animal.makeNoise
  echo animal.greet("James Bond")

# proc interactAll(animals: varargs[Animal]) =
#   for animal in animals:
#     animal.interact()

echo Human is Animal  # true
echo Dog is Animal    # true

when isMainModule:
  var
    me = Human(name: "Andrea")
    bau = Dog()
  
  me.interact
  bau.interact
  
  # for animal in @[me, bau]:
  #   echo "Number of legs: ", animal.legs()

  # interactAll(me, bau)

[konsumlamm]

Interfaces can almost be implemented with concept but there are several issues:

• varargs don't like concepts
• @[x, y] don't detect the common concept

These are the same issue: concepts don't support vtables, they're just constraints. For some concept A, you can't have a value "of type A". If you have a arg: A parameter to a proc, that's just sugar for a generic: arg: T where T: A is a generic type parameter.

[daiyam]

With

type
  MAnimal = ref object of RootObj
  Human = ref object of MAnimal
    name: string
  Dog = ref object of MAnimal

proc makeNoise(human: Human): string =
  "Hello, my name is " & human.name

proc makeNoise(dog: Dog): string = "Woof! Woof!"

type
  CAnimal = concept x
    x.makeNoise() is string

If we need type Animal = vtref CAnimal to transform the concept as the type, then why currently, the following code is working:

assert(Human is MAnimal, "Human is Animal")
assert(Dog is MAnimal, "Dog is Animal")
assert(Human is CAnimal, "Human is Animal")
assert(Dog is CAnimal, "Dog is Animal")

proc interact(animal: CAnimal) =
  echo animal.makeNoise

In those examples, CAnimal has the same behavior as a referenced object's type.

So I don't see any need for vtables/vtref. It has been at least 5 years since that feature is been discussed, it need to move forward.
Or I did misunderstood/missing something, maybe.

• proc interactAll(animals: varargs[Animal]) has to be supported since proc interact(animal: Animal) is supported.

• @[x, y] don't detect the common concept

We can't expect the compiler to test all the concepts to find the common concept. We need a way to tell that this type or that type are supporting this or that concept.
A solution would to use the keyword implements to do something like:

implements CAnimal for Human:
  proc makeNoise(human: Human): string =
    "Hello, my name is " & human.name

proc makeNoise(dog: Dog): string = "Woof! Woof!"

implements CAnimal for Dog

implements would validate the type against the concept and add the link to a table or map so that @[x, y] can be supported.
(I would prefer impl but it isn't a keyword)

I'm ok to implement those changes.

[andreaferretti]

why currently, the following code is working

Because you are using inheritance? :-) Interfaces are a different way to do dynamic dispatch, and some indirection, using vtables or other means, has to be used if you want to derive interfaces from concepts

[konsumlamm]

If we need type Animal = vtref CAnimal to transform the concept as the type, then why currently, the following code is working:

assert(Human is MAnimal, "Human is Animal")
assert(Dog is MAnimal, "Dog is Animal")
assert(Human is CAnimal, "Human is Animal")
assert(Dog is CAnimal, "Dog is Animal")

proc interact(animal: CAnimal) =
  echo animal.makeNoise

Because that is sugar for generics:

proc interact[T: CAnimal](animal: T) =
  echo animal.makeNoise

You can't have a variable of type CAnimal, because it's not an actual type, just a constraint.

So I don't see any need for vtables/vtref. It has been at least 5 years since that feature is been discussed, it need to move forward. Or I did misunderstood/missing something, maybe.

• proc interactAll(animals: varargs[Animal]) has to be supported since proc interact(animal: Animal) is supported.

• @[x, y] don't detect the common concept

Yes, you misunderstood how concepts currently work. You can't have an array/seq of different types implementing the same concept, just like you can't have an array/seq of ints and strings. That would require converting them to some common type (which would be a vtable for the concept).

We can't expect the compiler to test all the concepts to find the common concept. We need a way to tell that this type or that type are supporting this or that concept. A solution would to use the keyword implements to do something like:

A solution to what problem exactly? The problem isn't that the compiler has to "find the common concept".

[konsumlamm]

Since this hasn't been mentioned yet, some prior art (in order of relevance for Nim):

• Go has interfaces, which can be used as constraints for generics, but can also be automatically converted to a vtable (see e.g. https://go.dev/tour/methods/9 and the following examples). They're implemented implicitly, just like Nim's concepts.
• Rust has traits, which can also be used as constraints for generics or converted to vtables by making them into a "trait object" (see e.g. https://doc.rust-lang.org/book/ch17-02-trait-objects.html). Traits need an explicit implementation though.
• Java (and other OOP languages, like C#, D, Dart, ...) support interfaces, which work like inheritance (every class is a vtable anyway), so you can also have values of some interface type. These languages require to specify the interfaces a class implements when defining it, however.

[daiyam]

Because you are using inheritance?

@andreaferretti No, because my previous example is also working and it isn't using inheritance.

[daiyam]

@konsumlamm Let's say we have type VAnimal = vtype CAnimal, what will be the difference between proc interact(animal: CAnimal) and proc interact(animal: VAnimal)? CAnimal is implicitly used as a type.

In the following code, CAnimal is also implicitly used as a type (result as comment):

var
  me = Human(name: "Andrea")
  bau = Dog()

echo me is MAnimal # true
echo bau is MAnimal # true
echo me is CAnimal # true
echo bau is CAnimal # true

You can't have a variable of type CAnimal, because it's not an actual type, just a constraint.

Oh, I understand that.

A solution to what problem exactly?

To have interfaces in Nim. Interfaces are essentially concepts limited to functions and fields.

Interfaces or traits are used to shared common features between objects. This is why I tried to do with @[x, y]. I know it can't work now but it should with interface or something similar (concept).

What I'm confused is that CAnimal can be used as a type in some cases (proc interact(animal: CAnimal) or echo me is CAnimal) but not in some other cases (proc interactAll(animals: varargs[CAnimal]).
Also, internally, concepts are called UserTypeClass

[daiyam]

@konsumlamm My implements CAnimal for Human: is inspired from Rust which I find fitting for Nim.

[Menduist]

There seem to be a lot of confusion between runtime & compile time here
Basically, if your examples currently works, it means that the compiler is smart enough to find the value at compile time.

This is how it would work under the hood

# How concept currently works
block:
  proc greet(a: CAnimal): string = "Animal: " & a.makeNoise()
  echo dog.greet()
  # compiled to
  proc greet[T: CAnimal](a: T): string = "Animal: " & a.makeNoise()
  echo dog.greet[Dog]() # the compiler knows `dog` is a `Dog`

# A VAnimal would look like this instead
block:
  type
    VAnimal = vtype CAnimal
    # Compiled to ---
    VAnimal = object
      makeNoise: proc: string
  converter toVAnimal(canimal: CAnimal): VAnimal =
    result(makeNoise: proc: string = canimal.makeNoise())
  # -------
  proc greet(a: VAnimal): string = "Animal: " & a.makeNoise()

  echo dog.greet()
  # Compiled to (thanks to the converter)
  echo greet(toVAnimal(dog))
  # Once we have a VAnimal, it's a real value which can be used anywhere (stored in object, etc)
  var s: @[toVAnimal(dog), toVAnimal(human)]
  for an in s: s.greet() # this would be impossible with concepts, since the procedure
  # is used with different types depending on the runtime value

[konsumlamm]

@konsumlamm Let's say we have type VAnimal = vtype CAnimal, what will be the difference between proc interact(animal: CAnimal) and proc interact(animal: VAnimal)? CAnimal is implicitly used as a type.

The difference would be in how the parameter is represented. A CAnimal parameter is actually a generic parameter, so Nim generates a new version of the proc for every type you use it with and you just pass the concrete value to the respective version (the actual type is known at compile-time). A VAnimal otoh would be a vtable (a pointer to the table of methods) and a pointer the concrete value (or one pointer for both).

In the following code, CAnimal is also implicitly used as a type (result as comment):

var
  me = Human(name: "Andrea")
  bau = Dog()

echo me is MAnimal # true
echo bau is MAnimal # true
echo me is CAnimal # true
echo bau is CAnimal # true

It's not used as a type, concepts just also work with is.

You can't have a variable of type CAnimal, because it's not an actual type, just a constraint.

Oh, I understand that.

A solution to what problem exactly?

To have interfaces in Nim. Interfaces are essentially concepts limited to functions and fields.

But you don't need an explicit implements for that.

Interfaces or traits are used to shared common features between objects. This is why I tried to do with @[x, y]. I know it can't work now but it should with interface or something similar (concept).

What I'm confused is that CAnimal can be used as a type in some cases (proc interact(animal: CAnimal) or echo me is CAnimal) but not in some other cases (proc interactAll(animals: varargs[CAnimal]). Also, internally, concepts are called UserTypeClass

It's used as a type in none of these cases, the first is just sugar for generics and the second is builtin.

[daiyam]

@Menduist Thx The under the hood is helpful.

A CAnimal parameter is actually a generic parameter, so Nim generates a new version of the proc for every type you use it with and you just pass the concrete value to the respective version (the actual type is known at compile-time).

@konsumlamm You are right. Multiple generated functions, a sugar for generics. So the compiler already have the list of compatible types with the concept. This is what I was missing...

Please correct me if I'm wrong:

type
    VAnimal = vref CAnimal
    # would be compiled to
    VAnimal = ref object
      makeNoise: proc: string

and

type
    VAnimal = vptr CAnimal
    # would be compiled to
    VAnimal = ptr object
      makeNoise: proc: string

[daiyam]

the library-based one seems too limited

@Menduist What are you missing in the library interfaced?