Functions that create and return new classes/records (OOP wrappers)

[Ruin0x11]

There are a lot of pre-existing OOP wrappers in the Lua ecosystem. They basically have functions that return tables that act as classes, and there can be much variance between them. Here is an example of one.

local Color = class.class("Color")

function Color:__construct(r, g, b)
   self.r = r
   self.g = g
   self.b = b
end

function Color:__tostring()
   return ("%d %d %d"):format(self.r, self.g, self.b)
end

return Color

The idea is that Color is a special table with all the necessary bookkeeping for inheritance and mixins and similar things. It might also have autogenerated methods like :new(), where a method like __construct() is called internally by the class creation logic to mutate internal state and then returns it as a new "instance" of the class.

The question is, if this were to be converted into tl, then what would the return type of class.class() be? I tried the following:

function create_record(): record
   local rec = record
      a: number
   end
   return rec
end

a = create_record()
local b: a = { a = 1 }

But I got these errors:

c.tl:1:27: unknown type record
c.tl:5:11: in return value: got type {a: number}, expected record (an unknown type)
c.tl:8:1: unknown variable: a
c.tl:9:10: unknown type a
c.tl:9:14: in local declaration: b: got {a: number}, expected a

unknown type record doesn't make sense, because the documentation states the following:

Types in Teal are more specific than in Lua, because Lua tables are so general. These are the types in Teal:
...

• record
• arrayrecord

...

So is record actually not a type in the sense of string or number since it can't be used as a return value?

[euclidianAce]

local foo = record
   -- ...
end

creates a new type named foo, record in and of itself isn't a type (well, it is, but its a type of a type, so ¯\_(ツ)_/¯)
so in your example

function create_record(): record
   local rec = record
      a: number
   end
   return rec
end

rec is a new type, and types in Teal are not first-class values, so they can't be returned.

So is record actually not a type in the sense of string or number since it can't be used as a return value?

A record is a type in the sense of string or number, in that you can't return the literal type string or literal type number

There currently isn't a decent work around for this as OOP is kind of an open discussion/problem right now. Perhaps something could be hacked together with generic records but it would definitely feel clunky.
see #97 for more info

[Ruin0x11]

I see, thanks for the information.

In my opinion, I don't think that tl should come with its own class model, should and instead allow people to declare that a certain library is responsible for the class creation, such that tl only interfaces with how they're created. I think it would be neat if types could be first-class, but it's probably a lot of work to do that.

Thanks for your work. I'm really hoping this will end up as the definitive "Typescript for Lua" that I've been wanting for a long time.

[euclidianAce]

No problem 😄. Just to be clear, @hishamhm is the one who created Teal, I just contribute here and there, but he's definitely the one to thank as he's put in the brunt of the work.

[Ruin0x11]

I hacked on an example of the kind of feature I'm thinking of.

https://github.com/Ruin0x11/tl/tree/metaprogramming

Basically I edited the compiler to have a new global function which takes a string identifier and a record, and returns a new type. The expressions containing the arguments to the class function call are saved on the type itself as special metadata. Then I added some hooks in the compiler that get called if a method gets declared on the type later.

This is a custom hook specifically designed for the OOP wrapper I use. It autogenerates a new method on the class type if it detects an init method being assigned to the type's record.

local ClassMetadata = record
   name: string
   typedef: Type
end

on_assign_method = function(self: Type, node: Node, fn_type: Type, a_type: function(t: Type):(Type), node_error: function(node: Node, msg: string, ...:Type):(Type))
   local fn_name = node.name.tk
   if fn_name == "init" then
       local metadata = self.metadata as ClassMetadata
       local name = metadata.name
       local props_type = metadata.typedef
       for k, field in pairs(props_type.fields) do
          self.def.fields[k] = field
          table.insert(self.def.field_order, k)
       end
       fn_type.args[1].type = self
       fn_type.rets[1] = self
       self.def.fields["new"] = a_type {
           typename = "function",
           typeargs = fn_type.typeargs,
           args = fn_type.args,
           rets = fn_type.rets,
           is_method = true
       }
       table.insert(self.def.field_order, "new")
   end
end

After these changes, this program now typechecks:

local fields = record
    min: number
    max: number
end
local IntGen = class.class("IntGen", fields)

function IntGen:init(min: number, max: number): number
   self.min = min
   self.max = max
end

function IntGen:pick(): number
   return math.random(self.min, self.max)
end

local gen = IntGen:new()
local n = gen:pick()
print(n + n)

Again, only a hack, but this is the kind of thing I'm thinking of having. It would be some kind of way of hooking into the compiler to tell it how to interpret a function call that returns a type, because there is a lot of variance between the different OOP wrappers available for Lua. New syntax would probably needed to declare the on_assign_method function, like so:

local ClassType = metatype
   on_declare = function(self: table, name: string, fields: record) ... end
   on_assign_method = function(self: table, node: tl.Node, fn_type: tl.Type) ... end
end

local class = record
   class: function(name: string, field: record): ClassType
end

[hishamhm]

@Ruin0x11 That is super interesting. Compile-time metaprogramming is a super powerful tool and might indeed be the right tool for the problem here. I gave your branch a first look and will continue to think about this!

(On a bit of an offtopic note for this particular issue but still on the topic of language evolution, would appreciate your impressions on #194 !)

[Ruin0x11]

Another thing I thought of recently:

I think Lua is uncommon in programming languages in that it does not come with object-oriented programming support built-in. Even Javascript got its own class syntax because it was such a requested feature. Instead, people who want support for OOP need to bolt it on by using metaprogramming instead. This means that multiple different implementations of OOP already exist for Lua.

I think it may be difficult to capture the specific quirks of every custom OOP system if people want Teal versions of those libraries. Some systems support mixins, but some don't. Some support inheritance or interfaces, and some don't. I was weird and added in a delegation system that hacks into __index to forward properties of child objects to their parents (which was too confusing to be practical, but it worked). If Teal gets its own built-in syntax that declares "a class is defined like this" that's incompatible with those existing systems, it might be more difficult to port them to Teal.

I think the "metaprogram in your own class syntax" approach is one way of getting around this. The problem is that you can mix libraries that use incompatible OOP systems together, so you'd have to know which version of class belongs to what file. It's probably not solvable in the general case if metaprogramming is more flexable than a class system. Although, there is prior art in Lua for bridging between different OOP systems (https://github.com/bartbes/Class-Commons).

Another option would to just leave all the old OOP systems behind and introduce Teal's one class system like Javascript did, and continue until something that absolutely cannot be represented with that class system is found, then iterate. I think JS succeeded in its class adoption because the syntax was an official extension, not an offshoot of an existing language like Teal.

[lenscas]

Maybe I am missing something obvious (I probably am as I tend to avoid OOP in lua) but...

Does teal need to care on how a lua type managed to do its inheritance? If you just want to use an Admin type as a User, you don't, at least for as long as they come from the same library. So, for those cases, Teal can just pick a method and call it a day, for as long a .d.tl files can use it.

The only restriction would be inheritance from external types. That can be solved by allowing types to be tagged as "manual". This would Prevent teal from adding its own OOP code on this type and instead it is up to the programmer to get the OOP working. A manual type can only be extended by another manual type. Perhaps it can also lift any restrictions that the teal version of inheritance opposes in the inheritance tree (So, if normally a type can only extend 1 other type, a manual type can implement as many as it wants).

This way when writing pure Teal code you don't have to know how teal does it.
When using a lua library you don't have to know how teal does it most of the time
Only when extending a type from a lua library, do you need to care on how. But in those cases it matters on how the library does it, and not how teal is doing it or how to make the 2 compatible.

From there it is possible to create extra tags if desired that change the generated OOP code to other variants that are popular (and can only be extended by types with the same tag, and/or the manual tag) and/or add meta programming tricks to help with the plumbing of working with manual types.

[Ruin0x11]

Thinking about this a bit more:

Maybe Teal doesn't have to care about how an individual OOP system is implemented, delving into every detail of metatables and overloaded __index methods. If programmer error can be accepted, maybe all that Teal needs is a way to specify a function that takes some Lua table returned by require, sees if it looks like a thing that a known OOP wrapper can return, and then does the necessary work to allow the compiler to interpret the table as a proper type.

For example, irrespective of how inheritance can be implemented, the expectation is that somewhere, the OOP library will store some information about what interfaces/classes were inherited from in a class that's returned by require. If the information about those classes can be accessed, then there could be some way of munging all of that into what Teal ultimately sees as a record. Records are just tables, after all. The compiler could just ignore the complexities of everything the metatable could provide on said table and assume that nothing weird will happen when indexing fields normally, limiting the scope to what the programmer intends to model with their hand-rolled OOP systems. In exchange for trusting the programmer to specify everything correctly, you'd get Teal's type checking on top, which is the end goal.

Of course, this wouldn't catch everything that you could program into a metatable. You could do something evil like overload __index to return a string or number on a plain field dependent on the result of math.random(). That kind of thing probably couldn't be modeled using the field on a record. But I don't think it's very useful to overload fields like that in the first place.

So what would this kind of proposal look like in the real world? The problem with the definition syntax is that Teal requires that you declare a record's methods and fields up front, and you can't add any more after you've defined it. That means you have to rewrite a lot of Lua code that assigns methods to the class like function Class:foo(bar) ... end. It would be so much easier to make the source Teal-compatible if all you had to do was add the type signature to each method defined like this. If I could dream up a syntax that's close to ideal, it might look like this.

local Color_ = class.class("Color")
local record Color<Color_>
   r: integer
   g: integer
   b: integer
end

function Color:__construct(r: integer, g: integer, b: integer)
   self.r = r
   self.g = g
   self.b = b
end

function Color:__tostring(): string
   return ("%d %d %d"):format(self.r, self.g, self.b)
end

return Color

The extra parameter to record is a table. This table would be the result of calling the OOP library to do things like "define this new class", "inherit all these mixins/interfaces/whatever" and all the other weird things that Teal might not be able to model by itself.

There are still things like autogenerated methods created on the Color_ table that the Teal compiler won't see by just parsing the source file. The actual logic to ensure everything is found in the type checking stage could be declared in the build.tl file. You'd write a shim function that gets called during type checking each time a require happens that detects if the table the compiler got back looks like something returned from an OOP library, in this case the function class.class(). Then it would provide all the information the compiler would need to ensure type checking works.

-- build.tl
local class = require("build/shims/class")

return {
   class_shims = { class }
}

-- build/shims/class.tl
local function shim_for_class_class(t: tl.Type): boolean
   if this_looks_like_a_class(t) then
      -- In this example, `class.class()` adds a static :new() method that Teal doesn't see. We help out the compiler by indicating it exists here.
      -- This is the same thing as saying `function new(...)` in a `record` declaration, but with the correct arguments programmatically generated based on the table returned.
      -- It is the programmer's responsibility to ensure everything matches up with what the metatable will return.
      local _construct = tl.get_method(t, "__construct")
      tl.add_method(t, "new", construct.args)
      
      -- Perhaps class.class() also adds a `__name` field, containing the class name for purposes like reflection.
      -- This is the same thing as saying `__name: string` in a `record` declaration.
      tl.add_field(t, "__name", tl.Types.String)

      return true 
   end

   return false
end

return shim_for_class_class

The setup would be specific to each project and would not try to generalize, because there are infinitely many ways an OOP system could be implemented in Lua. But since you could reuse each shim between projects for the most popular OOP systems, it probably wouldn't be much of an issue.

This next part is only a "nice to have", but for the sake of user experience, perhaps what Teal could provide on top of this system is some default OOP implementation using metatables that works for 90% of cases. All this would mean is adding some extra keywords for the common things people demand like class, interface and extends/inherits, having the compiler include a small library to implement them with metatables, and making a shim to lift the tables into Teal types (as well as things like checking if interfaces are properly implemented, etc.). In keeping with Teal's minimalism, the default implementation would't have to include that many features. There would be nothing special that keeps track of extra class/interface features in the compiler, and everything would just look like a record. Internally this would be like prepending teal_default_shim to the class_shims field of build.tl. The "shim" feature would only be used for people wanting to migrate their OO Lua code to Teal without breaking the specific behaviors of their program's OOP library, or people for whom Teal's OOP library lacks the features they want (hopefully Teal's OOP library could provide enough features so that this complaint would be uncommon).

local interface ISized
   function size(table): integer
end

local class Rectangle implements ISized
   x: integer
   y: integer
   w: integer
   h: integer

   function new(self: Rectangle, x: integer, y: integer, w: integer, h: integer)
      self.x = x
      self.y = y
      self.w = w
      self.h = h
   end

   function size(self: Rectangle) return self.w * self.h end
end

One downside of this approach that I can see is it would prevent things like rect is ISized. Maybe Teal would ultimately need some generic way of testing if a type extends another type. Or maybe, like Class-Commons, Teal would define a contract that each OOP thing needs to fulfill (for example, the required functions/types on interfaces) so you could do things like call implements on a class and an interface that originate from two entirely different OOP systems.