[Feature Request] Implement additional helper types

[SkyyySi]

Note

Just a heads up, this is gonna be a lengthy issue.

Summary

I would like to see some helper types get implemented, which would simplify writing more complex type annotations. Some of these could probably be implemented just using existing LuaCATS syntax, but others most likely need special support directly embedded into LuaLS.

This issue includes markers to track which feature is implemented and which isn't.

The types in question

• never: Marks a function as not returning anything, and marks storing its value as an error (unlike @returns nil).
• Keys<T: table<K, V>>: Extracts K, may create a union type
• Values<T: table<K, V>>: Extracts V, may create a union type
• NonNilable<T>: Removes nil from the union type T, making a value as required.
• Partial<T>: Returns a copy of type/class T, but with each field marked as optional / nilable.
• Required<T>: The opposite of Partial<T>, marking all fields as required / non-nilable.
• Exclude<T, U>: Remove all types in the union U from the union T
• FunctionParameters<T: fun(...)>: Get the parameter names and types of a function.
• TypeParameters<T>: Returns a tuple of all type parameters passed T

Notes and examples

• TypeParameters<SomeGenericType<string, integer[], boolean>> ⇾ a tuple [string, integer[], boolean]

• Keys<T> and Values<T> ⇾ a union type if the keys / values can have multiple types. In other words:

  • Keys<{ foo: "abc", bar: "xyz" }> should give the type "foo"|"bar"
  • Values<{ foo: "abc", bar: "xyz" }> should give the type "abc"|"xyz"

• The Partial-type should work like this:

  Input:

  Partial<{
      user_name: string,
      email_address: string,
      verified: boolean,
      extra_data?: table<string, Any>
  }>

  Output:

  {
      user_name?: string,
      email_address?: string,
      verified?: boolean,
      extra_data?: table<string, Any>
  }

• FunctionParameters<fun(name: string, age: integer)> should give the type { name: string, age: integer }

  • If someone only needs the parameter names, they may use Keys<FunctionParameters<T>>

• never is useful for communicating to users of a marked function that its return values may change in the future. This is primarily used to communicate to other maintainers / your future self that they / you can change this if they / you want.

• Exclude<T, U> should only remove types from T that it can actually find in U. For example:

  • Exclude<string|number|boolean|function, number|boolean> should give the type string|function
  • Exclude<string|number|boolean, function|userdata> should give the type string|number|boolean, as neither function nor userdata are in T

Implementation

These types would most likely require implementing some features from TypeScript:

• Conditional types (The TS syntax is predicate ? T : U, but if predicate then T else U would probably fit in better with Lua, and potentially allow for elseif down the line if deemed to be necessary or just a good addition; plus, LuaCATS already uses : as the inheritance-operator and ? as the optional/nilable-operator). Examples:

  • ---@alias HasFourLegs<Animal> if Animal : { leg_count: 4 } then Animal else never
    ---@alias FourLeggedAnimals HasFourLegs<Bird|Dog|Ant|Wolf> --> Dog|Wolf

• Type indexing. Examples:
  • The first type of the tuple [string, number, boolean] could be accessed with [string, number, boolean][1]
  • The third parameter name biz of the function fun(foo, bar, biz, baz) could be retrieved with Keys<FunctionParameters<fun(foo, bar, biz, baz)>>[3]
  • The type of the biz field of the table {foo: "bar", biz: "baz"} could be accessed with {foo: "bar", biz: "baz"}["biz"]

Alternatively, instead of fully implementing these features, it might be enough to "simply" hard-code types like Keys and Values. It might also be a good idea to investigate allowing to write types in Lua (perhaps via plug-ins) instead.

If these are not implemented via hard-coding, the following keywords from TypeScript would most likely have to be implemented as well

• in (for collection-like types)
• keyof

Once that's all done, some of them could be the implemented like this:

--- These were partially taken and adapted from the TypeScript docs, see:
--- https://www.typescriptlang.org/docs/handbook/utility-types.html

---@alias Keys<T> keyof T
---@alias Values<T> T[keyof T]

--- ...

---@alias NonNilable<T> if T : nil then never else T
---@alias Partial<T> { [P in Keys<T>]?: T[P]|undefined }
---@alias Required<T> { [P in keyof T]: NonNilable<T[P]> }
---@alias Exclude<T, U> if T : U then never else T
--- Not sure how this would be implemented...
---@alias FunctionParameters<T> ???
--- Same goes for this...
---@alias TypeParameters<T> ???

Considerations

I am decently familiar with LPEG grammars, so I could probably help make all of this happen if given some pointers in the right directions.

[tomlau10]

I'm not familiar with typescript, but as an experienced Lua developer (~8 years) and an intermediate user of LuaLS (~3 months), I found the proposed syntax very difficult to understand 😕 . This is against simplicity (which is the design principle of Lua) and makes things very complicated / hard to learn, especially the conditional types. It's just unintuitive compared with current annotation syntax.

---

which would simplify writing more complex type annotations

I have read an article "Complexity Has to Live Somewhere"

• https://zhuanlan.zhihu.com/p/138145081
• https://ferd.ca/complexity-has-to-live-somewhere.html

In this case if it is very easy to write a complex type annotations, then I think the complexity is transferred to

• how LuaLS is implemented and
• how users need a higher cognitive load to understand the annotation syntax

For the 1st point, it affects the maintainability of LuaLS project. Because many new features / concepts are introduced, and there maybe many edge cases between these concepts when they are used together, thus it may introduce many new bugs / undefined behaviors.

For the 2nd point, a higher learning curve might greatly discourage new comers of LuaLS. Also users might be difficult to distinguish between they used the annotation wrongly or LuaLS really have bugs when they have unexpected result 😂 Because the syntax is so complicated that users don't want to fiddle around, and instead just file an issue every time they encountered something unexpected (and then leave... I have replied to some discussions / issues and never get replies from them later ☹️ )

---

Of course I am no author of LuaLS, I only know that the author is planning a complete revamp: #2366
Maybe he will try to adopt some of your ideas 😄

[SkyyySi]

@tomlau10

I do agree that this would add complexity and pose additional maintainance burdans. I do not, however, agree with this discuraging users from using / learning LuaLS.

As you said, complexity has to live somewhere. The reason I'd like to see the type system grow more advanced is because, currently, the type system is not powerfull enough to handle a dynamic language like Lua very well. As of right now, there's basically nothing you can do in a lot of cases except resorting to the any-type. But then, why use type hints to begin with? The necessary information for percise type hints is available, it's just not exposed to actually tap into.

Which brings me to my other point: Writing type hints with the current system can be very frustrating. If you want to do basically anything beyond what's hard-coded into the language server, you will hit a brick wall, and you will hit it fast. I'd like to provide detailed types because, while they may be more complex on the library author side, they make for an awesome user experience on the consuming side. I don't see how changing the answer of "How can I do [advanced thing]?" from "You can't, there's no way to do it." to "You can, but you'll have to learn some advanced typing features." would be discurraging

These are the same issues that TypeScript had to deal with: You can have either a very simple type system or one that's powerfull enough to describe the language without needing to turn it off half the time (with any). They chose the latter.

Besides that: This isn't extending Lua's syntax. This is specifically extending the LuaCATS comment syntax. Lua may be simple, but the task of trying to create helpful static type hints for it is not. Type hinting dynamic languages is actually a really complex task. Only going half way isn't solving any of that complexity - it's just burrying your head in the sand.

[CppCXY]

I am also interested in implementing a type system similar to TypeScript. I will try to implement this in my project(not luals), but it will still take a considerable amount of time.

[tomlau10]

Just come across an issue which proposed some similar syntax: #2278
Maybe should mark that as duplicate and track it here, as this issue is far more detailed 😄

[mikuhl-dev]

+1 for adding more TypeScript like features.

[Rathoz]

Keys<> and Values<> can be effectively done with the current LuaCATS, as their general types rather than literals

---@generic K, V
---@param tbl {[K]: V}
---@return K[]
local function extractKeys(tbl) end

local a = extractKeys({a = 1, b = 2, c = 3}) -- local a: string[]

I wholeheartly support more advanced annotations!

[SkyyySi]

@Rathoz This is not what I meant. This extractKeys-function returns string[]. But the type should actually be "a"|"b"|"c".

[tomlau10]

If I understand it correctly, it's more like the (key) attribute in @enum? 🤔
Currently we can do this in luals

---@enum (key) EnumKey
local Enum = {
    a = true,
    b = true,
    c = true,
}
--> EnumKey: "a"|"b"|"c"

---@param e EnumKey
local function test(e)
end

test( --< auto complete here will suggest "a", "b", "c"

And in this proposal, it is a helper type, which can be directly used on a type to give another type ?

[mikuhl-dev]

@Rathoz, these types are meant to produce no runtime effects, they are purely to assist with your IDE's autocomplete/intellisense/linting.

@SkyyySi The FunctionParameters would require another useful features, extends and infer:

---@alias FunctionParameters<T> T extends (fun(...: infer P): any) ? P : never

[SkyyySi]

@SkyyySi The FunctionParameters would require another useful features, extends and infer:

---@alias FunctionParameters<T> T extends (fun(...: infer P): any) ? P : never

@mikuhl-dev

As for the extends-keyword: I actually included that in my issue already:

---@alias NonNilable<T> if T : nil then never else T

The "type colon type"-syntax here (T : nil) is supposed to represent the inheritance operator, as used by the @class and @generic annotations already:

---@class Foo

---@class Bar : Foo

---@generic T: Foo

I think it would be a good idea to try to keep things consistent. (For similar reasons, I also think that if T extends nil then never else T would be better than T extends nil : never ? T - LuaCATS already uses both ? and :, each in cometely unrelated ways, and I don't think we should give both a double meaning.)

About the extends-operator... isn't that basically what the

---@generic T
---@param type_name `T`
---@return T
function create_type(name, ...)
    local result = {
        __name = name,
    }

    -- ...

    return result
end

-syntax already does? My TypeScript knowledge is rather limited, so I don't know.

Assuming that this is indeed the LuaCATS equivalent of TypeScript's extends, I guess it could be implemented like this:

---@alias FunctionParameters<T> if T : (fun(...: `P`): any) then P else never

That being said, this would return a union type of all parameter types (e.g. FunctionParameters<fun(foo: string, bar: number)> == string|number). What I had in mind was something like this: FunctionParameters<fun(foo: string, bar: number)> == { foo: string, bar: number } (or maybe a tuple [string, number]).

[mikuhl-dev]

@SkyyySi The `T` is something that LuaLS has that TypeScript actually doesn't, it represents the name of the type of a generic as a string. The closest is to use keyof on an interface:

interface MyTypes {
  Foo: Foo;
  Bar: Bar;
  Baz: Baz;
}

type MyType<T extends keyof MyTypes> = MyTypes[T];

type MyBar = MyType<"Bar">;
//   ^? type MyBar = Bar

The infer keyword lets you "extract" or "unwrap" types from a type and use them as a generic in the truthy section of an extends conditional:

type Foo = (string | number)[];

type Inferred = Foo extends (infer T)[] ? T : never;
//   ^? type Inferred = string | number

You can even do this on strings:

type Foo = "one,two,three";

type Inferred = Foo extends `${infer First},${infer Rest}` ? First : never;
//   ^? type Inferred = "one"

So to extract the parameters, or the return types:

type Foo = (a: string, b: number) => object;

type MyParameters = Foo extends (...args: infer P) => any ? P : never;
//   ^? type MyParameters = [a: string, b: number]
// Note that this tuple has labels, and they are not really accessible easily.
// It is basically an array, and the labels are simply for display.
// See: https://stackoverflow.com/a/69713319

type MyReturns = Foo extends (...args: any) => infer R ? R : never;
//   ^? type MyReturns = object

So, I don't think LuaLS has anything like this, so a new syntax or keyword would have to be used.

That being said, this would return a union type of all parameter types (e.g. FunctionParameters<fun(foo: string, bar: number)> == string|number). What I had in mind was something like this: FunctionParameters<fun(foo: string, bar: number)> == { foo: string, bar: number } (or maybe a tuple [string, number]).

It indeed should return a tuple, because that's what ... is, and you can index the nth element like FunctionParameters<fun(foo: string, bar: number)>[2] == number

[SkyyySi]

@mikuhl-dev Technically, ... isn't a tuple in the LuaLS sense. A tuple type like this

---@type [string, number, boolean]

... is actually a shorthand for writing this:

---@type { [1]: string, [2]:  number, [3]: boolean }

Perhaps, there should be a Vararg<T> type, which takes a tuple table like the above, and tells the language server that it should treat it as an ellipsis.

---

Type indexing would definitely be great addition. It's currently very easy to wrap a type, but unwrapping them often decays into ugly abominations. If you can even get it to work in the first place, that is.

[vnuxa]

conditional types would be very useful, the if type_a : type_b then type_c else type_d should also be extended with an elseif as well and then it would be absolutely perfect imo
if its hard to implement dynamically, how about just make it so that it only checks hardcoded values