lua static type generation

[shevernitskiy]

Lua LSP support type hinting. It helps much during development.
Currently there are no type hinting in lua part of codebase. But we can generate type definitions of static part for exposed api by using xml map of structures.

This PR introduces simple python script, that could do it.
To generate definitions just execute script in repo workspace.
There will be generated .luarc.json (config file for Lua LSP) and /types dir which will contain generated definitions and some lsp meta info.

Current issues:

• script handles inheritance by adding parent members to target class, but vmethods not copied
• some of complicated stl types not mapped (need to figure out how to do it correctly, or just disable warnings)

P.S.: i think it will be reasonable to make definition for global utils methods like curry in place

[myk002]

these lists will get out of date. can they be parsed from the XML or generated headers?

[shevernitskiy]

I parsed these lists traversing df and df.globals by lua script in game. If there is another static option instead, it would be better to use it. Give me a hint:)
More over this is not precise naming, there are some T_name tables, i don't figure out the logic for such a naming.

[shevernitskiy]

For the best result we should somehow reimplement of exposing logic in LuaWrapper.cpp

[myk002]

for the globals, you can get them from the xml:

fgrep '<global-object' library/xml/df.globals.xml | sed -r "s/.* name='([^']+).*/\1/" | sort

that's shell, but I'm sure you can translate that to python.

[myk002]

the ci/ directory is for scripts run in our continuous integration pipeline. this script is not intended for that, but I don't know a better directory to put it in. @lethosor what do you think?

Running this script should be part of the regular build, and should be integrated into our CMakeLists.txt logic. It would need an option guarding it since it introduces a python dependency, though.

[shevernitskiy]

As an alternative or an additional option for pure lua devs, it can be separate repo, that will build typing for every dfhack version on new tag in main repo. It is possible as far as script work only with static data. Advanced dev still can trigger script manualy as needed.

[myk002]

is types/ the standard location for these types of files? Who reads the output files and is there any setup required?

[shevernitskiy]

LSP wiki do not specify exact naming or location. Only internal structure (/library, config.json) is strict. Location can be passed as arg (even otside the workspace).
/definitions can be an alternative to /types.

[shevernitskiy]

As an addition, type hinting do not work in lua file, where _ENV is used. Dunno how to fix it yet.

[shevernitskiy]

should be squashed at merge i guess

[shevernitskiy]

I'd like to summarize the interim results.

• df object typing
  Works almostly correct. Needs to be tested.
  Main caveat: default methods for different kind of entities (may be incorrect, need to clarify logic to add correct pack of methods)

• dfhack signatures
  Founds methods from module exposing arrays in LuaApi.cpp. Other cases doesn't supported.
  Cepable to found ~75% of signatures in auto mode, ~65% are correct.
  Supports <expose> tag, that can explicitly declare signature for specific method in exposing array.
  Supports <force-expose> tag, that could explicitly declare whatever you want signature (with proper naming).
  Union and default values in signatures supported.
  Main work in this side is manual writing expose tags for missing or incorrect signatures.

• lua signatures
  This part is about internal lua glue and type hinting between modules and files.
  df and dfhack types available everywhere.
  Parses modules and global scripts to find global functions and variables.
  Can parse class methods in the files, but not internal variables (self. declaration inside functions).
  Constants will be typed correctly if it's literal type (int/string/bool), but will be any in other cases.
  Can do type injection into modules, but not into global scripts.
  Parser could be improved to handle more complicated cases. But the most of work should be on lua scripts side, imo.
  Main utility functions can be annotated at least.

• docs parsing
  Currently there are ~665 matches on docs parsing. About ~80% are valid.
  Works with signatures in lua and dfhack part.
  Regex pattern can be improved to find more matches.
  Docs can be slightly refactored to be more parsable for regex.

There are possibility to generate docs from typing. Could be useful for df object map i guess.

[myk002]

I'd prefer if we could get these signatures from Lua API.rst. They're just going to get out of date if put in the cpp files as comments.

[shevernitskiy]

which one?
if you are talking about <expose> tag - it is right near the declared method (it is a part of regex).. so it is hard to forgot about it if you are doing something with this method...
else if about <force-expose> - yes, it is not the best solution... maybe it is easier just make static file with manual annotations.
Lua API.rst could be the source of signatures only if will be more or less strictly formated (in terms of signatures) with proper typing... atm docs can't be the source of typings

[shevernitskiy]

tags are needed only if signature can't be found or found incorrect with auto mode

[myk002]

ah, I see. that's tricky. The reason I'm concerned is that you don't look at LuaApi.cpp or LuaTools.cpp if you change a function signature. For example, if I change the function signature of an existing function in the Textures module, you don't need to change the WRAPM lines in LuaApi.cpp. They just magically generate a new Lua binding to match the new function signature. Any <expose> tags next to the WRAPM lines won't be seen.

[shevernitskiy]

In case of WRAPM signatures found well by auto search (because it easy search for Module::Function), so if you change signatures somewhere in Textures.cpp, it should be all ok...
tag are needed, where it can't be found accroding this logic, for example if it is wrapper function in LuaApi.cpp
It's a concept, it's hard to discuss such things in here, need live dialogue:)