TypeScriptToLua modifies strings which contain lua code

[heinzbeinz]

Hi,

I am trying out this project and it is really awesome so far. I am trying to convert the testcases which are written in lua to typescript. These test often contain small test scripts which needs to be passed unmodified to the underlying test system.

It seems, that tstl modifies these snippets even when the code is a simple string constant.

Example:

declare function executeSnippet(this: void, snippet: string): void

let snippet = "local x = require('x')"
executeSnippet(snippet)

leads to the following invalid lua code (with modified require)

local snippet = "local x = require("bugreport.x")"
executeSnippet(snippet)

Can I somehow disable the processing of these strings?

Thanks in advance.

[heinzbeinz]

Strangely, this problem does not happen if I use different quotation marks for the string snippet like this:

declare function executeSnippet(this: void, snippet: string): void

let snippet = `local x = require("x")`
executeSnippet(snippet)

It seems that require('x') is treated differently than require("x")

[Perryvw]

Aha, I'm afraid you ran into an edge case of our module resolution, which simply does a regex on the output lua to rewrite require paths. It is not smart enough to recognize the require is in a string literal. What you observed about the quotes is probably due to the fact we only look for requires to rewrite with ' and not " (although we may have changed that in a newer version).

To do this totally right we would need a much more sophisticated way of finding and replacing these requires. Also to fix requires in comments being resolved like in #1330

[Feuermurmel]

To do this totally right we would need a much more sophisticated way of finding and replacing these requires.

I don't know the details of Lua's syntax, but maybe it's enough to parse the Lua code into a stream of tokens to find each string, without building a full AST? I'm thinking of something similar to what the C preprocessor has to do, which also needs to ignore preprocessor instructions (e.g. #ifdef) that appear inside of strings, but it does so with only minimal understanding of C's syntax. E.g.:

let generatedLua = `
local ____lualib = require("lualib_bundle")
local __TS__SourceMapTraceBack = ____lualib.__TS__SourceMapTraceBack
__TS__SourceMapTraceBack(debug.getinfo(1).short_src, {["10"] = 1,["11"] = 3});
print("local x = require('x')")
print("local x = require(\\"x\\")");
`;

let pattern = /(?<require>require\()?"(?<content>([^\\"]|\\.)*)"/g;

for (i of generatedLua.matchAll(pattern)) {
    if (i.groups.require) {
        console.log(`Rewriting: ${i.groups.content}`);
    } else {
        console.log(`Not touching: ${i[0]}`);
    }
}

Rewriting: lualib_bundle
Not touching: "10"
Not touching: "11"
Not touching: "local x = require('x')"
Not touching: "local x = require(\"x\")"

(of course the pattern needs to be extended to also recognize long string literals and comments, at the least)

[Perryvw]

I had a similar idea, yesterday I already wrote a parser that basically only knows requires, comments and string literals and should be able to skip all of the requires we want to skip, but I need some more time to add tests and make sure everything is nice. Current version is here for now: https://github.com/TypeScriptToLua/TypeScriptToLua/blob/better-require-search/src/transpilation/find-lua-requires.ts

[heinzbeinz]

Thank you for the quick help! Is there a tip jar or something for you guys? Edit: found 👍

[Zamiell]

https://github.com/sponsors/Perryvw