Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove comments from Lua code when bundle project #8187

Merged
merged 6 commits into from Oct 28, 2023
Merged

Remove comments from Lua code when bundle project #8187

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e64ebbb
remove empty life cycle functions
AGulev Oct 27, 2023
9e8396e
remove comments
AGulev Oct 27, 2023
0a5ae63
Merge branch 'dev' into remove-lua-comments
AGulev Oct 27, 2023
c34d8f5
revert local changes
AGulev Oct 27, 2023
133a3bb
minor fixes
AGulev Oct 27, 2023
62a32e2
fix comments
AGulev Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
130 changes: 108 additions & 22 deletions com.dynamo.cr/com.dynamo.cr.bob.test/src/com/dynamo/bob/pipeline/LuaScannerTest.java
View file
Open in desktop
Expand Up @@ -103,6 +103,10 @@ public void testScanner() throws Exception {
assertValidRequire("require 'foo.bar' --[[ some comment]]--", "foo.bar");
assertValidRequire("require (\"foo.bar\") --[[ some comment]]--", "foo.bar");
assertValidRequire("require ('foo.bar') --[[ some comment]]--", "foo.bar");

int linesInFile = file.split("\r\n|\r|\n").length;
int linesAfterScanner = scanner.getParsedLua().split("\r\n|\r|\n").length;
assertEquals(linesInFile, linesAfterScanner);
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure that after processing, we keep the same count of code lines


private Property findProperty(List<Property> properties, String name) {
Expand Down Expand Up @@ -169,6 +173,10 @@ public void testPropsStripped() throws Exception {
assertEquals(Status.INVALID_ARGS, properties.get(5).status);
assertPropertyStatus(properties, "three_args", Status.INVALID_VALUE, 19);
assertPropertyStatus(properties, "unknown_type", Status.INVALID_VALUE, 20);

int linesInSource = source.split("\r\n|\r|\n").length;
int linesAfterScanner = scanner.getParsedLua().split("\r\n|\r|\n").length;
assertEquals(linesInSource, linesAfterScanner);

// parse the already stripped source
// there should be no properties left
Expand Down Expand Up @@ -261,13 +269,92 @@ public void testPropsMaterial() throws Exception {
assertProperty(properties, "prop4", "material", 3);
}

public static String trimLines(String string) {
String[] lines = string.split("\\r?\\n");
StringBuilder result = new StringBuilder();
for (String line : lines) {
result.append(line.trim()).append("\n");
}
return result.toString();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using of this function is a bit tricky. It's easier to take into account spaces in expected results

@Test
public void testCommentRemoving() throws Exception {
// few comments
String luaCode =
"local var = 1\n" +
"-- comment 1\n" +
"-- comment 2\n" +
"-- comment 3\n" +
"local var2 = 2\n";
LuaScanner scanner = new LuaScanner();
scanner.parse(luaCode);
String parsed = scanner.getParsedLua();
String expected =
"local var = 1\n" +
"\n" +
"\n" +
"\n" +
"local var2 = 2\n";
assertEquals(expected, parsed);

// Function with comment
luaCode =
"local function run(self)\n" +
"self.temp = 1" +
"-- Some comment\n" +
"end\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"local function run(self)\n" +
"self.temp = 1" +
"\n" +
"end\n";
assertEquals(expected, parsed);

// With comment
luaCode =
"local var = 2 -- Some comment\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"local var = 2 \n";
assertEquals(expected, parsed);

// With multiline comment
luaCode =
"local var = 2 --[[Some multiline comment\n"+
"line two]]--\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"local var = 2 \n"+
"\n";
assertEquals(expected, parsed);

// With multiline comment
luaCode =
"--[[Some comment\n"+
"line two]]--\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"\n"+
"\n";
assertEquals(expected, parsed);

// With multiline comment
luaCode =
"--[[Some comment\n"+
"line two]]local var = 2\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"\n"+
"local var = 2\n";
assertEquals(expected, parsed);
}

@Test
Expand All @@ -284,11 +371,11 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
String parsed = scanner.getParsedLua();
String expected =
"local var = 1\n" +
"\n" +
" \n" +
"\n" +
"\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);

// With comment
luaCode =
Expand All @@ -303,29 +390,28 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
parsed = scanner.getParsedLua();
expected =
"local var = 1\n" +
" \n" +
"\n" +
"-- Some comment\n" +
"\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));

assertEquals(expected, parsed);

// With multiline comment
luaCode =
"function on_message(self)\n" +
"--[[ Some comment\n" +
"Some line2]]--\n" +
"--[[Some comment\n" +
"next line]]--\n" +
"end\n";

scanner = new LuaScanner();
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
" \n" +
"\n" +
"\n" +
"--[[ Some comment\n" +
"Some line2]]--\n" +
"\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);

// With _G
luaCode =
Expand All @@ -337,10 +423,10 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
scanner.parse(luaCode);
parsed = scanner.getParsedLua();
expected =
"\n" +
" \n" +
"\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);

// local function
luaCode =
Expand All @@ -359,7 +445,7 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
"\n" +
"end\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);

// local function 2
luaCode =
Expand All @@ -378,7 +464,7 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
"\n" +
"end\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);

// not empty
luaCode =
Expand All @@ -397,7 +483,7 @@ public void testRemoveEmptyLifecycleFunctions() throws Exception {
"self.var = 1\n" +
"end\n" +
"local var2 = 2\n";
assertEquals(expected, trimLines(parsed));
assertEquals(expected, parsed);
}

}
51 changes: 24 additions & 27 deletions com.dynamo.cr/com.dynamo.cr.bob/src/com/dynamo/bob/pipeline/LuaScanner.java
View file
Open in desktop
Expand Up @@ -48,6 +48,7 @@
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Token;
import org.antlr.v4.runtime.tree.TerminalNode;
import org.antlr.v4.runtime.TokenStreamRewriter;

public class LuaScanner extends LuaParserBaseListener {

Expand Down Expand Up @@ -96,8 +97,8 @@ public class LuaScanner extends LuaParserBaseListener {
put(LuaParser.CHARSTRING, "'");
}};

private StringBuffer parsedBuffer = null;
private CommonTokenStream tokenStream = null;
private TokenStreamRewriter rewriter;

private List<String> modules = new ArrayList<String>();
private List<Property> properties = new ArrayList<Property>();
Expand Down Expand Up @@ -180,28 +181,44 @@ public String parse(String str) {
modules.clear();
properties.clear();

parsedBuffer = new StringBuffer(str);

// set up the lexer and parser
// walk the generated parse tree from the
// first Lua chunk

LuaLexer lexer = new LuaLexer(CharStreams.fromString(str));
tokenStream = new CommonTokenStream(lexer);
LuaParser parser = new LuaParser(tokenStream);
rewriter = new TokenStreamRewriter(tokenStream);

// remove comments before parsing
tokenStream.fill();
for (Token token : tokenStream.getTokens()) {
if (token.getChannel() == LuaLexer.COMMENTS) {
int type = token.getType();
if (type == LuaLexer.LINE_COMMENT) {
rewriter.replace(token, System.lineSeparator());
}
else if (type == LuaLexer.COMMENT) {
// Multiline comment
rewriter.replace(token, System.lineSeparator().repeat(token.getText().split("\r\n|\r|\n").length - 1));
}
}
}
LuaParser parser = new LuaParser(rewriter.getTokenStream());

ParseTreeWalker walker = new ParseTreeWalker();
walker.walk(this, parser.chunk());
String resultText = rewriter.getText();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to include the first rewriter.getText() call in TimeProfiler.

From documentation about rewriter:
https://www.antlr.org/api/JavaTool/org/antlr/v4/runtime/TokenStreamRewriter.html

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String with TokenStream.getText(). This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the getText() method(s) scan the input token stream and check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

TimeProfiler.stop();
// return the parsed string
return parsedBuffer.toString();
return resultText;
}

/**
* Get the parsed Lua code
* @return The parsed Lua code
*/
public String getParsedLua() {
return parsedBuffer.toString();
return rewriter.getText();
}

/**
Expand Down Expand Up @@ -238,11 +255,7 @@ private List<Token> getTokens(ParserRuleContext ctx) {

// replace the token with an empty string
private void removeToken(Token token) {
int from = token.getStartIndex();
int to = from + token.getText().length() - 1;
for(int i = from; i <= to; i++) {
parsedBuffer.replace(i, i + 1, " ");
}
rewriter.delete(token);
}

private void removeTokens(List<Token> tokens) {
Expand Down Expand Up @@ -304,22 +317,6 @@ else if (count != resultArgs.length) {
return true;
}

/**
* Callback from ANTLR when a Lua chunk is encountered. We start from the
* main chunk when we call parse() above. This means that the ChunkContext
* will span the entire file and encompass all ANTLR tokens.
* We use this callback to remove all comments.
*/
@Override
public void enterChunk(LuaParser.ChunkContext ctx) {
List<Token> tokens = getTokens(ctx);
for(Token token : tokens) {
if (token.getChannel() == LuaLexer.COMMENT) {
removeToken(token);
}
}
}

/**
* Callback from ANTLR when a function call is entered. We use this to grab all
* require() calls and all go.property() calls.
Expand Down