Use Rust Parser from Java instead of AST.scala

[JaroslavTulach]

Enabling Rust based parser as the default Enso parser. Disabling few currently failing tests - marking their appropriate reproducers as @Ignored in EnsoCompilerTest. Introducing ENSO_PARSERproperty to allow anyone to switch to old parser in case of problems. Just use:

$ ENSO_PARSER=scala ./bin/enso --run ....
$ ENSO_PARSER=scala ./run ide watch

and you temporarily get the exact behavior you are used to.

Checklist

• All code conforms to the
  Scala,
  Java,
  style guides.
• All code has been tested:
  • Unit tests have been written where possible.
  • If GUI codebase was changed: Enso GUI was tested when built using BOTH
    ./run ide build and ./run ide watch.

[JaroslavTulach]

With 3ef9022 the testOnly *MethodsTest* test suite succeeds fine including one test which is using the new parser: "throw an exception when non-existent"!

[kazcw]

@JaroslavTulach How can I run these tests? I've tried ./run backend test standard-library but it fails with compile errors (cannot find Tree while compiling Parser).

[JaroslavTulach]

@JaroslavTulach How can I run these tests? I've tried ./run backend test standard-library but it fails with compile errors (cannot find Tree while compiling Parser).

There is a primitive build system in LoadParser.sh that invokes necessary commands.

[JaroslavTulach]

Updated to most recent develop branch + fixed compilation problems. Now the ./LoadParser.sh score is:

Found 190 files. 105 failed to parse
0 files failed because they have comments
From 76 files 65 failed to produce IR

[JaroslavTulach]

Twelve more files produce IR now:

Found 190 files. 105 failed to parse
0 files failed because they have comments
From 76 files 53 failed to produce IR

[JaroslavTulach]

I am trying:

enso$ rm -rf *; git checkout -f .
enso$ sbt --java-home ~/bin/graalvm bootstrap
enso$ sbt --java-home ~/bin/graalvm "runtime/testOnly *EnsoCompilerTest"

and that fails complaining there is no parser library .so and yes, there is none:

[error] Test org.enso.compiler.EnsoCompilerTest failed: java.lang.UnsatisfiedLinkError: Can't load library: /target/rust/debug/libenso_parser.so, took 0.0 sec
[error]     at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2630)
[error]     at java.lang.Runtime.load0(Runtime.java:768)
[error]     at java.lang.System.load(System.java:1835)
[error]     at org.enso.syntax2.Parser.<clinit>(Parser.java:24)
[error]     at org.enso.compiler.EnsoCompiler.<init>(EnsoCompiler.java:13)
[error]     at org.enso.compiler.EnsoCompilerTest.initEnsoCompiler(EnsoCompilerTest.java:27)

running sbt generateRustParserLib manually before running the test fixes the problem. However, I'd like this dependency to be built automatically, if possible.

[kazcw]

It's great to see that we're passing tests with the new parser--it is working for correct inputs. However there are a few (small) things necessary to handle syntax errors gracefully, and I think that's important enough to the libraries-developer experience that we should do it before changing the default parser.

The 3 things we need are:

• https://www.pivotaltracker.com/story/show/183405907: 2 days of work Rust side; this is the next thing on my agenda. (Edit: implemented in Ensure parses of invalid inputs represent all tokens #3860)
• https://www.pivotaltracker.com/story/show/183740897: Make errors (IR$Error$Syntax), not exceptions, in TreeToIr.
• When an invalid escape sequence is encountered, getValue() returns -1. Currently TreeToIr throws an exception in that case; I think inserting the Unicode "replacement character" U+FFFD (�) would be a better way to handle it. (In the future, the lexer will also emit a warning when encountering such a case; this is part of https://www.pivotaltracker.com/story/show/182963507 because lexer warnings depend on frontend-related UUID work).

[JaroslavTulach]

https://www.pivotaltracker.com/story/show/183740897: Make errors (IR$Error$Syntax), not exceptions, in TreeToIr.

Created #3861 to address the problem of errors (IR$Error$Syntax).

When an invalid escape sequence is encountered, getValue() returns -1. Currently TreeToIr throws an exception in that case; I think inserting the Unicode "replacement character" U+FFFD (�) would be a better way to handle it.

Shouldn't the getValue() return 0xFFFD instead of -1 then? Anyway addressing this on any side of the parser is easy once we have a testcase. So far I found that:

main = '\x'

behaves differently between the old and new parser. However there is no -1, just 0 returned from getValue(). If you have a testcase in mind, please share.

[hubertp]

lgtm

[hubertp]

a TODO to figure out why self parameter is needed?

[kazcw]

When an invalid escape sequence is encountered, getValue() returns -1. Currently TreeToIr throws an exception in that case; I think inserting the Unicode "replacement character" U+FFFD (�) would be a better way to handle it.

Shouldn't the getValue() return 0xFFFD instead of -1 then?

I think it's important to maintain a distinction between a valid escape with value 0xFFFD, and an invalid escape where we are just using that value as a placeholder in the backend to allow compilation to proceed in spite of an error. It may make a difference in editing or refactoring tools, where we wouldn't want to accidentally "promote" an invalid escape to a valid escape of the placeholder character.

Anyway addressing this on any side of the parser is easy once we have a testcase. So far I found that:

main = '\x'

behaves differently between the old and new parser. However there is no -1, just 0 returned from getValue(). If you have a testcase in mind, please share.

Ah, that's a great test case! The lexer accepts up to 2 hexadecimal characters in the \x type of escape. I will update it to specifically disallow a zero-length hex number, because we don't need \x to be an alias for \0 and \x0.

In the meantime, an example of an invalid escape would be: \c

[JaroslavTulach]

In the meantime, an example of an invalid escape would be: \c

Done in 56f04d6. Let's integrate.