Add support for Erlang code cells

[filmor]

Adds Erlang code cells and translates bindings (primitively).

I haven't been able to get webpack to actually include the Erlang syntax highlighting for Monaco (even though it's configured). I also haven't looked into integrating Intellisense, though I'd guess that it wouldn't be an enormous amount of work.

Touches on #190.

[CLAassistant]

All committers have signed the CLA.

[josevalim]

I actually like this approach a lot. It is a lightweight introduction of Erlang features and keeps compatibility with smart cells and everything else. The camelize/underscore is a bit unfortunate but also probably the best we can do here. We probably should also change the pretty printing to be Erlang based.

We also need a visual indicator somewhere in the cell which reveals the language being used. Potentially in the top right corner of the cell but that space does not exist for one-liners.

I think this could be enough for an initial version (unless @jonatanklosko has other thoughts) and then we can work on intellisense next.

[filmor]

I actually like this approach a lot. It is a lightweight introduction of Erlang features and keeps compatibility with smart cells and everything else. The camelize/underscore is a bit unfortunate but also probably the best we can do here. We probably should also change the pretty printing to be Erlang based.

Yeah, I willfully ignored the documentation here because I was too lazy to write this code myself ;). There is no unsafety here (I think) because Erlang is not able to update entries (could be added with a local function, like u('message', <<"new erlang message">>).) and Elixir will not produce a variable name that can't be converted (like bla/blubb).

We also need a visual indicator somewhere in the cell which reveals the language being used. Potentially in the top right corner of the cell but that space does not exist for one-liners.
Definitely, I'll see whether I can add something.

Two questions:

1. Could you approve the CI workflows?
2. Any idea how I can convince webpack to include the Erlang syntax highlighting? Strike that, upstream monaco just doesn't include Erlang, I had the same issue with Jupyter. I'll update the dependencies accordingly.

[josevalim]

Strike that, upstream monaco just doesn't include Erlang, I had the same issue with Jupyter. I'll update the dependencies accordingly.

FWIW, they didn't include Elixir either but they did accept a PR.

[github-actions]

Uffizzi Preview deployment-23876 was deleted.

[filmor]

Strike that, upstream monaco just doesn't include Erlang, I had the same issue with Jupyter. I'll update the dependencies accordingly.

FWIW, they didn't include Elixir either but they did accept a PR.

There doesn't seem to be a Monarch grammar, yet. I'll see if I can whip something up (unless you are planning to switch to Treesitter or Textmate grammars eventually :)).

[josevalim]

No plans to swap at the moment. :) So I believe a Monaco grammar will be needed then!

[jonatanklosko]

Hey, thanks for the PR! Here's a list of things that we need for an initial version:

1. Monarch grammar upstream.
2. To track cell dependencies ("identifier dependencies"), we need to determine which variables are used in the Erlang code. Should be doable by just looking at the AST.
   This is so that if you have cells X = 1. and X., then change the first one to X = 2., the second one is marked as stale.
3. We need to adjust .livemd import/export to also handle erlang snippets.

Then we just need a few UI adjustments and I can handle those. We can have an arrow next to the + Code that allows to change the default language (as we do with the Evaluate button). We also need to indicate which cells are erlang, could be an icon on either side of the editor or in the gutter.

With those changes in place it should be good to go and we can expand on intellisense features separately.

[filmor]

1. I'll have a look at that this weekend, but actual upstreaming could require a few iterations. We could include it as a "custom language" as a first step
2. Can you point me to the respective part for the Elixir eval?
3. Would it be enough for now to just have "```erlang" as the leader for Erlang cells?

[jonatanklosko]

1. Yeah sure, that's how we added Elixir initially too.
2. We compute relevant identifiers here:

   livebook/lib/livebook/runtime/evaluator.ex

   Lines 439 to 440 in e7f8ee8

   	{identifiers_used, identifiers_defined} =
   	identifier_dependencies(new_context, tracer_info, context)

   
   For Elixir we use compilation tracer to track imports/aliases/modules etc, I think for Erlang we just need to figure out the variables from the AST.
3. Yeah, it's likely a matter of changing all "elixir" to language in ["elixir", "erlang"] and vice versa :)

[michalmuskala]

Thanks for working on this, this is quite exciting. It would be awesome to see Erlang support in livebook

[filmor]

Regarding tracing: This is a bit more complicated to do than the rest as Erlang's evaluation doesn't have a notion of tracing, so I'd either have to fake trace events after inspecting the AST, or adjust this whole code path a bit.

[josevalim]

You don’t need to do tracing, only reproduce its results. Since Erlang does not have aliases and imports, we only need to know the variables read by Erlang, which is equivalent to traversing its AST blindly and collecting all names from var nodes.

Note to self: we also need to make sure the first setup cell is always in Elixir.

[filmor]

I added a very simple token-based tracing now. It's not actually correct, since it can't handle cases like

Cell 1

message = "test"

Cell 2

F = fun(Message) ->
    Message
end,
F(1).

(it will think that the Cell 2 depends on Cell 1, even though it doesn't).

Fixing this would require actually visiting the whole AST which seems like overkill at this stage :)

[filmor]

The only part missing is translation of the lexer/parser/interpreter errors, I'll do that today. I think I handled everything that @jonatanklosko suggested.

[josevalim]

@filmor the most important is the example where: "if cell 1 defines X, cell 2 defines Y, and cell 3 does X + Y, modifying cell 1 only marks cell 3 as stale". False positives because of anonymous function shadowing is fine IMO.

[filmor]

Working on this a bit more, I think I've understood the variable tracking now :)

[filmor]

Maybe not. It would be great if one of you could chime in here. I've got it to the point that variable dependencies are correctly detected and the recompilation is triggered, but apparently it's not picking up the new binding in this situation. The code to update the versioned_vars to produce a similar output to what Elixir is doing is this:

https://github.com/livebook-dev/livebook/pull/1892/files#diff-7c5945fe98b063bc5ee3f91b876caad483ebba1c6fd5e95eae3873139e3ca079R663-R677

Strike that, seems to be working fine. Anything else you'd want to see handled before merging this?

[filmor]

(I plan to contribute more to LiveBook in the future, it would be helpful if I could always get CI feedback).

[jonatanklosko]

(@filmor CI should be enabled for subsequent PRs, the manual approval is just in case for first-time contributions :)

[jonatanklosko]

FTR waiting for the next Elixir rc, so we can merge #1918. I will resolve the conflicts!

[jonatanklosko]

@filmor I updated the PR, the only thing remaining is better error messages, especially syntax/compiler ones. For example:

This makes sense in erl where the expression must be somehow terminated, but here "before: " is confusing. Generally, we can match on those errors, map them to custom elixir exceptions and provide more useful messages. Can you make the improvements? :)

[jonatanklosko]

@filmor I'm gonna merge this PR, feel free to open next one for the errors and then for intellisense features if you want :)

[josevalim]

Thank you so much for @filmor!

To clarify about exceptions, the issue is here:

https://github.com/livebook-dev/livebook/pull/1892/files#diff-7c5945fe98b063bc5ee3f91b876caad483ebba1c6fd5e95eae3873139e3ca079R723-R742

If you return {:error, anything, stack}, if anything is not an Elixir exception, it will be considered an Erlang error like above. We have two options here:

1. Introduce an ErlangSyntaxError Elixir exception and wrap anything in that

2. Or change the whole exception handling to print the error message and stacktrace as Erlang term (you can crib the default implementation from the Erlang shell)

Option 2 is better but Option 1 will be enough in the short term.

The other thing is that, if the parser has an incomplete expression, it returns a particular error tuple to say so. This is used by the shell, for example, to know the expression is not complete and prompt for a new line. In our case, we want to detect this case (I am almost sure it is when the token is an empty list) and say something like: "incomplete expression".

I would say those are "must fix" before a release. :)

---

Then there are a bunch of enhancements we could add:

1. Intellisense (most likely over several PRs and some of it can be shared with Elixir, such as docs)

2. Allow records to be imported into Livebook (this means you will need to track record usage across cells, in the same way Elixir tracks aliases)

3. Allow functions to be defined in the shell, as Erlang/OTP 26 shell allows. Although I would only do that after the record one as that would share most of the infrastructure

But I think at this point we can open up the challenges to the Erlang community as well.

Once again, thank you! ❤️

[filmor]

I'll try to improve the error handling. I don't want to inspect the error string to decide whether the expression is complete, but traversing it to check for completeness is something I have done before (just needs to be updated to take maybe into account): https://github.com/filmor/ierl/blob/eae2966f507e71fa59f5701b2d374f0caeb54374/apps/ierl/src/ierl_backend_erlang.erl#L185C29-L221. I guess it's TokenMissingError then, right?

[josevalim]

The Erlang shell already detects when it is incomplete and IIRC it is when the parser returns an empty list as the error token. We shouldn’t need to pre-parse it. But ideally, we would also print all errors like the Erlang shell does (and not as Elixir exceptions).