Don't allow selecting iex> in doctests

[michalmuskala]

Today copying from doctests is a bit annoying, especially if there are multiple lines involved, because you'll also select the iex> or ...> parts.

It would be great, if we could make it so that those parts are skipped over when you try to select and copy the code from examples. I did a little research and it seems it's possible to do this in css with user-select: none, but I'm not really sure how to integrate it into the ex_doc pipeline.

[josevalim]

Oh, I like this. I think a separate pass that finds code tags and fixes the iex> inside them would work?

[eksperimental]

I wanted to do this for the elixir website but i never found a way around. If you find it please lets port it there as well

[nscyclone]

I've proposed a PR to highlight.js that might help with this. But unfortunately the project is not very active lately (last change to master was made in Jun 2017 and there are a lot of PRs requiring attention).
Maybe it makes sense to temporarily fork it into elixir-lang, then patch it on our own and update dependencies?

[josevalim]

@nscyclone yes, let's do that. I have forked it here and given you commit access: https://github.com/elixir-lang/highlight.js

Btw, if highlight.js is not being maintained, should we look for alternatives?

[OvermindDL1]

Btw, if highlight.js is not being maintained, should we look for alternatives?

Elixir Library MarkUp? It's fantastic for Elixir and easy to add more languages.

[josevalim]

Oh right! @tmbb, how are things going on the markup front? We have talked about it in the past but I don't remember where we left things off. I believe we have added bindings to ExDoc to support it fully, right?

Also, should we add this feature to markup too?

[tmbb]

This is a long response, and its divided into 4 parts

Part 1: Answering @OvermindDL1 :

Elixir Library MarkUp?

Again OvermindDL1's misspellings attack again. There is no such thing as MarkUp, It's called Makeup for the obvious reasons (it makes your source code all pretty and colorful)

It's fantastic for Elixir

That remains true, it's still the best Elixir highlighter I've seen in the wild.

and easy to add more languages.

Easy if you're willing to write custom lexers for such languages. Which is actually pretty easy usually, but it's a labor-intensive process. Things like string interpolations are not trivial to get right, even though Makeup's parser combinators help a lot. Still beats writing Pygments lexers, though.

BTW, contributors are welcome... I still have to write some docs on how to contribute (and an ExSpirit tutorial, becaus efiguring it out through trial and error like I did is suboptimal xD)

Part 2: Answering @josevalim:

I believe we have added bindings to ExDoc to support it fully, right?

Yes indeed. I've written a "markdown implementation" (unimaginatively named ExDocMakeup) which is just earmark customized to use Makeup for syntax highlighting of the supported languages.

Here is a link to the elixirforum topic. There is a link to a much more detailed description of the package here. I have used ExDocMakeup to prepare the documentation for some larger Elixir packages, such as phoenix and ecto. In fact, I've submitted a PR for phoenix so that it uses ExDocMakeup, but it hasn't been accepted yet due to lack of test. Which brings us to the next point:

Oh right! @tmbb, how are things going on the markup front? We have talked about it in the past but I don't remember where we left things off

Makeup is very functional. It highlights Elixir better than anything else I've seen around, including VSCode, for example (and much better than highlight.js, of course). AFAIK, it's the only highlighter that correctly highlights:

def x + y, do: x - y

(note that the above highlights x as if it were a function name, which is incorrect)

Currently makeup only supports two languages: Elixir and HTML. Because Makeup uses a real parser and not weird regex stuff, lexers are trivial to compose and I can produce a decent .html.eex lexer quickly if I want. Making it perfect might take some work, and that's the only reason why I haven't published such lexer yet (It's kinda hard to explain what makes it so hard).

The big red flag here is that neither Makeup currently has almost no tests. This is mainly because writing unit tests for a syntax highlight is very, very boring, but I'm hoping the new assert_value package might help with that. The lack of tests is what's preventing it from being used to document Phoenix.

I'm currently writing a Python lexer (just for fun, it's pretty easy except for some string interpolation weirdness), but I can drop it to write some tests so that Makeup gets some decent coverage.

If Makeup is meant to be included as part of ExDoc, I propose the following plan:

1. write tests for all makeup common combinators (those that are included in Makeup itself and are used by multiple lexers)

2. write tests for all rules in both the Elixir lexer and the HTML lexer

3. write an EEx lexer and include it my default with ExDoc. This makes sense because of Elixir's strong web presence. Documenting projects such as Phoenix or Drab benefits a lot from having a good EEx highlighter.

4. define the scope of Makeup: should it include language detection (Pygments does this)? How should one pick a lexer from a string? Should this be handled by Makeup itself or should this be handled by the end-user? I'd love @josevalim's help on this one... I lean towards keeping a lean core that only lexes stuff and formats the results.

5. Based on the above, stabilize the API and publish Makeup version 1.0.0 (which can be included in ExDoc version X.Y.Z)

6. Currently compiling makeup produces some "scary" warnings. Those warnings come from compiling ExSpirit macros. Those macros do something that's tagged as unsafe by the compiler but are actually safe. The only way to avoid those warnings would be to dynamically generate atom names at compilation time, which @OvermindDL1 is reluctant to do. This should be discussed between @OvermindDL1 and @josevalim and addressed, as I don't think it's reasonable that ExDoc's compilation produces such weird warnings. In any case, ExSpirit is very small (although pretty gnarly for performance reasons) and maintaining a separate fork is not unreasonable.

7. Based on the point above, stabilize ExSpirit's API (at least the ExSpirit.Parser module) and release version 1.0.0.

Part 3: handling IEx prompts

MakeupElixir already handles IEx prompts (iex>, iex(123)> and ...>). They are recognized by the normal Elixir lexer as a special token and highlighted in a specific way. With a trivial change in the lexer and a slightly less trivial change in the formatter it is possible to add the necessary CSS attributes.

A negative side effect of having the default Elixir lexer highlight these prompts is that they might cause us to wrongly highlight the string iex> ... as a pompt instead of as iex is greater than .... This case is very rare, and simplifies the use of the library with ExDoc. I can elaborate on this topic a little further if you want.

Part 4: Changes on ExDoc's side

There is no need to change anything on ExDoc's side except depending on ExDocMakeup (which will recursively depend on Makeup, MakeupElixir and MakeupHTML, and, of course, ExSpirit). This will increase ExDoc's compilation time, as ExSpirit parsers take a while to compile (it seems that Elixir is not very quick at expanding macros and compiling them into BEAM files). For the 3 parsers I want to include, this is doable, as the compilation time is a one-time cost.

ExDocMakeup has sensible defaults and requires no user configuration for default results, so it's possible that most users wouldn't even notice we were using it.

[josevalim]

The big red flag here is that neither Makeup currently has almost no tests.

To be honest, I don't think the markups themselves need to have tests. It is very likely that none of the options we would consider from JS would have them. However, it is important that Makeup itself has tests and I would start by focusing there.

For the each language implementation, I would recommend having a couple of tests as examples, so you can at least require future changes to the grammar to contain tests.

Those macros do something that's tagged as unsafe by the compiler but are actually safe

Those warnings should be fixed because they will emit compilation errors on v1.7. See explanation here.

Regarding the compilation speed, I can take a look at them and see if we could make anything faster.

define the scope of Makeup: should it include language detection (Pygments does this)?

IMO it is fine not supporting language detection. However, we need to make sure Earmark+Makeup supports fenced code blocks like GitHub (i.e. triple backticks followed by the language).

MakeupElixir already handles IEx prompts (iex>, iex(123)> and ...>).

Beautiful. I agree with your decision. The chance of false positives with iex> is not worth worrying about. And we have the same issue with highlight.js anyway. However, it would be great if we get the user-select: none clauses in those prompts. Could you please investigate it?

I think something else we need to do is to remove highlight.js from the default build and include it only only when using the earmark markdown engine. @nscyclone, could we use your help on this one? For now, compiling highlight.js to a separate javascript file and including it in the layouts shall suffice. We can move it to the markdown engines later.

If everybody is on board, I will open up a new issue with those TODOs so we can tackle them.

[tmbb]

However, it would be great if we get the user-select: none clauses in those prompts. Could you please investigate it?

This is easy. I can tag these tokens with a {:selectable, false} attribute and change the HTML formatter to emit the appropriate CSS. My attribute system is highly extensible to anecipate these needs.

[tmbb]

However, we need to make sure Earmark+Makeup supports fenced code blocks like GitHub (i.e. triple backticks followed by the language).

It does. IIRC there was a slight hack that I needed to make this work, I'll check when I get home.

[tmbb]

To be honest, I don't think the markups themselves need to have tests.

The most important test for a highlighter is to make sure that highlight(text) never chokes on any input (even if the highlighted code is illegal). This is easy to guarantee, but it's the obly general property I can inagine. The rest must be example-based. As I said above, the assert_value package might help a lot.

[tmbb]

One final thing: My lexers highlight matching parentheses, brackets, angle brackets, etc. on mouseover. The implementation details don't matter, but this feature required making the output nondeterministic...

I need html attribute vamues that are globally unique for the HTML document. The easiest way is to use random numbers, which I am doing now.

There are ways of making this deterministic by using the process dictionary If I assume that the entire HTML document will be enerated by the same process. Needless to say, this is not a safe assumption in general, so I'm stuck with using globally unique random values. I'm ok with this approach, but I felt he need to disclose this source of non-determinism.

[tmbb]

I think something else we need to do is to remove highlight.js from the default build and include it only only when using the earmark markdown engine.

I don't think this is a good idea. As ugly as highlight.js is, it does highlight pretty much everything under the sun and is a useful fallback for languages not supported by Makeup.