non-standard FormatCodes in Code snippets

[cfa]

L<need|/language/modules#need> MyModule;

in docs/language/modules.pod6 is currently rendered as

<a href="/language/modules.html#need">need</a> MyModule;

on https://new-raku.finanalyst.org/language/modules.html#use.

[coke]

new-raku is a one off put out by @finanalyst.

This looks fine on the soon-to-be-live

https://docs-dev.raku.org/language/modules.html#use

[cfa]

Same rendering issue on my local machine. As discussed on IRC, the problem (and difference from docs-dev) seems to be the highlighter.

[finanalyst]

Yes the highlighter has not been installed in the build environment for doc-website.

It is a one-off installation, but requires a node.js install as well. There is a warning when raku-pod-render is installed by zef if node.js v14 is not present.

This extra stack of software just to have highlighting is why developing a raku highlighter is important.

@dontlaugh there is a separate binary when raku-pod-render is installed that will install the highlighter.

[finanalyst]

@dontlaugh further to the above, the following is installed by raku-pod-render

raku-pod-render-install-highlighter

Run it once in the build environment. You will need to have versions of npm etc, but the installation binary should tell you which version.

[dontlaugh]

Does this depend on an exact version of node? Installation fails with this error

/home/builder/MoarVM/install/share/perl6/site/bin

Using npm version 8.5.5 and node 16.

Trying to create highlighter at /home/builder/.raku-pod-render/highlights

Use of uninitialized value element of type Any in string context.
Methods .^name, .raku, .gist, or .say can be used to stringify it to something meaningful.
  in block  at /home/builder/MoarVM/install/share/perl6/site/sources/307B635565AFF273CDA2EBA8DC255EDDB9F7C46A (InstallAtomHighlighter) line 40
Tried to open directory /home/builder/MoarVM/install/share/perl6/site/resources
  in sub MAIN at /home/builder/MoarVM/install/share/perl6/site/sources/307B635565AFF273CDA2EBA8DC255EDDB9F7C46A (InstallAtomHighlighter) line 40
  in block <unit> at /home/builder/MoarVM/install/share/perl6/site/resources/BB27173280B12F052060F9D682B4D0907F55A443 line 2
  in sub MAIN at /home/builder/MoarVM/install/share/perl6/site/bin/raku-pod-render-install-highlighter line 3
  in block <unit> at /home/builder/MoarVM/install/share/perl6/site/bin/raku-pod-render-install-highlighter line 1

[dontlaugh]

@finanalyst the open PR in your raku-pod-render repo appears to explain my error

[finanalyst]

@dontlaugh I've just merged the change. @cfa+ I've bumped the version of raku-pod-render to 4.4.1 and pushed it to fez. Please try fez upgrade Raku::Pod::Render , then run raku-pod-render-install-highlighter and let me know if this works.

[dontlaugh]

@finanalyst I installed the highlighter tools into the build environments. Check the dev environment? https://docs-dev.raku.org/language/5to6-nutshell.html#constant

I see some highlighted stuff. :)

Update (for those unaware): note that the hourly build overwrites this public container, so anyone can pull/run the following container registry https://quay.io/repository/colemanx/raku-doc-website

[coke]

OPs original bug is now visible on docs-dev

[finanalyst]

Just looked at the original issue.

The problem is actually quite severe!

The issue concerns an example snippet of Raku code.

=begin code :allow<L> :skip-test<needs dummy module>
L<need|/language/modules#need> MyModule;
import MyModule;
=end code

ProcessedPod currently ignores all metadata with code. The string is rendered and the content is given to the template for code-block.

I have only just discovered that according to S26, :allow is specified. However, the behaviour of L<> is not specified.

No extra behavior is documented according to Raku/language/pod.

So the Rakudoc in the snippet above is a non-documented extension from the standard.

Of the 412 Rakudoc filles in repo, only 13 contain code :allow< Only 11 lines have :allow< L

What is the utility of re-writing the template to allow for a non-standard Rakudoc ?

The use case for :allow is to allow for emphasing, and I think was before syntax highlighting was introduced.

I would prefer to remove all the 11 examples of the redundant L<> from all code sections than to create a generic template with other possible errors.

[cfa]

Interesting.

My reading of S26 is that L<> is permitted and should behave as it does outside of a code block:

Although =code blocks automatically disregard all formatting codes, occasionally you may still need to specify some formatting within a code block. For example, you may wish to emphasize a particular keyword in an example (using a B<> code). Or you may want to indicate that part of the example is metasyntactic (using the R<> code). Or you might need to insert a non-ASCII character (using the E<> code).

You can specify a list of formatting codes that should still be recognized within a code block using the :allow option. The value of the :allow option must be a list of the (single-letter) names of one or more formatting codes. Those codes will then remain active inside the code block.

(emphasis mine)

So, I'm not sure L<> needs to be explicitly specified in this case. Rather, the issue seems to be whether :allow, as outlined in the synopsis, is the currently expected behaviour.

Regarding

The use case for :allow is to allow for emphasing, and I think was before syntax highlighting was introduced.

the use case seems broader than that—it's to allow for formatting. Even with syntax highlighting enabled I may still want to link a substring in a code block (as in this need example).

+1 to documenting :allow on https://docs-dev.raku.org/language/pod.html.

[finanalyst]

@cfa I am in favour of not allowing Format codes inside a code block. IMO a code block should be Raku compilable code. A copy/paste of a snippet with Formating will lead to non compilation.

Also, looking more broadly, what happens when more Format codes, such as D<> are implemented with far more content?

[cfa]

IMO a code block should be Raku compilable code. A copy/paste of a snippet with Formating will lead to non compilation.

But code blocks aren't necessarily Raku code. For example,

=for code :lang<ruby>
foo +     # In Ruby a trailing operator means parsing should continue
  bar +
  baz

in rb-nutshell.pod6 should be highlighted as Ruby, not Raku. At the moment this is not the case (see here); perhaps this ought to be filed as a separate issue?

Meanwhile (to your broader point), we have sundry formatting issues like this one where B<...> is rendered as <strong>...</strong>.

[cfa]

P.S., these are all potentially problematic at the moment due to :allow or :lang specifications.

[finanalyst]

In rb-nutshell.pod6 should be highlighted as Ruby, not Raku. At the moment this is not the case (see here); perhaps this ought to be filed as a separate issue?

We need two issues:

• whether or not to continue to support the :allow metatab on =code blocks. Lets leave this discussion in this thread.
• new syntax highlighter. There is another issue for a syntax highlighter. I am intending to do this, but for Raku only. However, I can try to make it more generic. The aim is for an illustrative syntax highlighter that only works for short snippets and may not be completely correct. Eg editor syntax highlighters always have to be correct. see Check and upgrade highlights #111

[2colours]

whether or not to continue to support the :allow metatab on =code blocks. Lets leave this discussion in this thread.

Does that discussion not affect the specification of Raku overall? In which case it would make sense to migrate it to a place where everybody can see it as a language design question.

[finanalyst]

@2colours I was thinking about this too.
@coke can we move the last few posts here to Raku/doc?

[coke]

We can move the entire ticket to raku/doc, but not parts of it. If you want to do a partial move, please create a curated issue/discussion on raku/doc. If you want the whole thing, give this a thumbs up.

[cfa]

Might be more straightforward to create a new issue there, summarising and linking this discussion?

[cfa]

(The followup doc discussion is here.)

[cfa]

Another instance of this issue was just reported: Raku/doc#4250

[finanalyst]

The issue then is to revise the rendering of code-blocks so that :allow is correctly recognised.

There should be an update of Raku/doc/Pod.rakudoc to document the various code block tabs.

[finanalyst]

@cfa This is now resolved by a patch to the Block code template.

The result can be viewed at new raku pod.

Note The pod file at new-raku is a revision of the Pod file in Raku/doc. It has quite a few of the examples re-instated - they did not work in Pod::To::HTML