POD not as documented

[finanalyst]

The Problem

In Raku/doc/doc/Language/pod.rakupod and in S26 the blocks =input and =output are defined as named blocks, but are parsed by Rakudo into code-block Structures.

Expected Behavior

A =input should lead to Pod::Block::Named{:name("input")}

If not ::Named then perhaps ::Input, but not ::Code because there is no way to distinguish =input, =output from =code.

Actual Behavior

Actually we get Pod::Block::Code

Steps to Reproduce

test file ttt.raku contains

use v6.d;
=begin pod

=begin input
12345678901234567890
=end input

=begin some-name
12345678901234567890
=end some-name

=end pod
say $=pod;

run with raku ttt.raku yields:

[Pod::Block::Named{:name("pod")}
  Pod::Block::Code
    12345678901234567890


  Pod::Block::Named{:name("some-name")}
    Pod::Block::Para
      12345678901234567890
]

Environment

• Operating system: Ubuntu
• Compiler version (perl6 -v or raku -v):
• #Welcome to Rakudo™ v2023.02.
  Implementing the Raku® Programming Language v6.d.
  Built on MoarVM version 2023.02.

[lizmat]

It is my understanding that these blocks should keep the whitespace as is from the pod. That is correct, right?

FWIW, I think there are spectest expecting =input / =output to be Code blocks. Will check. In any case, with the current RakuAST implementation I'd like to stay as close to the legacy implementation, including any bugs. To be replaced by another, more sensible implementation for 6.e. Or alternately, run the pod through the .AST method, and work off of the simplified RakuAST::Doc::Block objects.

[finanalyst]

@lizmat They should keep the whitespace as in file. Yes. But they should not be CodeBlocks.

S26 says that code, input and output rendering should be different, eg. by typeface. However, by the time of the Renderer, there is no way to distinguish between the three, so effectively, they are the same. This is a big bug. But because no one ever used =input int the documentation sources, its not been picked up.

I'm thinking of suggesting that they be removed from the Pod Documentation.

Is there a link to how to access the RakuAst blocks? I'm looking forward to experimenting with that.

[lizmat]

Currently, the .AST function should parse RakuDoc and produce RakuAST::Doc::xxx objects. You can try this on a rakudoc file with:

$ RAKUDO_RAKUAST=1 raku -e 'say "filename.rakudoc".IO.slurp.AST'

For example, for a file with content:

=begin pod

=head1 NAME

eigenstates - expose the eigenstates of an object

=head1 SYNOPSIS

=begin code :lang<raku>

use eigenstates;

say eigenstates(1|2|3);  # 1 2 3

say eigenstates(42);     # 42

=end code

=head1 DESCRIPTION

The eigenstates distribution exports a single subroutine called C<eigenstates>
that returns a list of the eigenstates of an object.  For all objects except
C<Junction>s, that's a list with the given object.  For a C<Junction>, it
returns the internal values (aka "eigenstates") as a List.

=end pod

you'd get:

RakuAST::StatementList.new(
  RakuAST::Doc::Block.new(
    type       => "pod",
    paragraphs => (
      RakuAST::Doc::Block.new(
        type        => "head",
        level       => "1",
        abbreviated => True,
        paragraphs  => (
          RakuAST::Doc::Paragraph.new(
            "NAME\n"
          ),
        )
      ),
      RakuAST::Doc::Paragraph.new(
        "eigenstates - expose the eigenstates of an object\n"
      ),
      RakuAST::Doc::Block.new(
        type        => "head",
        level       => "1",
        abbreviated => True,
        paragraphs  => (
          RakuAST::Doc::Paragraph.new(
            "SYNOPSIS\n"
          ),
        )
      ),
      RakuAST::Doc::Block.new(
        type       => "code",
        config     => ${:lang("raku")},
        paragraphs => (
          "use eigenstates;\n\nsay eigenstates(1|2|3);  # 1 2 3\n\nsay eigenstates(42);     # 42\n\n",
        )
      ),
      RakuAST::Doc::Block.new(
        type        => "head",
        level       => "1",
        abbreviated => True,
        paragraphs  => (
          RakuAST::Doc::Paragraph.new(
            "DESCRIPTION\n"
          ),
        )
      ),
      RakuAST::Doc::Paragraph.new(
        "The eigenstates distribution exports a single subroutine called ",
        RakuAST::Doc::Markup.new(
          letter => "C",
          atoms  => (
            "eigenstates",
          )
        ),
        "that returns a list of the eigenstates of an object.  For all objects except",
        RakuAST::Doc::Markup.new(
          letter => "C",
          atoms  => (
            "Junction",
          )
        ),
        "s, that's a list with the given object.  For a ",
        RakuAST::Doc::Markup.new(
          letter => "C",
          atoms  => (
            "Junction",
          )
        ),
        ", itreturns the internal values (aka \"eigenstates\") as a List.\n"
      ),
    )
  )
)

Now looking at this output, I immediately see some buglets, but hopefully this will give you an idea.

[finanalyst]

which version of raku?
Just tried this. Got:

 RAKUDO_RAKUAST=1 raku -e 'say "ttt.rakudoc".IO.slurp.AST'
===SORRY!===
Cannot find method 'resolve-lexical-constant-in-setting' on object of type NQPMu
richard@ged:~/development/raku-pod-render$ raku -v
Welcome to Rakudo™ v2023.02.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2023.02.

[lizmat]

You would need the HEAD version, post 2023.04 even.

[lizmat]

FWIW, I think the conversion to text (a la --doc) will be at least 3x as fast as before, when based on the RakuAST tree.

[finanalyst]

@lizmat I tried using 2023.04, doesn't work

in continuation of this theme, but off-topic: what is a convenient way of keeping a Raku version available at the bleeding edge? I now rely on the rakudo-pkg deb.

[finanalyst]

@lizmat Regarding --doc, have you got rid of the irritating bug in the render method? I have raised an issue about it. Basically, if --doc=SomeNew that accesses Pod::To::SomeNew, the compiler calls the render method of the class in SomeNew. However, it calls the renderer twice, not once. So, there needs to be a hack to avoid the second render. An example can be found in my Pod::To::HTML2 class.

[lizmat]

The only way to keep on HEAD, is to checkout the Rakudo repo, and build from there. 2023.04 doesn't have all the nooks and crannies yet, I implemented those post 2023.04.

[lizmat]

Currently, --doc doesn't seem to do anything with RakuAST. There is at least one spectest depending on it, so I guess I'll get to that sooner or later. FWIW, I'll probably won't look at fixing it with the legacy grammar, unless it is bleedingly obvious :-)

[patrickbkr]

@finanalyst: There is no way around building your own Rakudo. That boils down to:

git clone https://github.com/rakudo/rakudo
cd rakudo
Configure.pl --prefix ../install --gen-moar --make-install # or some other path
echo "export PATH=/home/path/to/install/bin:/home/path/to/install/share/perl6/site/bin:\$PATH" > ~/.bashrc

Once set up as above update with:

git pull
make realclean
Configure.pl --prefix ../install --gen-moar --make-install

If you don't want to manually set the paths like above and be able to quickly switch between different Rakudos, use Rakubrew. To do so build a Rakudo like above (but leave PATH alone) and then:

rakubrew register my-build /home/path/to/install

Then you can rakubrew switch my-build to use that version. To automatically put a zef in that custom built Rakudo you can then do rakubrew build-zef.

[niner]

On Thursday, 4 May 2023 11:01:44 CEST Patrick Böker wrote: @finanalyst: There is no way around building your own Rakudo.

Actually, there is. There are up-to-date Rakudo main builds available on the OBS at: https://build.opensuse.org/project/show/home:niner9:rakudo-git

[finanalyst]

Just to report:

• The AST statement list correctly distinguishes between code/input/output.
• The pod tree gives all of them as Pod::Block::Code

$ raku -v
Welcome to Rakudo™ v2023.04-50-gd1ca18f17.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2023.04.

File for testing

=begin pod
=begin code
say 'hello'
=end code
=begin input
say 'hello'
=end input
=begin output
say 'hello'
=end output
=end pod

AST output

$ RAKUDO_RAKUAST=1 raku -e "'ttt.rakudoc'.IO.slurp.AST.say"
RakuAST::StatementList.new(
  RakuAST::Doc::Block.new(
    type       => "pod",
    paragraphs => (
      RakuAST::Doc::Block.new(
        type       => "code",
        paragraphs => (
          "say 'hello'\n",
        )
      ),
      RakuAST::Doc::Block.new(
        type       => "input",
        paragraphs => (
          "say 'hello'\n",
        )
      ),
      RakuAST::Doc::Block.new(
        type       => "output",
        paragraphs => (
          "say 'hello'\n",
        )
      ),
    )
  )
)

Pod output

$ raku -e "('ttt.rakudoc'.IO.slurp ~ ';say $=pod').EVAL.say"
[Pod::Block::Named{:name("pod")}
  Pod::Block::Code
    say 'hello'


  Pod::Block::Code
    say 'hello'


  Pod::Block::Code
    say 'hello'


]
True