Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Summary of Alex Zahatski's suggestions for Rakudoc #5

Closed
thoughtstream opened this issue Oct 19, 2023 · 33 comments
Closed

Summary of Alex Zahatski's suggestions for Rakudoc #5

thoughtstream opened this issue Oct 19, 2023 · 33 comments

Comments

@thoughtstream
Copy link

Here’s my summary of Alex’s suggestions
(drawn from an email discussion we had before he submitted them)...

New

  • add C<:id>, C<:caption>, C<:lang> attributes,

These are already in the current specification

  • task lists, C<=item :checked>,

This is not in the current specification,
but I think it certainly should be.

Note, however, that we are also removing "magical" inline options like:

=item *  A bulleted point
=item 1. A numbered point

...so we could not accept another magical inline option specifier like:

=item [ ] An unchecked point
=item [x] A checked point

But adding a :checked metaoption for any block
(or, at least, for =item and =headN blocks)
seems like an excellent idea.

  • add Selectors,

This is a complex new feature and needs separate and more-extensive discussion.

  • advanced table format, C<=row>,C<=cell> blocks, C<:header>, C<:rowspan>, C<:colspan> attributes,

These are already in the current draft specification,
(except that =row is a directive, not a block, and we also support =column
to provide more flexibility in laying out tables).

  • table of contents, C<=toc>,
  • include block, C<=include>,

Even after discussing these with Alex,
I'm still not convinced that these need to be blocks,
rather than kinds of placement links.

I can see Alex’s point that making the table-of-contents a block,
rather than just a P<toc:...> placement link avoids the problem
of how to cleanly render “embedded” TOC placements like:

This is a para with a P<toc: 1..3> in the middle of it.
  • inserting pictures, C<=picture>, C<P<>>,

This is already covered in the current draft specification,
except that the syntax is =graphic and G<...>.
(Because not all graphical elements are "pictures" :-)

However, Alex rightly pointed out that G<image.jpg> for inline images could
just as easily be P<graphic: image.jpg> or even just P<image.jpg>.

Except that, if we were to do that, then we’d have to extend P<...>
codes to allow for an ALT text as well. That might not be a bad idea
anyway, since other kinds of P<...> placements might also benefit
from ALT tags (especially when the source being inlined is not
accessible for some reason).

  • mathematical formulas, C<=formula>, C<F<>>,

This is not in the current specification,
but I think it should be.

The only question is which internal syntax for specifying the formulas
should be used:

Note that the above choices are ordered from my personal most-preferred (Typst or ASCIIMath)
to my OMG-PLEASE-NOOOOOOOOOOOOO!!!!!!!!!!! preferences (MathML, OpenMath, or OMDoc) :-)
I’d be okay with LaTeX syntax, but I think the more humanist Typst or ASCIMath
would fit better into the Rakudoc universe.

  • mistake marking, C<O<>>,
  • text positioning, C<J<>>, C<H<>>,

These are already part of the current draft specification,
with the same codes.

  • Emoji, C<E<>>,

Though I almost never use emjoi myself, allowing E<> to specify emoji
does seem like a reasonable extension that would be useful to many people.
(I confess there are plenty of times I’ve been sorely tempted to document
some annoying bug, misfeature, or other issue with a 💩 emoji ;-)

  • backward links, C<W<>>

It took me some time to comprehend what these are.

Finally, Alex and I concluded the best way to describe them
is that they are P<> codes that display like N<> codes.

That is they are a mechanism for specifying pop-ups (or sidebars or footnotes)
whose contents are drawn from another document (or another part
of the current document).

I think they would be a good addition to the current specification.

Deprecated/removed:

  • C<:margin> attributes,
  • C<=encoding> directive,

These are already removed from the current draft specification

  • C<Ptoc:> due to C<=toc> block,
  • C<Pman:>, C<Pdoc:> due to C<=include> block,

As I mentioned above, we will need to further discuss
the entire issue of placement links vs dedicated blocks.
@finanalyst
Copy link
Contributor

@thoughtstream Thank you for this analysis, and I'm really glad that there has been offline communication between you and Alex.

Regarding the P<> as an inline rather than a block. If a block, then =include seems a good name for it. So =include :toc would be better than P<toc:>, but there are other `P< schema: > that would be better.

I think I'll discuss each of the other ideas separately

@finanalyst
Copy link
Contributor

@thoughtstream @zag @lizmat
I finally have some spare time to look at these extensions - sorry for the wait, and some time to think things through.

First, I think its important to distinguish between the specification for the minimal renderer, and the ability to customise.

The specification we are creating here is entirely backwards compatible, with one minor exception that was never implemented.

The v2 specification allows for customisation, so if eg pod6lite implements eg. =Formula then it can. The only requirement is that the block meets the custom naming with a mix of upper and lower case letters. In fact, my own Raku::Pod::Render renderer comes with =Latex which takes a mathematical formula in Latex and converts it to a formula png.

I think that this 'custom' way must be a part of the specification because, as @thoughtstream points out, there are several markup variations for formulas. So we could have =Latex and =Typist etc

The question is how to signal to the renderer where to get the customisation. This is contained in the DOC use pragma at the start. If a source does not contain any custom blocks, then no DOC use line is needed.

If there is eg =Latex, and =Latex is defined in RakuAST-Render (which it will be), then if a source has =Latex, there must be a line DOC use RakuAST-Render.

Consequently, to use =Formula as in pod6lite, a source file should contain the pragma DOC use pod6lite, or an equivalent.

@zag we did discuss at some point privately how to distinguish between your =Image and the =Image I had implemented. The DOC use pragma seems a better way to do this.

Also, I am willing to try to implement in RakuAST-Render all of the custom blocks you implement for pod6lite, so the pod6lite and RakuAST-Render will be consistent.

However, for the specification, I think we need to keep to the minimum blocks idea.

I am going to discuss custom markup codes in another issue, because there may be more discussion about it. Also, I want to discuss the P<> and block issue separately.

@finanalyst
Copy link
Contributor

Regarding @zag proposals about new markup codes

The fundamental problem is limited 'real-estate'. There are not many ASCII uppercase letters, and most of them are used up. In fact we had to resort to Δ for delta because D was already used for inline definitions.

There is going to be substantial disagreement about individual usage. So I don't like Emoji, but I do like Font-Awesome icons. So, in Raku::Pod::Render, I implemented F<> for font awesome.

I also want to have the space for inline formulas, and for PayPal buttons. Custom blocks is not the answer here because they need to be inline, not a block.

Given the limited 'space' for markup letters, the current specification has two ways for creating custom markup codes: Any Unicode character with the property 'upper' can be used for custom markup, and M< ... | ... > for custom markup. The specification document has an example that shows how this will work.

I am going to need to change my F<> to M< car | Font-Awesome, fa-car>, and so on.

You could have M< smiley | emoji > for example.

This seems to allow the specification the flexibility to have any number of custom markup, without needing to include ones we might like in the minimal specification.

Does this seem reasonable to you?

@finanalyst
Copy link
Contributor

@zag @thoughtstream
@zag did raise a topic that both @thoughtstream and I agree on, namely the mixing of block semantics and inline mark up code syntax.

We have a number of block/inline pairs, for example, =alias and A<> and =defn and D<>.

It seems a good extension of the same paradigm to have =place and P<> as a pair. In this way, where block rendering / semantics is appropriate, we would have eg =place toc: or

=place
toc:

However, even as I write, I am unhappy with this. We already have :toc as a metadata option for all blocks, because we want to control how the block name is included in the Table of Contents. So toc: and :toc would be different, and we could have:

=for place :!toc 
toc:

So, instead may be we could define =place to be a directive, in the same way as =alias is a directive. What comes after the place is the name of the schema. If =place is a directive, then the :toc metadata option is not valid, nor would any other metadata option. Perhaps the full syntax of place would be something like

=place SCHEMA URL ALT

where

  • SCHEMA would be the name of the schema, something like file or toc or rakudoc or link
  • URL would be dependent on the schema, eg file this_directory/some_file.txt or link https://example.com/stuff.
  • ALT would be rendered by the renderer in the event that it cannot access the URL

For the inline version, perhaps something like P< SCHEMA | URL ; ALT>

Actually, I like the idea of having an explicit ALT because internet connection errors play havoc with P<https://...> links.

I also like the idea of eliminating =graphic and inline G<>

My only concern is that =place would not by default affect the Table of Content, but we might want a graphic to appear in a Table of Content, so we could optionally extend to

=place SCHEMA URL ALT CAPTION LEVEL

CAPTION is inserted into the ToC at the position of LEVEL, by default 1

@finanalyst
Copy link
Contributor

@zag @thoughtstream It seems from what I have read that there are suggestions for forward references with W<>.

RakuDoc v2 has a number of components that have forward references (forward may not be the correct term).

The clearest forward reference is P<toc:>. The usual place for a ToC is at the top. However, the renderer cannot add the content of P<toc:> to the rendered stream because it is incomplete until the last =head or =CustomBlock has been encountered.

However, suppose we have P<semantic: AUTHORS> and we define =AUTHORS at the end, but place it at the beginning? Similarly, there is no reason in the RakuDoc specification why =ALIAS TERMS should be specified before A<TERMS> is used.

I think all of these are forward references, and the renderer simply has to find a way to handle them.

@zag
Copy link
Member

zag commented Nov 5, 2023

@thoughtstream Thank you, Damian, for taking the time to analyze my proposal.

I’d be okay with LaTeX syntax, but I think the more humanist Typst or ASCIMath
would fit better into the Rakudoc universe

I really like LaTeX, and I'm glad that it's also acceptable to you. LaTeX is the go-to choice for entering mathematical formulas. I have nothing against other formula formats, but in my opinion, LaTeX is more widely recognized and will interest authors of mathematical articles. Additionally, LaTeX is supported by GitHub, which could be a valuable plus.

Thanks for the additional information about formats. It's entirely possible to implement them as Named blocks and custom markup codes 👍

(I confess there are plenty of times I’ve been sorely tempted to document
some annoying bug, misfeature, or other issue with a 💩 emoji ;-)

Haha, my frequently used emoji is E<:party popper:>: 🥳, especially its animated version:

backward links, C<W<>>

It took me some time to comprehend what these are.

Finally, Alex and I concluded the best way to describe them
is that they are P<> codes that display like N<> codes.

That is they are a mechanism for specifying pop-ups (or sidebars or footnotes)
whose contents are drawn from another document (or another part
of the current document).

For backward links, I've come up with a more precise definition that reflects their essence: contextual backward links.

Rakudoc introduces the concept of contextual backward links C<W<>> (or contextual backlinks),
enhancing the traditional L<linking|#Links> mechanism by allowing to not
only reference another location but also provide specific
context around that reference. This context can take the form of opinions,
additional information, or any relevant commentary that enhances the understanding 
of the referenced material.

Thank you!

@finanalyst , Thank you, Richard, for your attention and ideas.

It seems from what I have read that there are suggestions for forward references with W<>.

I'm not sure they're the same thing. contextual backward link is attributed by the surrounding context and it can be displayed differently on the destination page. Keep in mind what destination can be defn:

This text can reveal the meaning of B<W<glossoligation|defn:glossoligation>>

We have a number of block/inline pairs, for example, =alias and A<> and =defn and D<>

I'm not sure if there's a problem as described.

From my perspective, the =toc block is more related to the group of blocks like =head1, =item, =table, for which inline elements are not used.

When placed within a paragraph, P<toc:...> disrupts the rendering because TOC is a formatted list of elements. So, in my version, P<toc:...> is deprecated.

=place toc:
=place SCHEMA URL ALT CAPTION LEVEL

Regarding the naming, I could provide a more specific response if I knew what you find unsatisfactory about =toc. From my point of view,=toc a simple and clear block name.
It can have the same attributes as other blocks. For example, :caption:

 =for toc :caption('Table of contents')
 head1, head2, head3

I am going to need to change my F<> to M< car | Font-Awesome, fa-car>, and so on.

I'm sorry that F<> was used by you, but we can extend E<> to support Font Awesome 😀.
Just add a prefix or enclose it with a special character.
Any way you prefer, for example:
For example:

E<&fa-face-awesome>
or 
E<&fa-face-awesome&>

What do you think?

So we could have =Latex and =Typist etc

I like the idea of extending support for other formulas, also =ASCIIMath 👍

thank you

To maintain consistency, I'll leave a link to the full text of my proposal here: Rakudoc proposal
It is 100% compatible with already written documentation in Raku Pod (aka Pod6), guaranteed by using the current specification as a base (Synopsis 26).

with best regards,
Alex

@finanalyst
Copy link
Contributor

@zag I have now looked carefully at the proposal linked in the comment above.

For the record, it should be noted that you were included in the group of people with access to the RakuDoc-BETA repo, and I regularly asked for your feedback about various issues. The aim of a collaborative discussion is to avoid the sort of situation we now face with respect to the RakuDoc specification, with two quite separate documents.

During the discussion on RakuDoc-BETA we spent quite a bit of time on

  • the relationship between RakuDoc for documenting Raku code and making information available to IDE's. These include the #| =data and =finish RakuDoc forms.
  • the different paradigms for specifying tables.
    • There is an 'HTML' paradigm, in which rows are created by adding cells, and the columns are automatically created.
    • There is a GTK grid paradigm (which is tracked by the CSS grid specification). Here cells are positioned within a predetermined grid
    • Both paradigms have weaknesses and strengths, but Damian came up with another 'procedural' definition, quite separate from those. The detail and much of the discussion is included in the document in this Repo. In addition, by carefully constructing the syntax, it is possible to have both the original form of Table and the new one. I have implemented it as can be seen by looking at new raku rakudoc.
  • The rational for some significant editing of the order of the specification
  • The rational for distinguishing between 'markup code' and 'formatting code'
  • The rational for new format codes and why they should be semantically named, rather than by reference to their appearance.
  • The rational for the spelling rules to be applied to semantic blocks, built in blocks, and custom blocks.
  • The rational for removing various options, such as :numbered and #.

As a result of those many discussions, Damian, Liz and I created the RakuDoc specification set out in this repo and which is rendered into HTML in the above link. From the posts in RakuDoc-BETA, you will see that Damian, who wrote S26, significantly edited the new specification (going through it with a tooth-comb at least three times).

The new version is backwards compatible with S26 except for some forms that were never implemented.

Backwards compatibility is not guaranteed by keeping to the textual form of the original specfication, but by the way it is rendered.

As @thoughtstream (Damian Conway) pointed out in the opening post, you have some great suggestions.

Since they are good, and we have gone past the deadline for suggestions, the deadline has been extended.

I am certain that some of the suggestions will be included in the new specification. Other ideas create problems, and I will explain them in detail in new posts to this issue.

@finanalyst
Copy link
Contributor

@zag This is about the paradigm for RakuDoc, and the way the specification exists as it does.

RakuDoc is not just targetted at HTML. Even though, HTML is used for websites, there are other formats into which documents do and will appear. These include XHTML (for epub books), pdf, and other other formats for printing.

The aim of RakuDoc is to be declarative in nature so that when new formats appear, the RakuDoc files are not themselves changed.

Consequently references to other markup languages (such as HTML and Markdown, for text, or .dot for directed graphs, or Latex for formulae) are not included in the specification.

There is, and will be, a need to include other markup text. The approach adopted for RakuDoc as hinted at in S26, but expanded far more in the new specification, is to allow for customised blocks.

As mentioned in a previous post, I have already developed custom blocks for Maps, Latex formula, graphs. For historical reasons it is necessary to provide =Html as a Custom block.

As for Markdown, it is essential in the current environment, to ensure that all RakuDoc files can be converted into Markdown output. So there must be a RakuDoc::To::Markdown renderer.

@finanalyst
Copy link
Contributor

finanalyst commented Nov 5, 2023 •
edited
Loading

@zag Concerning emojis

Originally I thought that the idea was a custom markup code. As I tried to argue against the idea, I realised that in fact, you are extending the paradigm of the E<...> markup code. From what I understand, there are tables of emojis with names and glyph values. Since E<> uses Unicode expressions, then emojis seems a reasonable extension.

I've run out of time, but I'll be discussing =place and =include later. There are some really fundamental problems with =include. The construct is in other markup languages and in SASS files, eg. This is not a construct to be included lightly into RakuDoc.

@zag
Copy link
Member

zag commented Nov 6, 2023

Hi, @finanalyst,

I think this discussion is about the ideas from my version of the Rakudoc specification, rather than the specification itself.

For the record, it should be noted that you were included in the group of people with access to the RakuDoc-BETA repo, and I regularly asked for your feedback about various issues. The aim of a collaborative discussion is to avoid the sort of situation we now face with respect to the RakuDoc specification, with two quite separate documents.

But let me remind you that you simply sent your draft version of the specification and suggested working on it.
I want to remind you that I disagreed with your approach and mentioned that I plan to provide my own specification in a closed group and publicly.

I guess you used user documentation of Raku Pod named Pod6 as a basis for the specification instead of the actual Raku Pod specification. In my view, that's incorrect.

In situations like this, I believe it would have been better to gather opinions beforehand. You could do this publicly or write to me and ask for input, especially since we have known each other for a long time, and you are aware of my work on Podlite.

I am certain that some of the suggestions will be included in the new specification. Other ideas create problems, and I will explain them in detail in new posts to this issue.

I'm glad my ideas have generated such interest.
Feel free to incorporate them as you see fit.

Thank you

@finanalyst
Copy link
Contributor

finanalyst commented Nov 7, 2023 •
edited
Loading

@zag we disagree on process, but then we are human both of us, so disagreement is inevitable. The podlite project is something I do want to be able to use in a project I am thinking about. So, I do appreciate your work.

Focusing on the RakuDoc, I want to discuss some of the new ideas you have, and some ways we can get them to work for both a RakuDoc v2 compliant renderer and podlite.

So, I'll list them here:

Emojis

As I mentioned above, adding another syntax to E<> to describe a character in another way seems a reasonable extension. E<> already has ways to specify HTML entities, escaped characters as per the URL specification, and characters using a UNICODE sequence. So the emoji syntax fits in.

Contextual links

As you said, these are extensions of L<> links. You have suggested W as the markup letter.

I would like to see how they could be used in practice with some use cases.

It seems to me that the extension could first be implemented in three ways:

  1. Using a custom markup code letter in the 'custom' range. For example, both ℂ and Ⓛ (Unicode U+2102 and Unicode U+24c1) would fit (or some other character that is a better mnemonic than W).
  2. The L<> syntax already allows for customisation (or perhaps we should make that clearer). The point is that data after the | in several markdown codes has the form <entity>[,<entity>]+ [; <entity>[,<entity>]+]* . That means a contextual link could be specified eg as L<Look at this site'| https://math.example.com/special-function ; Special functions are defined here in a different way to this text>. The renderer can take the extra information given to it as the second parameter after the expected URL and either ignore it (RakuDoc minimal specification) or create some extra text such as a popup or aside.
  3. The M<> markup has been reserved for renderers to handle as they wish.

RakuDoc v2 as it stands allows for renderers to provide new codes and blocks. So, using this convention podlite could immediately implement the new markup, and if it is useful, it will be copied by other renderers.

Place or include

RakuDoc v1 specified P< schema: ... > and as Damian explained the schema's were to be implementation dependent. His main idea was to be able to include not just text from files or and HTTP links, but more broadly any text that could be specified in some way, such as man:, which would take the parameter and textify it using the equivalent of a Unix man command, or by specifying an ISDN number for a publication, retrieve publisher/publication data.

V1 also included the schema toc: and I have suggested this be extended to index: to place the data from X<> markup.

The problem, as you have pointed out, and which Damian accepted in the first post, is that a complete Table of Contents is really a block, not an inline.

The same could also be said of text from large files, or an HTTP link.

So my suggestion is to have both =place and P<> with a consistent syntax and the semantic meaning of place an object here that is retrieved from an external source.

By having both a block =place and an inline P<>, RakuDoc offers the author of the choice as to how it will be rendered.

I do not think it is necessary to deprecate any of the schemas, eg P<toc:>. I can imagine it would be useful to have an inline version of a partial ToC (RakuDoc v1 has a variety of ways to reduce the ToC elements). For example, different partial segments of the ToC could be put into a Table such as in the introduction to a book where the author is explaining about the book's different sections.

@finanalyst
Copy link
Contributor

Closed was a mouse misclick. Reopening

@finanalyst finanalyst reopened this Nov 7, 2023
@finanalyst
Copy link
Contributor

finanalyst commented Nov 7, 2023 •
edited
Loading

@zag @thoughtstream Before making a mistake when inputting my last post, I was describing =place but I wanted to contrast it with =include.

Do not include things.

The reason I feel strongly about =include is that it is used in several other languages to include into the current file contents, the contents of another file, and to treat that included content as the same as the current content.
I had for some time thought it would be good to have some form of =include 'file:/settings.rakudoc'so that the RakuDoc in the settings file could be used uniformly over many rakudoc files.
However, this would create problems related to inheritance and circularity. How for example do you handle the rendering of A.rakudoc which has the line =include B.rakudoc, which in turn has the line =include A.rakudoc?
To avoid all such problems, P<> and I suggest =place is specified in terms of an object such as a png, or simply text. The renderer does not 'look' at the result and is simply inserted verbatim.

@zag By the way, you commented above that you did not see the connection between =alias/A<>, =defn/D<> and the suggested =place/P<> pairs. Indeed each pair works slightly differently. What I meant is that at a more general level we create a language that has a similar pattern for different things.

What about a =toc?

If we rename this block as =ToC, then it obeys the naming / spelling rules for a custom block. Any renderer may specify as many custom blocks as the developer wishes.

I can see that having a separate =ToC block in place of =place toc: would be more convenient. However, by keeping it as a custom block instead of a built in block, we reduce the number of RakuDoc forms, but retain the flexibility of having a more convenient form, a bit like some syntactic sugar.

Selectors

As @thoughtstream commented, I looked at the selectors concept and took a while to work it out, though I don't think I have completely.

Selectors are proposed for =toc and =include (using @zag 's suggested block names). They operate as a way of filtering text that is being drawn in to the final output.

However, I think that from an implementation point of view, selectors are operating on different sorts of objects. The ToC is going to be formed as a data structure during the rendering of the current RakuDoc source. Every =head and custom / semantic block is going to add information to the ToC structure. When the ToC is placed or injected into the final output, the data structure will be rendered and so can be filtered.

On the other hand, when a complete document is 'included', it is not yet specified as to whether the Renderer has to render the new document, or whether it is included verbatim. If the inclusion is as verbatim text, then to implement a selector, the file will need to be parsed for the elements required.

As an aside, I had to implement something similar for the raku docs website, eg message. The routine 'message' is documented in many source files, and the brief was to have the relevant texts brought together in one file. Raku docs contains several hundred Rakudoc source files. The task was handled by a separate plugin that looks at each heading and extracts the information, which is then rendered together in a new file.

The problem is that there is some ambiguity about what to include. For example, if a source contains the following:

=head1 Main heading
text1
=head2 Sub heading
text2
=head2 new subheading
text3
=head1 Back to the main text
text4
etc

And suppose we want to include the section starting with the first =head1. How much is then included. By usage, we mostly want to include all the subheads as well. But sometimes not.

This problem is why the new =section block was introduced to RakuDoc v2. RakuDoc has a heuristic for a section in the absence of an explicit =section.

In addition, =section provides a much neater way of defining the block scope that is needed for =config directives to work as first specified.

Conclusion on include and selectors

I can see some use cases for the proposed additions, but I think there needs to be a set of use cases that gives us a way of abstracting the idea.

Having said all that, so long as the naming rules are adhered to, there is no reason why =Selectors and =Include should not be implemented in podlite. As custom blocks, they are automatically compliant with RakuDoc v2.

Future of RakuDoc

Depending on the usage of RakuDoc, we may see another revision in the future. Then features that are first implemented as CustomBlocks, and found useful, can be included as built-ins.
Obviously, RakuDoc will need to be used far more widely than Raku programs.

@zag 's work on podlite, and the plans I have for multilingual documentation sites, may help in this direction.

@finanalyst
Copy link
Contributor

done it again. Reopen

@finanalyst finanalyst reopened this Nov 7, 2023
@codesections
Copy link

I want to thank everyone involved here for the time and thought they've put into this issue. You've all clearly put a ton of effort into thinking through the various implications. I'm confident that whatever consensus you come to will result in a strong design. And, since I haven't put the same careful attention into the design considerations, I won't weigh in on the substance.

But I do want to raise one procedural point: In the (hopefully unlikely!) eventuality that we can't reach consensus, the Raku community as a whole will decide on the RakuDoc specification using its normal process for making decisions in the absence of a consensus (i.e., the Problem Solving repo and, if needed, RSC involvement). Of course, @zag, if you end up disliking the RakuDoc specification that the Raku community creates, Podlite could use a different specification; you could even fork the RakuDoc spec, as permitted under its license. However, if you end up doing so, we would ask that you use some name other than "RakuDoc" for that specification both to avoid confusion and because the "Raku" label belongs to the community as a whole (in a moral sense, and to the Raku Foundation in a trademark-law sense).

Again, I don't see any reason to think that we'll fail to reach consensus on a RakuDoc spec that works for Raku and for Podlite. But I think it's important to be clear on the procedure just in case ☺

@thoughtstream
Copy link
Author

Do not include things

To avoid all such problems, P<> and I suggest =place is specified in terms of an object such as a png, or simply text. The renderer does not 'look' at the result and is simply inserted verbatim.

I don't think that approach works. The current draft of the spec includes the examples:

=COPYRIGHT
P<file:/shared/docs/std_copyright.rakudoc>

=DISCLAIMER
P<http://www.MegaGigaTeraPetaCorp.com/std/disclaimer.txt>

Both of which insert text with RakuDoc markup in them.
I would fully expect that markup to be honoured in the final rendering,
so I don't see how the insertion can be verbatim or unlooked-at by the renderer.

For a start, the placement mechanism would be much less useful if the placed content
cannot itself have placed content. This would be analogous to Raku modules not being able
to use other modules.

The circular inclusion problem is, frankly, a solved problem. The renderer simply uses a stack
to track each URL that is placed (either placed directly by the current document or placed recursively
by some already-placed document) and flags an error when it comes across a nested placement request
whose URL is already in the stack. It need not even flag an error, if we prefer. It could simply not place the
nested circular URL at all; just render it as a URL (the way it does when the URL target is missing).

@thoughtstream
Copy link
Author

Placement blocks and inlines

Here are my thoughts about the notion of adding a =place block/directive
to complement the existing P<> inline markup.

In short, I am strongly in favour of this, but I think both mechanisms
need a little more design work to cover all our requirements.

I will also touch on a few other issues in my discussion, including the removal
of the =graphic/G<> pair and whether P<toc:...> should be =toc instead.

1. Inline vs block placements

We definitely should support a block-oriented placement form, but it is
a tricky question whether that form should be a block or a directive.

Personally, I suspect =place should be a directive, because blocks
(such as =head, =para, =code, =item, =cell, etc.) all contain
actual content, which is to be directly rendered in some way.

In contrast, directives (such as =alias, =column, =row, =config, etc.)
all specify behaviours rather than content. Those behaviours
affect what is rendered, but the behavioural information in the directive
is not itself directly rendered.

Looked at from that perspective, =place is clearly more like a directive
than a block. The URL is behavioural information (“Go here and insert what you find there”)
rather than direct contents itself. The URL is not intended to be directly rendered,
and is in fact only ever rendered as a last-resort representation
in the case of a missing-target error.

So, to complement the P<URL> construct I think we need a =place URL directive.
For example:

    =COPYRIGHT

    =place file:/shared/docs/std_copyright.rakudoc


    =DISCLAIMER

    =place http://www.MegaGigaTeraPetaCorp.com/std/disclaimer.txt

    =place https://www.MegaGigaTeraPetaCorp.com/logos/logo_small.jpg


    =head1 Contents

    =place toc:1..3


    =head1 Credits

    =place semantic:AUTHORS


    =head1 Index

    =place index:

2. Graphics as placements

As the above example implies, I think that graphic inserts (images, charts, diagrams, etc.)
could be incorporated into the P<> and =place syntax, without even the need to add an
extra schema. We simply specify that, if the URL leads to a graphic file (JPEG, PNG, GIF, MPEG, etc.),
that graphic is emplaced at the location.

This would mean we can remove the =graphic and G<> syntaxes, freeing the latter up
for some future use.

However, as @finanalyst has already pointed out, unlike the other placement types,
graphics often require more associated information than just a source URL.
For a start, for proper accessibility they should always be specified with an ALT text.
In addition, they are often given captions. And they may need to be included in the ToC.

Blocks can already be given captions and included in the ToC, through the use of the
appropriate :caption and :toc metadata options. There’s no reason that a =place
directive couldn’t allows those as well, plus an :alt option for the ALT tag.
Note that we already have prior art on directives that take metadata options:
the =config directive already does so.

So we could allow placements like:

    =place http​s://raku.org/camelia-logo.png :caption<Raku's mascot> :alt<A multicoloured butterfly>

...and:

    =place semantic:AUTHORS :toc(1)

We also have prior art on inlines that take ALT-like texts: the optional “visible text” fields
of the Δ<>, L<>, M<>, and X<> markup instructions. Indeed, when we thought we needed
a separate G<> instruction, its ALT text was specified in just that way.
So there’s no reason why the P<> instruction couldn’t provide that option as well:

    Our logo is a butterfly: P<The Camelia butterfly|http​s://raku.org/camelia-logo.png>.

This might even be handy in the general case. We could change the rule for how missing
placements (both =place and P<>) are rendered. Namely: if the missing placement has an ALT text,
it is rendered as that ALT text; otherwise it is rendered as its URL. For example:

    =place http​://www.MegaGigaTeraPetaCorp.com/std/disclaimer.txt :alt<You can't sue us for ANYTHING!!!>

..or:

    Our logo is a ravenous lion: P<a lion salivating | http​s://clipground.com/lion-eating-clipart.html>

..or:

    =place semantic:AUTHORS :alt<We're not sure who created this>

3. Inline vs block ToCs

As for the question of whether we need a separate =toc directive (not block!),
I incline towards thinking that we don’t need it.

In my view, the range of heading levels that a =place toc:1..3 or a P<toc:1..3> specifies
is simply the “address” portion of a toc: URL.

And, if my above suggestion for adding metadata options to the =place directive
were adopted, then tables of content can still be captioned or ALT-texted:

    =place toc: 1..3 :caption<Table of Contents> :alt<MISSING ToC HERE!>

I think it’s better that every kind of insertion into the text should be specified
with a =place (or P<>), rather than some of them using those mechanisms
whilst others using their own separate special-purpose directive.

I’d rather we had the uniformity of:

    =COPYRIGHT
    =place file:/shared/docs/std_copyright.rakudoc

    =DISCLAIMER
    =place http​://www.MegaGigaTeraPetaCorp.com/std/disclaimer.txt
    =place https://www.MegaGigaTeraPetaCorp.com/logos/logo_small.jpg

    =head1 Contents
    =place toc:1..3

    =head1 Credits
    =place semantic:AUTHORS

    =head1 Index
    =place index:

...than the semi-arbitrary diversity of:

    =COPYRIGHT
    =place file:/shared/docs/std_copyright.rakudoc

    =DISCLAIMER
    =place http://www.MegaGigaTeraPetaCorp.com/std/disclaimer.txt
    =graphic https://www.MegaGigaTeraPetaCorp.com/logos/logo_small.jpg

    =head1 Contents
    =toc 1..3

    =head1 Credits
    =place semantic:AUTHORS

    =head1 Index
    =index

@zag
Copy link
Member

zag commented Nov 10, 2023

hi, @codesections ! Thank you for the positive vibes! Your acknowledgment of the hard work put into the RakuDoc specification is motivating. I love that my ideas improve Rakudoc 🥳

I appreciate the clarification on the use of the "RakuDoc" name. I just to provide some additional information from my side. I've used a separate branch specifically to prepare the RakuDoc proposal:

https://github.com/zag/specs/tree/rakudoc-proposal

The name "RakuDoc" is only used within this branch for the purpose of presenting my work.
I'm still developing ideas without using the name "Rakudoc". There might be a few "Raku" leftovers in the working file based on the updated Synopsis26, but I'm on it, fixing those.
If you have any thoughts on that, feel free to let me know.

On another note, I want to highlight that I've intentionally chosen to stay away from discussions that get too personal. I truly believe that focusing on ideas and keeping the conversation positive is the key to progress.

Sparkling and smoky discussions are eye-catching, but it's the warmth that drives progress.😄

Thank you for your understanding

Warm regards,
Alex

@finanalyst
Copy link
Contributor

@thoughtstream

Placement of RakuDoc sources

I don't think that we have ever stated explicitly that a P<file:/somefile> would ever be considered as anything but text. If circularity is a solved problem (how to torture implementors!), then perhaps we should indicate that the contents of a rakudoc source should be rendered.
Eg P<rakudoc:/somefile.rakudoc>
Since any statement like this would inevitably set up a block scope, any =config directives in the rakudoc source would only be true of that block scope.

=place in a block

You have an example

=COPYRIGHT
=place file:/shared/docs/std_copyright.rakudoc

What is to be the contents of the COPYRIGHT block?

Should this not be

=begin COPYRIGHT
=place ...
=end COPYRIGHT

@thoughtstream
Copy link
Author

Placement of RakuDoc sources

I don't think that we have ever stated explicitly that a P<file:/somefile>
would ever be considered as anything but text.

My original intention (and still my preferred approach) is that
the scheme merely specifies where to look for the external content:

Schema Where to look for the resource
http(s): Look on the web
file: Look on the local filesystem
rakudoc: Look in the usual places for Raku documentation
man: Look via a local man(1)
defn: Look in the current document
etc: etc.

Meanwhile, the kind of content being placed is inferred from the
final extension of the URL:

Extension How to treat the contents
.txt Render the contents as plaintext
.rakudoc Render the contents as RakuDoc
.html Render the contents as XHTML
.md Render the contents as Markdown
.json Render the contents as JSON
.jpg Render the contents as an image
.mp4 Render the contents as a video
.etc etc.

Renderers are not required to render anything but plaintext and RakuDoc,
but may also support other formats if they wish.

In the case where a URL does not end in a recognized extension:

    =place https://example.org/landingpage

    =place rakudoc: App::Rak

    =place file:/usr/share/legal/std_disclaimer

...then the type of content (and hence rendering) may be inferred from the schema:

Schema Inferred content type if no final extension
http(s): XHTML
file: plaintext
rakudoc: RakuDoc
etc: etc.

If circularity is a solved problem (how to torture implementors!),
then perhaps we should indicate that the contents of a rakudoc source should be rendered.
Eg P<rakudoc:/somefile.rakudoc>

Agreed.

Since any statement like this would inevitably set up a block scope,
any =config directives in the rakudoc source would only be true of that block scope.

Yes. Every kind of placement should certainly be treated as being in its own block scope.
That is much safer than allowing =config directives to leak out of a placement and
into the rest of the document.

=place in a block

You have an example

=COPYRIGHT
=place file:/shared/docs/std_copyright.rakudoc

What is to be the contents of the COPYRIGHT block?

Nothing.

In that example, the =place simply follows on after the semantic block.
And the =COPYRIGHT is merely being used as a shortcut for =head1 COPYRIGHT.

Should this not be

=begin COPYRIGHT
=place ...
=end COPYRIGHT

Probably. That’s certainly a cleaner use of a semantic block
than the old-school approach I took in that example. :-)

finanalyst added a commit that referenced this issue Nov 12, 2023
@finanalyst
To address issue #5
ef6c493 
- first draft for =place
- changes needed to remove =graphic
- summary tables need changing
finanalyst added a commit that referenced this issue Nov 13, 2023
@finanalyst
To address issue #5 (#10)
54ed577 
* To address issue #5
- first draft for =place
- changes needed to remove =graphic
- summary tables need changing

* incorporate review, remove =graphic/G references
@finanalyst
Copy link
Contributor

@zag @thoughtstream I was looking at the emoji suggestion. There is actually no need to change the specification because E<> already takes Unicode combinations, which is documented.
So, even now, E< grinning face > will generate 😀
The current document contains an example with a Ukrainian flag, and I could add an example with an emoji.

@thoughtstream
Copy link
Author

There is actually no need to change the specification because E<> already takes Unicode combinations, which is documented.

Agreed. But I also agree that an example of this feature would be helpful.

@raiph
Copy link

raiph commented Nov 14, 2023

I've read through this thread carefully three times. I have a couple "technical" things which I'll write as separate comments, but otherwise little to express beyond the emojis I chose during a fourth pass to summarize my thoughts and feelings about each comment. It's not often I read such a design thread and end up getting to marvel at both the substance and the quality of the exchange. Thanks! I'd like to celebrate the palpable sense of family and offspring -- of multiple "parents" and their multiple "children" -- by writing a comment that I look forward to seeing output via a future Rakudoc renderer: E<MAN, EMOJI MODIFIER FITZPATRICK TYPE-6, ZERO WIDTH JOINER, WOMAN, EMOJI MODIFIER FITZPATRICK TYPE-1-2, ZERO WIDTH JOINER, GIRL, EMOJI MODIFIER FITZPATRICK TYPE-3, ZERO WIDTH JOINER, BOY, EMOJI MODIFIER FITZPATRICK TYPE-4]> (👨🏿‍👩🏻‍👧🏼‍👦🏽)

@raiph
Copy link

raiph commented Nov 14, 2023 •
edited
Loading

This comment and my next are two design suggestions. Apologies if either or both have already been discussed / accepted / rejected.

I see that RakuDoc has some support for what look like named parameters/arguments for some constructs.

For example in =config code :delta(v2.3.2+, 'oFun enhancements'), I presume the position of the :delta argument floats among other arguments of the same general form (:foo). But can it be put before the code?

More generally, is this "named" parameter/argument support pervasive across other constructs?

For example, Richard wrote about =place SCHEMA URL ALT CAPTION LEVEL. The SCHEMA etc is clearly metasyntax. Are any of those 5 named parameters?

What about in P<...>? It looks like the general principle is to use separators like |, , and ; in inlines, and I see merit in that, but I still think named parameters/arguments are wonderful for authors and readers for the same reasons they are in regular code, but even more so, because endweight considerations are amplified when reading a mix of things like URLs and alt= arguments.

@raiph
Copy link

raiph commented Nov 14, 2023 •
edited
Loading

I recall wondering a while back what P<> was. Then I read that it was "place". I recall feeling that was slightly odd. But still, it's just paint, so the wondering wandered out of my head and was forgotten.

Then the discussion arose as to whether the content being placed was or wasn't to be interpreted. So, there's an ambiguity about what "place" might mean, and that must be resolved. Further, I hope it's resolved as Damian suggests, and that got me thinking.

If =place ... / P<...> are about going to some content/data elsewhere and trying to process/parse it before deciding what to do about it, what if:

  • Immediately inline rendering of the result of that processing/parsing was merely the default behavior of a place directive/inline? What if it also covered processing/parsing content/data from elsewhere and then :!render specifies not rendering it (instead just making the results of the processing/parsing available to the RakuDoc's further processing)?

  • The key concept/word was "parse" (or "process" or some other concept that generalizes away from "place" toward parsing/processing)? So that, when reading RakuDoc, one reads =process ... or =parse ... (or P<...>), and the "place" sense is mentioned in the documentation of this directive/inline pair as a handy alternative mnemonic for the "P" that expresses the intuitive implicit default result of the parsing/processing?

@thoughtstream
Copy link
Author

@raiph asked:

Are any of those 5 named parameters?

Yes. Named arguments are clearly much better than positionals
when the number of arguments is greater than one.

The current proposal is that the syntax is simply: =place URL
but that the =place directive also has named metadata options for the other three possible arguments:

=place file:standard_copyright.rakudoc

=place file:standard_copyright.rakudoc  :alt<ALT text here>

=place file:standard_copyright.rakudoc  :caption<caption here>

=place file:standard_copyright.rakudoc  :toc<ToC level here>

As for P<>, only the ALT text is relevant (because inlined placements have nowhere to put a caption,
and they really shouldn't be part of the ToC), so we just provide P<URL> and P<ALT | URL> to cover
those two cases.

I actually did consider whether to extend the inline syntax to allow named arguments, but decided that
it just made those instructions too complex, both visually and syntactically. The goal is to keep inline instructions
as simple as possible: either with just content, or with content-plus-ALT-text.

@thoughtstream
Copy link
Author

@raiph asked:

what if ... the key concept/word was "parse" (or "process" or some other concept
that generalizes away from "place" toward parsing/processing)

I'm all in favour of finding a better keyword than =place, if we can. But I couldn't.
I did consider =put, but I didn't think that was as distinct or as precise.

My problem with generalizing =place links to be =parse links or =process links
is that I think either of those terms are far too general. Everything in a RakuDoc document
is parsed, and everything is then processed. So neither =parse nor =process
make it clear what a placement link is actually doing, which is that it's incorporating
external content in some way. (Frankly, =incorporate would be perfect.
Except that it doesn't start with a 'P', which is essential for syntactic consistency
at this point.)

What if it also covered processing/parsing content/data from elsewhere and then :!render specifies not rendering it (instead just making the results of the processing/parsing available to the RakuDoc's further processing)?

Parsing a particular block and making the results available for further processing is one of the main use-cases
for user-defined blocks and user-defined inline codes. So we actually already have this capacity,
without overloading the behaviour of the placement-link mechanism.

@finanalyst
Copy link
Contributor

closing as completed

@raiph
Copy link

raiph commented Dec 8, 2023

Hi @finanalyst

closing as completed.

To recap a comment I wrote earlier:

It's not often I read such a design thread and end up getting to marvel at both the substance and the quality of the exchange. Thanks!

That said, I'm curious what process was followed prior to closing it. I'm used to seeing a general theme in which, to some degree, there are linked commits, or linked new issues, or discussion that concludes a point raised has been sufficiently addressed, before it gets closed.

It looks like that doesn't apply to this issue. For example, near the start of @thoughtstream's opening comment he wrote:

New

task lists, C<=item :checked>,

This is not in the current specification, but I think it certainly should be.

An in-page search for "checked" only caught his own further commentary ("adding a :checked metaoption for any block (or, at least, for =item and =headN blocks) seems like an excellent idea") and a search for "task lists" didn't catch anything other than the quote from the email Damian was responding to.

So I'm wondering which way to think about things:

  • In general, it was a productive discussion, and one can expect what was agreed (eg Damian and Alex presumptively agreed about :checked, and no one disagreed) have been, or definitely will be, incorporated into the RakuDoc spec text, even though, evident agreement aside, there's no indication in this issue that they have been or will; or

  • In general, while it was a productive discussion, and readers can reasonably assume that what appeared to have been agreed will not be blocked from inclusion in the RakuDoc spec, it will be up to participants to chase things to write commits or PRs that add corresponding verbiage to the spec text; or

  • Something else.

?

Thank you for your focus on RakuDoc!

@finanalyst
Copy link
Contributor

finanalyst commented Dec 8, 2023 via email

@thoughtstream thoughtstream mentioned this issue Dec 9, 2023
Open
@raiph
Copy link

raiph commented Dec 9, 2023

Hi @finanalyst

Does this help?

Yes, thank you. I still had questions about 6 thru 8, but given Damian's recent comment and Liz's thumbs up of it I'm going to leave them unspoken. I'm going to presume the collaboration with Alex, including the processing of this issue, went well, notwithstanding the lack of links to commits and confusion about :checked.

Happy holidays! 🙏

@thoughtstream
Copy link
Author

I've just opened an issue (hopefully the last! :-) regarding the :checked proposal.

@finanalyst
Copy link
Contributor

@raiph Liz just pointed me at the reddit comments about Pod. I had not read your input there when I wrote my comment above. Hopefully, you might have seen that a recent commit includes Alexandr as an author of the revision document, something he was happy to agree to. We may not agree about everything, but I think we all respect each others' work and contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@zag @finanalyst @thoughtstream @raiph @codesections