Restore tested code listings

[krilnon]

The shipping book uses a custom syntax for its tests; we preserved these as DocC comments in the Markdown source, but use regular Swift code blocks to render them without running and testing the code as part of the build process. We should evaluate how to bring these tests back, potentially using Swift Snippets.

[amartini51]

The comments in the markdown that show the version of a code listing that was tested under the RST-based pipeline have a marker at the start of each that tells you what kind of line it is. Those markers are as follows:

Input to the REPL

-> (or no marker) text sent to the REPL when testing, and included in the code listing when publishing

>> text sent to the REPL when testing, but omitting from the code listing when publication

--- a blank line, included in the code listing when publishing

Output from the REPL

<< expected output from the REPL when running the test

<~ expected output from the REPL when running the test, checking against the beginning of the line

<$ expected output from the REPL when running the test, checking against the end of the line

<- expected output from the REPL when running the test, included as a "Prints" comment when publishing

Expected Errors

!! an expected error message from the compiler

!$ an expected error message from the compiler, checking against the end of the line

!!- an expected blank line in an error message from the compiler

xx an expected run time trap/error

Tested Comments

/> marks a tested comment, using the same syntax as print — like "Now the value of x is (x)"

</ marks the expected result for a tested comment — like "Now the value of x is 12" — included as a comment when publishing

[amartini51]

For an example of the kinds of mistakes that tested code listings prevents, see #45. In that case, I had to write some code listings without tests, because the "swifttest" portion of the legacy pipeline wasn't set up to test async code.

[natikgadzhi]

@amartini51, was the code that implemented the REPL checks open source? I got curious, and considered poking around a little bit.

Do you think a capability to execute swifttest code blocks in the same manner would be useful for docc at large? Perhaps that's something that could be implemented upstream in docc?

[amartini51]

When releasing TSPL as an open-source project, only the English text was included, not any of the old build system's code. Before migrating TSPL to DocC, the book was build using a Sphinx with a number of custom directives. The code that handled the swifttest blocks was also responsible for formatting and syntax highlighting code listings, in addition to running the test code. The code to parse and run swifttest tests isn't especially complex; if DocC had a way for books to define custom syntax blocks or a pre-processor, it wouldn't be particularly difficult to re-implement.

DocC has some support for testing code listings using snippets (see Kyle's notes above). It's probably possible to adopt snippets for parts of the book — I'd be inclined to start with the guided tour as a test. TSPL's needs are a little unusual compared to other documentation — the book includes many tests that are expected to not compile, testing prose that makes assertions about the language's syntax, and many tests that are expected to trap/crash at run time.

[natikgadzhi]

@amartini51 thank you for linking Snippets. I did not know they were a thing — did not stumble at them when I read up on Swift DocC documentation. Perhaps I document them 👀

So, DocC already has support for snippets code blocks that are built at docc build time, but they don't support the syntax you described above. I don't have a gut feeling that adding REPL commands support to snippets is a universally good idea, but that sounds like something that could be a good addition — should I draft an RFC for it and try and spike on a proof of concept for it?

I imagine that:

• It should be an opt-in support for REPL-style commands.
• Building it would be a little bit tricky, since those are not valid swift, they won't just compile / parse with swift-syntax. It would require an extra pre-compilation step to extract them I guess.
• We might need an extra command, i.e. docc snippets test or similar that would run all snippets and their REPL-style commands and exit with 0 if successful, or fail with an error message otherwise

Something like that?

UPD:

• I'm catching up on Snippets evolution proposal, and will play around with them, see what the implementation actually looks like.
• I see the evo proposal mentions literate programming techniques that would inline snippet code into markdown. It feels like what we have in swift-book are more like these. There are also projects out there that do similar things, like runme.dev.
• Naive optimism: it feels like if we could make DocC inline @Snippet support, and if we support REPL-like commands within snippets in comments, we could come up with something workable?
• swift-book swifttest code blocks are hidden, and Snippets are designed to be mostly visible with a guarantee that they build. I wonder if there is a way to align the two.