Look at mdbook-keeper

[mgeisler]

We currently have a number of code blocks marked with compile_fail: this is often because they depend on third-party crates which aren't available when running mdbook test.

We should look into using https://github.com/tfpk/mdbook-keeper/ to handle testing instead. It explicitly supports using third-party crates in code snippets.

[mgeisler]

This is of course a bit of an open ended issue 🙂 What I hope for is that someone will look at what mdbook keeper can do for us:

• can it run our current tests?
• can we use it to run tests with more dependencies?

The goal would be remove the compile_fail from our code blocks.

If you can get this working locally, then we can work on a updating the GitHub action which runs the tests for us on every PR.

[mauvealerts]

I started looking into this. Some issues/notes so far:

• Tests in sub-chapters aren't run (Extract tests from sub-chapters tfpk/mdbook-keeper#2)
• It doesn't seem to understand {{#include some/file.rs}}
• Output doesn't appear incrementally
• The tests block progress on mdbook build and mdbook serve

On my machine, it takes ~3min to run the tests which is pretty rough when I just want to mdbook serve. I think there's hope, potentially using:

• maybe includes can be fixed via ordering preprocessors
• controlling when it runs, locking to a renderer, possibly with a custom renderer

I intend to work on this some more. In case my work stalls out, or anyone's curious, my keeper branch has the WIP.

[mgeisler]

Thanks @mauvealerts! That's already a very valuable analysis! For example, I had not realized that you run the tests all the time... so we would definitely only enable it on demand like we do for the mdbook-xgettext renderer.

[mauvealerts]

I have something that I'm fairly happy with, when combined with all of these changes to mdbook-keeper

• Extract tests from sub-chapters tfpk/mdbook-keeper#2
• Add support for compile_fail tfpk/mdbook-keeper#3
• Use --crate_type=lib with no_run tfpk/mdbook-keeper#4

I have things running comparable to mdbook test. IMO the steps necessary to make a mergable PR are:

• Decide where the skeleton project should go (currently mdbook-keeper-deps)
• Don't run tests by default for mdbook serve
• mdbook-keeper has released a version with the changes
• Prove out the usefulness by testing some snippets

[mgeisler]

Thanks for pushing this!

• Decide where the skeleton project should go (currently mdbook-keeper-deps)

I think this should be the root of the repository, or perhaps the src/ folder. We already have a Cargo workspace setup so that cargo check does something useful for the code in our exercises. I hope we can build on this for the book itself: perhaps we can move up the src/exercises/Cargo.toml file to src/ and then use its dependencies across the examples?

[mauvealerts]

I didn't think it was possible to have both a [workspace] and (e.g.) a lib in the same Cargo.toml, but apparently it is. I've updated my branch to just use Cargo.toml in the root directory, at least for now. I've also locked mdbook-keeper to the markdown output format, so it doesn't slow down mdbook serve

I started updating some examples, but ran into link errors (tfpk/mdbook-keeper#9). In particular, not being able to use tokio on Windows makes me hesitant to migrate the async examples (which was most of what I planned to do).

[mgeisler]

I've also locked mdbook-keeper to the markdown output format, so it doesn't slow down mdbook serve

Ah, that's a clever work-around 😄 However, it almost suggests that mdbook-keeper should be a renderer instead of a preprocessor. @tfpk, have you considered that approach?

I started updating some examples, but ran into link errors (tfpk/mdbook-keeper#9). In particular, not being able to use tokio on Windows makes me hesitant to migrate the async examples (which was most of what I planned to do).

Oh, I see... I agree that it would be important to have test coverage for all the small code snippets involving Tokio and that was also my main reason to look at mdbook-keeper 🙂