Docception

Run doctests on arbitrary markdown files.

Usage

$ mix docception markdown-files

See the example/ project on how to use an alias in mix.exs to run Docception with mix test.

Installation

def deps do
  [
    {:docception, "~> 0.3.3", runtime: false}
  ]
end

Disclaimer

This tool executes any Elixir doctest it encounters (think eval). Ensure that it does not encounter any harmful code!

Example

This file can also be checked with docception. The following example is run through doctest and results in an error:

iex> :hello
:crash

$ mix docception README.md
** (ExUnit.AssertionError)

Doctest failed
code:  :hello === :crash
left:  :hello
right: :crash

The following example works:

iex> :hello
:hello

TODOs

• Clean up temporary directory where .beam files are stored

• Do not hard-code the temporary directory

• Fix "wrong indent" warnings for heredocs

• Try to fix wrong line number reports (the offset is always the same!)

• Check if this also works when a doctest refers to dependencies of the project

• Document how to use it with an alias in order to simplify running it in CI

• Publish

• Determine how to ensure that errors are written before the task ends; avoid stopping ex_unit in mix task.

  The root cause of this is `ExUnit.CLIFormatter`. The formatter writes the error message.
  Docception can possibly exit before the formatter is done writing. This could be solvable
  by using a wrapper around `ExUnit.CLIFormatter` as a custom formatter instead. This custom
  formatter would then be shut down manually. Alas, I did not find a way to inject such a
  custom formatter.
  

• Check if anybody is actually interested in this

How..?

Docception's approach is pretty simple:

1. Read in a markdown file
2. Create an Erlang Form that makes the file look like something that can be executed.
   1. Add a module attribute
   2. Export an __info__/1 function which wraps module_info/1
3. Compile that thing into a binary
4. Store the resulting .beam in the filesystem to make Code.fetch_docs/1 work
5. Call into the undocumented-and-totally-not-for-public-use ExUnit.DocTest.__doctests__/1 function
6. For each of the quoted doctests, spawn a process and call let it run the quoted doctest
7. Gather the results and raise on error

So, except for calling into __doctests__/1, this is pretty straight forward. Note that the implementation relies heavily on the hard work that went into elixir_erl.erl from Elixir itself. Docception needs to create a binary where the 'Docs' chunk matches such chunks as generated by elixir_erl.erl.