New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate tests from code example docstrings (to fight bitrot) #2925

Closed
olsonjeffery opened this Issue Jul 15, 2012 · 8 comments

Comments

Projects
None yet
7 participants
@olsonjeffery
Contributor

olsonjeffery commented Jul 15, 2012

So I'm going through my contributions and adding code examples for most of the API stuff out there that isn't immediately obvious.

I only have one example of an existing code sample that is live in the libstd docs right now, and that's for net::tcp::listen. Suffice to say: it is very out-of-date with the current state of the art for rust. This is a sad discovery, on my part.

Since more literate code examples in the docs is probably a good thing, this is where I'm putting my energies for documentation of my work (as opposed to blog posts, emails to the list, etc). I'm trying to make the examples complete, illustrative and durable (the latter of which I have no control over).

Cutting to the chase:

I'd be happy with something like what (I'm told) we have in the tutorial markdown: have the build automation (rustdoc, in this case?) do a run-pass on all code blocks (by some known convention) that appear in docstrings, for the greater good.

Thoughts?

@catamorphism

This comment has been minimized.

Show comment
Hide comment
@catamorphism

catamorphism Jul 20, 2012

Contributor

Yes, this seems like an obviously good idea to me. I think it is an RFC-level change, though, since it would result in rejecting more programs (which is a good thing!), so I'm adding the RFC label.

Contributor

catamorphism commented Jul 20, 2012

Yes, this seems like an obviously good idea to me. I think it is an RFC-level change, though, since it would result in rejecting more programs (which is a good thing!), so I'm adding the RFC label.

@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson Jul 26, 2012

Contributor

The easiest way is probably to use rustdoc to convert the documentation to markdown, then run the python script over it to generate tests.

Here's my preference:

  • Move the --test flag out of rustc completely, to a new tool called rusttest (or just leave all this logic inside rustc, but I'm wary of continuing to stuff non-compiler stuff into rustc)
  • Teach rusttest to parse test cases out of doc attributes the way that the existing python script does
  • Inject new tests into the AST based on the docs
  • Compile the whole thing as a test runner

Later, once we've created a unified rust tool we would end up just writing rust test foo.rc

Contributor

brson commented Jul 26, 2012

The easiest way is probably to use rustdoc to convert the documentation to markdown, then run the python script over it to generate tests.

Here's my preference:

  • Move the --test flag out of rustc completely, to a new tool called rusttest (or just leave all this logic inside rustc, but I'm wary of continuing to stuff non-compiler stuff into rustc)
  • Teach rusttest to parse test cases out of doc attributes the way that the existing python script does
  • Inject new tests into the AST based on the docs
  • Compile the whole thing as a test runner

Later, once we've created a unified rust tool we would end up just writing rust test foo.rc

@pnkfelix

This comment has been minimized.

Show comment
Hide comment
@pnkfelix

pnkfelix Apr 25, 2013

Member

So, this issue is a request for infrastructure for generating execution unit tests from the embedded documentation for a symbol. That sounds like a match for Maturity Milestone 4: well covered.

Member

pnkfelix commented Apr 25, 2013

So, this issue is a request for infrastructure for generating execution unit tests from the embedded documentation for a symbol. That sounds like a match for Maturity Milestone 4: well covered.

@ghost ghost assigned 6a68 and cmr Jul 5, 2013

@cmr

This comment has been minimized.

Show comment
Hide comment
@cmr

cmr Jul 5, 2013

Member

Absorbing under rustdoc2

Member

cmr commented Jul 5, 2013

Absorbing under rustdoc2

@cmr

This comment has been minimized.

Show comment
Hide comment
@cmr

cmr Aug 15, 2013

Member

Tests can be extracted from the doc comments present in the JSON rustdoc_ng emits. Tooling needs to be made to do so, and to build+run the tests.

Member

cmr commented Aug 15, 2013

Tests can be extracted from the doc comments present in the JSON rustdoc_ng emits. Tooling needs to be made to do so, and to build+run the tests.

@pnkfelix

This comment has been minimized.

Show comment
Hide comment
@pnkfelix

pnkfelix Aug 18, 2013

Member

part of #8125

Member

pnkfelix commented Aug 18, 2013

part of #8125

@cmr

This comment has been minimized.

Show comment
Hide comment
@cmr

cmr Oct 24, 2013

Member

Unassigning self.

Member

cmr commented Oct 24, 2013

Unassigning self.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Dec 19, 2013

Member

Nominating, this is the only way we're going to have code examples be reliable

Member

alexcrichton commented Dec 19, 2013

Nominating, this is the only way we're going to have code examples be reliable

bors added a commit that referenced this issue Dec 23, 2013

auto merge of #11120 : alexcrichton/rust/rustdoc-test, r=brson
This commit adds a `--test` flag to rustdoc to expose the ability to test code examples in doc strings. This work by using sundown's `lang` attribute to figure out how a code block should be tested. The format for this is:

```
1. ```rust
2. ```rust,ignore
3. ```rust,notest
4. ```rust,should_fail
```

Where `rust` means that rustdoc will attempt to test is, `ignore` means that it will not execute the test but it will compile it, `notest` means that rustdoc completely ignores it, and `should_fail` means that the test should fail. This commit also leverages `extra::test` for the testing harness in order to allow parallelization and whatnot.

I have fixed all existing code examples in libstd and libextra, but I have not made a pass through the crates in order to change code blocks to testable code blocks.

It may also be a questionable decision to require opting-in to a testable code block.

Finally, I kept our sugar in the doc suite to omit lines starting with `#` in documentation but still process them during tests.

Closes #2925

bors added a commit that referenced this issue Dec 23, 2013

auto merge of #11120 : alexcrichton/rust/rustdoc-test, r=brson
This commit adds a `--test` flag to rustdoc to expose the ability to test code examples in doc strings. This work by using sundown's `lang` attribute to figure out how a code block should be tested. The format for this is:

```
1. ```rust
2. ```rust,ignore
3. ```rust,notest
4. ```rust,should_fail
```

Where `rust` means that rustdoc will attempt to test is, `ignore` means that it will not execute the test but it will compile it, `notest` means that rustdoc completely ignores it, and `should_fail` means that the test should fail. This commit also leverages `extra::test` for the testing harness in order to allow parallelization and whatnot.

I have fixed all existing code examples in libstd and libextra, but I have not made a pass through the crates in order to change code blocks to testable code blocks.

It may also be a questionable decision to require opting-in to a testable code block.

Finally, I kept our sugar in the doc suite to omit lines starting with `#` in documentation but still process them during tests.

Closes #2925

@bors bors closed this in d9c0658 Dec 23, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment