Make documentation code samples subject to regular automated tests

[DavidMGross]

(This was discussed in Issue #20 by @headinthebox, @GeorgiKhomeriki, @samuelgruetter, @benjchristensen, @staltz & @DavidMGross but deserves a stand-alone issue.)

The documentation is full of code samples. To ensure that these are (and remain) accurate, they should be tested and these tests should be re-run whenever changes are made to the samples, the ReactiveX implementation they are written against, or relevant parts of the language/OS they run within.

How to do this well is a bit of a puzzle. Some of the issues/ideas:

• we could have the code samples in the test suites for the various ReactiveX implementations, and then pull the code directly from those samples to insert in the appropriate places in the web pages; single-sourcing the samples in this way would ensure that they don't get out-of-sync.
  • however: many code samples are just sections of code that aren't executable stand-alone. To make them testable would mean adding a lot of cruft around them that would make for inappropriate noise in the documentation context. We'd need to have some way (or perhaps a variety of language-specific ways) of isolating the part of the code that is meant for documentation from the part that's necessary to make it testable.
  • we'd need a strategy for pulling the code samples into the documentation
    • if we do this dynamically when the page is requested, this puts extra load on the server
    • if we periodically build static pages via a make-like process, we need to trigger the make appropriately
• Many of the samples in the documentation also show the expected output; is there a way to keep that automatically in sync with the tests run on the samples, or, in order to make that synchronicity testable would we have to change these so instead of the expected output we show the tests the code is expected to pass (a bit like the Hystrix docs do here: https://github.com/Netflix/Hystrix/wiki/How-To-Use#Synchronous-Execution) -- something I think would make for more awkward and less clear documentation in our case, but maybe it's worth that cost.

Thoughts?

[flatmap13]

I've created a new branch which contains a custom jekyll plugin that can load code samples from GitHub.

This is just a proposal, but if we use this plugin we can add code samples to markdown pages as follows:

{% snippet https://api.github.com/repos/ReactiveX/RxScala/contents/examples/src/main/scala/Hello.scala, scala %}

The plugin loads the samples during the build of the website, so in production the samples are static (as opposed to my previous implementation).

To test this, you should locally checkout the snippets branch, build the website with rake, and then browse to http://localhost:4000/embed-test.html.

[samuelgruetter]

@GeorgiKhomeriki that's very nice, IMHO that's the way to go.
A nice feature to add to this plugin would be to allow markers in the source file which tell what part to include, so that we don't always print all the boilerplate such as imports etc. And if these markers are named, we could even put several snippets in the same file, which would be very handy. For example, RxScalaDemo might look then as follows:

@Test def exampleWithoutReplay() {
  // {BEGIN exampleNoReplay}
  val numbers = Observable.interval(1000 millis).take(6)
  val sharedNumbers = numbers.publish
  sharedNumbers.subscribe(n => println(s"subscriber 1 gets $n"))
  sharedNumbers.connect
  // subscriber 2 misses 0, 1, 2!
  doLater(3500 millis, () => { sharedNumbers.subscribe(n => println(s"subscriber 2 gets $n")) })
  // {END exampleNoReplay}
  waitFor(sharedNumbers)
}

@Test def exampleWithReplay() {
  // {BEGIN exampleReplay}
  val numbers = Observable.interval(1000 millis).take(6)
  val sharedNumbers = numbers.replay
  sharedNumbers.subscribe(n => println(s"subscriber 1 gets $n"))
  sharedNumbers.connect
  // subscriber 2 subscribes later but still gets all numbers
  doLater(3500 millis, () => { sharedNumbers.subscribe(n => println(s"subscriber 2 gets $n")) })
  // {END exampleReplay}
  waitFor(sharedNumbers)
}

And to include it, one would also have to give the identifier of the snippet within the source file. Something like

{% snippet https://api.github.com/repos/ReactiveX/RxScala/contents/examples/src/test/scala/rx/lang/scala/examples/RxScalaDemo.scala, exampleNoReplay, scala %}