Behat doc testing

[dantleech]

I would really like to test out my idea of functional testing documentation with Sulu. (https://github.com/dantleech/sphinx-behat)

A prerequisite for this would be to make the documentations in RestructuredText format and use Sphinx (as with the symfony, doctrine and CMF docs).

A start might be to implement the installation instructions.

Functional testing the documentation would provide an additional smoke test and ensure that the documentation is always "working".

[webmozart]

Interesting idea :) A big issue that I see here is that Behat tests are, while very precise, not necessarily easy to read. They serve as specification, so that's fine. Documentation, on the other hand, serves as learning resource. Readability and understandability should be key there. I don't think we can achieve that with Behat tests.

Second, Behat tests require changes to the FeatureContext every time a sentence is reworded. This would slow down work on the documentation considerably.

Apart from this issue, I think that starting to write Behat tests for Sulu is a great idea.

[dantleech]

Well, I think (I had to check) the idea of the extension is to embed the behat sentences in the RST file, but they would not be rendered in the documentation. So they are essentially static and invisible.

[dantleech]

btw. we are already using Behat to test Sulu Standard, but the tests are quite flaky IIRC due to Husky/JS.

[danrot]

@dantleech Would not even say that this is caused by Husky or JS, I am sure quite often we are not waiting for the right events to happen. If we write at some points only And I wait for a second it doesn't surprise me that the tests are flaky.

And apart from that it might not even be necessary to test this kind of stuff using behat. We could simply take the commands written, execute all of them, and see if we have a working sulu installation at the end. However, this is probably not as easy as it sounds 😕

[dantleech]

On Thu, Mar 10, 2016 at 08:28:58AM -0800, Daniel Rotter wrote:

[1]@dantleech Would not even say that this is caused by Husky or JS, I am
sure quite often we are not waiting for the right events to happen. If we
write at some points only And I wait for a second it doesn't surprise me
that the tests are flaky.

The point is there are no events we can listen to to say "OK" this is
loaded, its very hit-and-miss, and then it is flaky still, for whatever
reason (at least the last time I saw it, which tbh was over half a year
ago).

And apart from that it might not even be necessary to test this kind of
stuff using behat. We could simply take the commands written, execute all
of them, and see if we have a working sulu installation at the end.
However, this is probably not as easy as it sounds [2]:confused:

Behat was just a tool that could be used to execute the generated file.
There are other ways to do it, but all would involve either annotating
the markup or adding a fair amound of regexes in different file.

This is also not about executing (and testing) commands, but also code
snippets and configuration.