[tests] Idea: Use a runtime test library for tests

[jeffmo]

@alexeygolev showed an interesting way of writing tests for the ramda libdef over in this pull request: #95

Basically the idea is that we could have the test runner (a) typecheck the test code (contains type asserts just like we have today) and then (b) actually run the tests (using the actual npm library) to make sure they pass.

This could help bridge the uncertainty gap between a libdef and it's actual implementation.

[ryyppy]

So this is actually just running the flow-checking mechanism and then a [generic-test-framework] right after? looks nice, although I would definitely opt for tape instead of mocha, since it's much better isolated from the global environment...

[jeffmo]

So this is actually just running the flow-checking mechanism and then a [generic-test-framework] right after?

@ryyppy yea, that's right. The idea is to both confirm the validated types and give a means of confirming that the libdef matches the implementation as well.

[ryyppy]

I just made another thought about that... will this scale? When I imagine to run all tests and we need to install all npm dependencies for all libdefs just to run the tests, we will have first of all a huge install process and other than that a huge disk space waste just for developing some simple declarations...

Wouldn't it be better to tell lib-maintainers who commit stuff here to integrate the flow-tests process in their lib-tests themselves? Maybe give them some best-practise guidance for adding libdef testing via npm test or so?

[jeffmo]

That's a good point with regard to the need to run all tests. We currently run all tests in travis on pull requests, but we really should only run tests for things that have changed. Once we're doing that, I think the speed will be manageable.

That said, running all tests is still useful sometimes (and is already slow), so perhaps the default behavior here should be that runtime tests only execute when a special flag is passed to run-tests? That way you can just do static-check types independent of the runtime tests.

Wouldn't it be better to tell lib-maintainers who commit stuff here to integrate the flow-tests process in their lib-tests themselves?

This is generally not a bad idea, but I don't know how practical it is on a broad scale. In order to verify that a libdef matches the implementation you really do need a statically-typed consumer of the library (like a statically-typed test). While I would always encourage lib maintainers to write tests, it's also common to write tests in ways that are hard (if not impossible) to statically verify. Meta-programming is pretty common and useful in testing (think about dynamic test harnesses and data-dependent conditionals, etc).

Overall I think you might've convinced me that we should at least make these kinds of runtime tests optional since they could pose a lot more burden for writing a simple test than is necessary in some cases. Perhaps they're only useful when someone as a reason to be extra thorough with their libdefs.