Document Test Type Definitions

[kennylavender]

There is still some uncertainty around which category a test falls in issues/prs.

There doesn't seem to globally accepted test definitions in software development, so I think it would help if we document our test type definitions.

Here are a couple links to definitions that don't completely agree. Just providing for information and guidance, not suggesting we follow one or the other. We had originally agreed to follow Eric's but we never documented it and we could add more clarification for ourselves.

• Eric: https://www.sitepoint.com/javascript-testing-unit-functional-integration/
• Robert C. Martin: http://blog.cleancoder.com/uncle-bob/2017/05/05/TestDefinitions.html

[lautarodragan]

Agreed 100%. Great initiative 🚀

I find Eric's, RCM's and every other definition I've read vague at best.

Like you said, we can use their (and other's) definitions for guidance but we'll have to define our own.

Personally I like this approach:

Functional Tests

Functional tests treat the application being tested as a black box. They are not aware of whether inside the application there are controllers, DAOs or whatever. They do have knowledge of the external dependencies of the application, such as MySQL or MongoDB, and are free to mock them or not, but they can only "inject" these mocks via the normal usage of the application (passing the URL via environment variables for example).

This seems similar to what RCM defines as System Tests. System feels less confusing to me than functional, though.

I've mostly seen automated UI tests get called functional tests, too, even though they are completely different. All UI tests are rather intrusive and most times don't treat the application as a black box.

For example, GET /works is what I call a functional test.

Integration Tests

Regarding integration tests, I agree with RCM's vision, although I find no reason why only the architect / tech lead should write them.

To me, integration test is a catch-all term referring to any test that is in the middle of a functional and unit test. That is, they don't treat the application as a black box, they can be aware of specific functions, classes or files, but they are broader in scope then unit tests.

For example, BlockInfoDAO.insertBlockInfo is what I call an integration test.

Unit Tests

In the case of unit tests, from RCM's definition, what he calls micro-test is what I call unit-test. Under most definitions, unit tests means literally anything. Everything can be considered a unit.

In my mind, a test is a unit test if it tests a pure function or composition of pure functions.

Maybe pure test would be a more intuitive name under that description?

Notes

Unfortunately, too many programmers think that the rules for unit tests are different – that they can be “junk code” written using any ad hoc style that they find convenient.

Couldn't agree more. In every project I've worked, as the tests grew larger and larger, the also grew more disorganized and copy-pasted.

[krobi64]

All UI tests are rather intrusive and most times don't treat the application as a black box.

How so? UI tests approach an application the way the end user does, who definitely treats the application as a black box. Yes, they are fragile because of how automating interaction with the UI works today, but if you put a human as the UI testing engine, the application is a black box.

From Robert C Martin wrt integration AND system tests:

These are plumbing tests. They are not business rule tests. Their purpose is to confirm that the sub-assembly (or entire system) has been integrated and configured correctly.

Those are tests that ensure that containers, applications, etc. are simply able to communicate with each other correctly. Can Frost-API reach poet-node, can poet-node reach rabbitmq. That does not appear to be our use of functional or integration. And that is why he thinks the Tech Lead or architect should write them: to ensure the system is communicating among the various sub-parts as designed.

That the View Module displays a claim uploaded to the API module and eventually displays the block Info associated with that claim appears to fall under Acceptance Testing.

[krobi64]

oops

[geoffturk]

Please review our current testing process doc and suggest changes/additions:

https://github.com/poetapp/documentation/blob/master/process/testing.md

[kennylavender]

Oops, guess we did document it, completely forgot about that.

[geoffturk]

Yeah, well have a read and let me know if you would change anything we wrote there. I put that together with Warren, but evidently did not bring enough attention to it! 😃