Test Strategy
These are tests that verify the behaviour of the service and are written using Spec Flow. They are the prime tests used to verify that the system operates as expected. Given that the behaviour is exposed as a web api, then the behavioural tests operate by invoking the controller.
Clearly the controller needs to work against the external environment, however we want the behavioural tests to run in the build so that we do not deploy broken behaviour. This is achieved by using the Adapter pattern. We implement two adapters using the same interface, one emulates the environment and one is the real adapter that actually interacts with the real environment. The two adapters are tested by a single set of tests to ensure that they both implement the same contract, the emulator tests run as part of the unit test suite and the real adapter tests run as the integration tests.
Behavioural tests therefore operate in two modes. In the build they are set up to use the emulated adapters. In the integration environment tests they are set up to use the real adapters.
Behavioural tests can modify the state of the system and are therefore not used in the production environment.
These test the real adapters and can only run in the integration environment. The intent is to verify the contract between two components/services only. It is not the intent of these tests to verify complete end to end functionality, as that is a different problem, as described below.
End to end tests are used to test complete scenarios, generally covering multiple end-user actions. They tend to be complex and brittle. Given all the other test types we have, end to end tests would not pick up many defects, but provide a valuable level of assurance that the system works correctly. Therefore we will only have one or two end to end tests at most. As they change state, they are only run in the integration environment.
These tests verify that the service is actually hosted correctly and ready to use (including any external dependencies). The service must expose an endpoint that allows its health to be checked and can be used as part of an Application Insights availability test.
The smoke tests do not change state and are run in every Release Management environment, including Production.
Unit tests are tests that can run in an environment-independent manner, in the sense that they are entirely self-contained and do not require any kind of environment setup. They can run on developer machines and build servers. Unit tests are run in every build. They could be run in the release environments too, but there is not much point in doing so.
We only use unit tests where the behavioural tests do not provide coverage. Typically this would be for "library" code. We also use them to test the emulated adapters, using the same test suite as the integration tests that test the real adapters.