Skip to content
Permalink
Browse files

Added criteria for poor and excellent for each question

  • Loading branch information...
Ash Winter
Ash Winter committed Nov 12, 2018
1 parent 9bb1651 commit c7a7a6cc90a3b8faa8183da7d1e551a5e5084ea3
Showing with 26 additions and 28 deletions.
  1. +26 −28 testability-questions.md
@@ -35,9 +35,7 @@ _We have implemented circuit breakers between our services in order to both isol

### Evidence

_We test that Correlation IDs are working properly with BDD tests written in Cucumber. See `tests/correlation-tests` folder e.g. `trace-id.feature`_

_We have a grafana dashboard showing circuit breaker behaviour over the past month, with events detailing how often they have opened, been intentionally opened, automatically recovered or required intervention to close.
_We have a grafana dashboard showing circuit breaker behaviour over the past month, with events detailing how often they have opened, been intentionally opened, automatically recovered or required intervention to close._

### Score

@@ -79,7 +77,7 @@ _4_

> We recognise the importance of our internal and external dependencies. The testability of dependencies effects your testability.
* Poor: _We contact our dependencies when there is something wrong_
* Poor: _We contact our dependencies when there is a problem_
* Excellent: _We collaborate with our dependencies by attending their ceremonies where appropriate and monitoring their uptime_

### Who?
@@ -94,8 +92,8 @@ _4_

> We need the ability to test both scheduled tasks which lifts and shifts or replicates data and other operations which occur on demand.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _We wait to test scheduled tasks in later environments_
* Excellent: _We can trigger both synchronous and asynchronous tasks when needed, close to development_

### Who?

@@ -111,8 +109,8 @@ _4_

> We need the ability to set both data and configuration to recreate the conditions for a specific test, to assist with consistency and reproducability.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _Data and configuration is static, requires a separate team to schedule a change_
* Excellent: _Data and configuration is dynamic and can be set by the team_

### Who?

@@ -124,10 +122,10 @@ _4_

## CONTROLLABILITY: Is each member of the team able to create a disposable test environment?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need all members of the team to be able to build a test environment with a specific version of the application and tear it down afterwards to enable early showcasing and testing.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _All our environments are persistent with a long build performed outside the team_
* Excellent: _An environment can be created and destroyed by each team member in a self service manner_

### Who?

@@ -139,10 +137,10 @@ _4_

## CONTROLLABILITY: If you ask the team to change their codebase do they react positively?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need confidence when changes that add value are required we can change the code or configuration with knowledge of potential impacts.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _We are reluctant to change some or part of the codebase without deep analysis_
* Excellent: _We are confident that we can make isolatable changes which can realise business value while minismising risk_

### Who?

@@ -154,10 +152,10 @@ _4_

## CONTROLLABILITY: Is each member of the team able to run automated unit tests?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need a clear understanding what a unit is in the context of our system and each team member can execute our lowest level test suite on demand.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _We have no unit tests_
* Excellent: _Unit tests are part of our definition of done, run as part of continuous integration and executable by any team member_

### Who?

@@ -171,10 +169,10 @@ _4_

## UNDERSTANDING: Does the team curate a living knowledge base about the system it maintains?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need a store of information that is kept up to date in order for our stakeholders to integrate with the team which is up to date with our system.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _We have a team wiki page which is updated periodically_
* Excellent: _We have a living executable specification describing our system and documentation such as [Swagger](https://swagger.io/) for those who wish to integrate with us_

### Who?

@@ -186,10 +184,10 @@ _4_

## UNDERSTANDING: Does the team know which parts of the codebase are subject to the most change?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need a clear understanding which areas of our codebase are subject to the most change so we can target testing appropriately.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _We rely on intuition alone to describe changes_
* Excellent: _We visualise our source code repositories using tooling such as [Gource](https://gource.io/)_

### Who?

@@ -201,10 +199,10 @@ _4_

## UNDERSTANDING: Does each member of the team have access to the system source control?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need all members of the team to access one of the main collaboration artefacts for a development team, for code and configuration.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Poor: _Only the developers have access to the code_
* Excellent: _Developers, test, operations and product stakeholders can at least view the code_

### Who?

@@ -216,10 +214,10 @@ _4_

## UNDERSTANDING: Does the team have regular contact with the users of the system?

> We need a clear understanding of the people and teams that can help to make the software systems work well.
> We need regular contact with our users, internal or external, with a preference for face to face if possible.
* Poor: _We fix any operator problems after go-live_
* Excellent: _We collaborate with the live service / operations teams from the start of the project_
* Excellent: _We collaborate with our users often, inviting internal users to our ceremonies and with monitoring of journeys to complement direct feedback by external users_

### Who?

0 comments on commit c7a7a6c

Please sign in to comment.
You can’t perform that action at this time.