001-meta-model-of-software-testing.dot

digraph TestingBasics {


  # make graph compound because we want to link to clusters
  graph [compound=true];

  # label text alignment: use backslash l to have left aligned new lines and n for centered new lines

  rankdir=TB;
  newrank="true";

  #bgcolor=transparent;
  
  clusterrank=local

  /*

Testing is underpinned by models.

e.g. requirements, user stories, scenarios, etc.
  
  */
  
  model;
  
  # STEP
  /*
  
We have the 'thing we are comparing with'. Usually a System Under Test.

  */

  comparison_thing [label="System\n Under\n Test"];

  # STEP
  /*

The model and the SUT are inputs to our testing process.

  */
  subgraph cluster_inputs {
    label="Inputs to Testing";
    rank=same;
    model;
    comparison_thing;
  }

  # STEP
  /*

The simplest way to use a model for testing is to compare
the model with the System Under Test.

  */

  compare [shape=square];
  subgraph cluster_evaluation {
      compare;
  }

  # STEP basic compare inputs
  # UNCOMMENT_//
  # NOTE: this line only used in early versions of the diagram
  # later we link to the cluster, not the node 
// {model comparison_thing} -> compare;


  # STEP

  /*
This generates output results from the comparison.
  */

  subgraph cluster_comparison_outputs {
    label="Comparison Output";
    rank=same
    node [rankdir=LR]
    // allow it to render without any visible nodes
    output [ label="" style = invis ];
  }

  compare [shape=square, label="Comparison"];
  compare -> output [label="Comparisons" lhead=cluster_comparison_outputs];

  # STEP
  /*
Evidential Observation Output can be used for
negative or positive confirmation.
  */
  

  /*
Positive confirmation is often viewed as a 'Test Pass'.
And basically means that 'when we did something we observed
that the system behaved the way that we expect'.

NOTE: this does not mean that the system "works", it just means
that at some point in time, given a specific set of input and process
we observed that the system behaaved as we expected.
  */

  subgraph cluster_comparison_outputs {
    label="Comparison Output";
    rank=same
    node [rankdir=LR]
    // allow it to render without any visible nodes
    output [ label="" style = invis fixedsize=true width=0 height=0];
    positive_confirmation;
  }

  # STEP
  positive_confirmation [
    label="Positive\n Confirmation",
    fillcolor=green,
    style=filled
    ];

  # STEP
  /*
Negative Confirmation is often called a 'Test Fail'.

This means we observed something we did not expect.
  */

  subgraph cluster_comparison_outputs {
    label="Comparison Output";
    rank=same
    node [rankdir=LR]
    positive_confirmation;
    // allow it to render without any visible nodes
    output [ label="" style = invis fixedsize=true width=1 height=0];
    negative_confirmation;
  }

  # STEP
  negative_confirmation [
    label ="Negative\n Confirmation",
    fillcolor=red,
    style=filled
  ]

  # STEP

  /*
These positive and negative reports are a visible output
from the Testing process.
  */


  /*
So we have to investigate what failed during our comparison process.
  */

  negative_confirmation -> investigation;
  investigation [
    label="Investigate",
    shape=square
  ]

  /*
This might mean:

- a problem with our model
- a problem with the System Under Test
- a problem with our comparison process

Either way it is something we have to investigate.
  */

  # STEP
  investigation -> model [
    label="- Model does not \l  match reality \l- Change the model.\l"
  ];

  # STEP
  investigation -> compare [
    label="- Process Issue.\l- We did not use\l  the system properly.\l- Re-evaluate.\l"
    lhead=cluster_evaluation
  ];

  # STEP immediate bug fix
  # UNCOMMENT_//
  // investigation -> comparison_thing [
  //   label="System does not match\n what we require.\n Found a Bug.\n Fix the System."
  // ];

  # DIRECTIVE
  # DISABLE_STEP immediate bug fix
  /*
    System Under Test changes might require a defect process.
    Rather than a direct system fix.
  */

# NOTE: remove earlier investigation -> comparison_thing approx line 164-166

  # STEP
  investigation -> raise_bug [
    label="- System does not match\n what we require."
  ];
  raise_bug [label="Raise Bug"];
  raise_bug -> fix_bug;
  fix_bug [label="Fix Bug"];
  fix_bug -> comparison_thing [label="- Fix the System"];


  subgraph cluster_investigate {
    investigation;

        subgraph cluster_defect_process {
          rank=same
          node [rankdir=LR]
          raise_bug;
          fix_bug;
        }
  }

  # STEP
  /*
A comparison process from model to 'thing' isn't the only process that
Software Testing involves, it's just one of the most obvious ways we have
of evaluating the Software.

And Software Testing is a process of evaluating the software.

  */
  compare [style=filled, fillcolor=yellow];
  subgraph cluster_evaluation {
      label="Evaluation";
      compare;
  }

  # STEP
  /*

We can directly compare Requirements with the Software for high level surface
positive or negative confirmation that a requirement has been implemented.

e.g. "A User must be able to login with their username and password." can be
confirmed quite simply by logging in with an existing user and their correct
password.

But during the Software Testing process we also try to expand our model beyond
Requirements.

We also explore our model in conjunction with the system to learn more about
our model and as a consequence, the system itself.
  
  */

  compare [style=filled, fillcolor=white];
  subgraph cluster_evaluation {
      rank=same
      node [rankdir=LR]
      label="Evaluation";
      compare;
      modelling;
      exploration;
  }

  # STEP
  /*

Software Testing has to explore our model to identify:

- risks
- ambiguity
- gaps in the models and implementation
- etc.

This raises questions which we have to investigate and discuss.

The ongoing process of modelling is how we generate our ideas
for testing and how we find new ways to explore the software.

  */


  modelling [label="Modeling", shape=square, style=filled, fillcolor=yellow];
  modelling -> model [
    label="Expand Model:\l\l- risks\l- ambiguity\l- gaps\l- questions\l - etc.\l"
    ltail=cluster_evaluation;
  ];

  # STEP
  /*
The model is both an input to testing, and an output from Testing.

Parts of the model will be stored in the Tester's head as a mental model.

Other parts will be visible in the form of diagrams, reports,
lists of test ideas, etc.
  */
  model [label="Model" style=filled fillcolor=yellow];

  # STEP
  /*

Software Testing then has to explore the model in conjunction
with the system to identify if any of these manifest as issues
or potentially require reworking the implementation.

  */

  modelling [label="Modeling", shape=square, style=filled, fillcolor=white];
  exploration [label="Exploring", shape=square, style=filled, fillcolor=yellow];

  # DIRECTIVE
  #   DISABLE_STEP basic compare inputs
  // NOTE: remove line ~= 57 i.e. {model comparison_thing} -> compare;

  # STEP
  {model comparison_thing} -> compare [lhead=cluster_evaluation];


    /*
One example of ambiguity arises from requirements being naturally incomplete because
of assumptions in understanding.

The requirement for logging into the system does not describe:

- what happens if the user is already logged in?
- what message is displayed if the user gets the password wrong?
- what message should be displayed if the user does not exist?

Part of the Evaluation process that we call Software Testing is about
expanding our model so that we can explore the System more thoroughly
and provide more information.

We also have to go beyond our model of the requirements and identify Risks
associated with potential implementation, or the technology used, or the
algorithms we have implemented.

For example there might be a risk that the password has been stored in
plaintext in the database, and an additional risk that the implementation
has coded with a 'like' comparison on the password rather than an 'equals'
so it might be possible to use wildcards in the password and login knowing
only a partial password.

This would also lead to risks about the security of the application and SQL
Injection.

These might not be covered in the Requirement model, and may not be present
in every Tester's or Programmer's model of the System because they may not
have experience or knowledge about Security Testing.

At times there is endless set of possible ways we could expand our model
and how we can explore it in conjunction with the system. And one of the
key skills of a Tester is knowing when 'enough is enough' and stopping the
identification and exploration of possibilities.
  */
  
  # STEP
  /*
## Exploring

The act of exploring a model, or using the model as a basis for exploring the
system, is the main way that we test softare.

Some of the outputs from this process are captured as part of the model itself.

Other outputs vary depending on the process used but include artifacts like:

- execution or exploration logs
- coverage reports
- questions
- decisions
- ideas for new exploration approaches
- issues
- issue investigations

  */


    subgraph cluster_exploration_outputs {
      label="Exploration Artifacts";
      rank=same
      node [rankdir=LR]
      // allow it to render without any visible nodes
      exploration_artifacts [
        label= "- execution or exploration logs\l
- coverage reports\l
- questions\l
- decisions\l
- ideas for new exploration approaches\l
- issues\l
- issue investigations\l"
    shape=square
      ];
    }

  # STEP
  exploration -> exploration_artifacts [label="Exploration" lhead=cluster_exploration_outputs];

  # STEP
  /*
Exploration is usually pursued in small chunks of time to make it easier to document
and respond to the information gained during the exploration to make decisions
about what to test or investigate next.
  */
  exploration -> investigation [label="Investigate\n Findings"];

  # STEP
//   # Fin - reset any temporary highlights
  exploration [label="Exploring", shape=square, style=filled, fillcolor=white];
  
  # END
}