Skip to content

Commit

Permalink
updated readme with all the stuff from compendiumde.co.uk
Browse files Browse the repository at this point in the history
  • Loading branch information
@eviltester
eviltester committed Nov 23, 2012
1 parent b18f903 commit a367c6d
Showing 1 changed file with 232 additions and 5 deletions.
237 changes: 232 additions & 5 deletions README.md
View file
Expand Up @@ -105,9 +105,11 @@ I hope it helps your testing.
Examples
--------

#### Pairs1 - a general flow graph with basic selections
This section illustrates the examples in the downloadable archive. As you can see the simple algorithm for loop detection used by the script generates a lot of paths for the nested loop examples.

#### [input\pairs1.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/pairs1.txt)
### Pairs1 - a general flow graph with basic selections

##### [input\pairs1.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/pairs1.txt)
```
- 0
0 1
Expand All @@ -121,15 +123,240 @@ Examples
7 8
7 9
```
#### Model
##### Model

![pairs1.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/pairs1.gif)

#### [output\outputpairs1.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/outputpairs1.txt)
##### [output\outputpairs1.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/outputpairs1.txt)
```
- 0 1 2
- 0 1 3 9 4
- 0 1 3 5 6
- 0 1 3 5 7 8
- 0 1 3 5 7 9 4
```
```

### Pairs2 - a general flow graph with basic selections

#### [input\pairs2.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/pairs2.txt)
```
- 0
0 1
1 2
1 3
2 4
3 8
3 13
8 9
8 10
9 4
10 4
13 4
4 5
4 6
5 14
6 14
14 7
```
##### Model

![pairs2.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/pairs2.gif)

##### [output\outputpairs2.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/outputpairs1.txt)
```
- 0 1 2 4 5 14 7
- 0 1 2 4 6 14 7
- 0 1 3 8 9 4 5 14 7
- 0 1 3 8 9 4 6 14 7
- 0 1 3 8 10 4 5 14 7
- 0 1 3 8 10 4 6 14 7
- 0 1 3 13 4 5 14 7
- 0 1 3 13 4 6 14 7
```

### Loop - a simple loop

As you can see the loop is traversed twice by the script's algorithm.

#### [input\loop.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/loop.txt)
```
- 1
1 2
2 3
2 5
3 4
5 7
6 5
7 4
7 6
```
##### Model

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/loop.gif)

##### [output\outputloop.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/outputloop.txt)
```
- 1 2 3 4
- 1 2 5 7 4
- 1 2 5 7 6 5 7 4
- 1 2 5 7 6 5 7 6 5 7 4
```

### Nested Loop - a simple nested loop

#### [input\nestedloop.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/nestedloop.txt)
```
- 1
1 2
1 3
3 4
4 5
5 7
7 4
5 6
6 8
8 3
6 9
2 9
```
##### Model

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/nested.gif)

##### [output\outputnest.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/outputnest.txt)
```
- 1 2 9
- 1 3 4 5 7 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 7 4 5 7 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 7 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 7 4 5 7 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 9
- 1 3 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 7 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 7 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 7 4 5 7 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 7 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 6 8 3 4 5 6 9
- 1 3 4 5 6 8 3 4 5 7 4 5 6 9
- 1 3 4 5 6 8 3 4 5 6 9
- 1 3 4 5 6 9
```

### 3 Nested Loop (1) - a 3 way nested loop

#### [input\3nested.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/3nested.txt)
```
- 1
1 2
1 3
3 4
4 3
4 5
5 3
5 6
6 3
6 10
```
##### Model

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/3nested1.gif)

##### [output\output3nest.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/output3nest.txt)
```
138 paths identified,
please see the file in the archive
```

### 3 Nested Loop (2) - a 3 way nested loop with additional branch

This highlights the effects of combinatorial explosion in testing.

#### [input\3nested2.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/input/3nested2.txt)
```
- 1
1 2
1 3
3 4
4 3
4 5
4 7
7 6
5 3
5 6
6 3
6 10
```
##### Model

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/3nested2.gif)

##### [output\output3nest2.txt](https://github.com/eviltester/perlPathAnalysis/blob/master/output/output3nest2.txt)
```
2,971 paths identified,
please see the file in the archive
```

Critique
--------

### A more detailed analysis of the algorithm's limitations

The script as written is a naive tester. He very thoroughly addresses link coverage and that is why he gets so many paths, but the problem is that there is nothing in the presented result set that suggests priorities; every path is presented as an equal.

This may or may not be true, but from the information contained in the graph it isn't really applicable to present such a detailed analysis of the paths in the model as, in the case of example "3 nested loop (2)" the important paths can get lost among the morass of loop coverage.

It would be better, and I think that most testers will do this either consciously or unconsciously, to present the following details for link coverage:

* Each sequential path should be presented and covered.
* Each loop path should be presented as a meta path and not unrolled.

Loop is very simple:

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/loop.gif)

Our sequential paths in the above model are:

* 1 2 3 4
* 1 2 5 7 4

Our meta path is:

* 1 2 [5 7 6]* 5 7 4

In order to read an understand the meta path we have to know that []* represents a loop construct which occurs from 1 to Many times. The exit condition for the loop may well be documented in the graph as part of the weighting for a predicate node link but the script does not support this.

The script has no concept of a meta path, indeed the script has a variety of hack constructs to try and identify loops in order to prevent the processing time approaching infinity.

The script, although it is unaware of it, applies the "Once" and the "Twice" strategies to the meta path.

In the above example the "Once" strategy will achieve link coverage and the "Twice" strategy is overkill. It may be important but not in terms of the information in the model.

The script presents the meta path as two unrolled sequential paths. This is valid in terms of writing scripts and associating scripts with tests where the script must be executed in the same way each time so non-deterministic loops will not be represented, and indeed loops will often be unrolled.

This is most seriously demonstrated by example "3 nested loop (2)":

![loop.gif](https://raw.github.com/eviltester/perlPathAnalysis/master/modelgifs/3nested2.gif)

The script has identified 2,971 paths. That's great, that tells us that although the model is small, it hides a great deal of complexity and correspondingly may have been tricky to implement and may be a den of defects.

The problem is that the script has presented us with too much information, what we actually want to see, and what will be represented in the tester's mental model for link coverage, is in fact 3 paths:

* 1 2
* 1 [[3 4]* 7 6]* 10
* 1 [[[3 4]* 5]* 6]* 10

This representation is enough to cover the links. It should be noted that none of the links that cause a loop to repeat will be covered unless the loops are traversed twice as the loops are post test and not pre test. i.e. do ... while, rather than while...do.

That is pretty much all I wanted to say on this subject to highlight the limitations of the script and to begin to highlight some of the processes that occur in the testers mind. But, and I think the last model presents it best, testing is not simple. The 3 paths above provide link coverage but they are not completely representative of the tests that a tester would construct from that model. The path 1 3 4 5 3 4 7 6 10 is not represented by the meta paths above. The meta paths were constructed on the basis of link coverage and link coverage does not represent all that a tester does.

A slightly malformed meta representation of the path can be formed by combining the two meta paths above into 1 [3 4]* [(7 6]*)|(5]*6]*)] 10. It is malformed because I haven't equalised the brackets.

A far better representation is 1 [3 [ 2 ( [1 3 4 ]*1 7 )|( [2 [ 1 3 4 ]*1 5 ]*2)] 6 ] *3 10. This does introduce a new representation in terms of the superscripted loop numbers but hopefully it makes sense. (I actually don't know why I thought the first representation was appropriate but I have left it in place as it highlights the ease of defect introduction into any human endeavor.)

Postscript: The script actually does identify 1 3 4 5 3 4 7 6 10, so it does more than link coverage and is often too enthusiastic with it's loop handling.

Related information is contained in our essay on [Structural Path Analysis](http://www.compendiumdev.co.uk/essays/pathexploration/pathanalysis.htm).

0 comments on commit a367c6d

Please sign in to comment.