2. Software Testing.md

2. Software Testing

Software Testing is an investigation conducted to verify if the software works as intended, identifying existing errors, and possibly assess the quality of the software.

Testing

• Testing is an essential part of program development

  • testing is ongoing, shouldn't just do it at the end
  • begins before you start coding
  • continues while you start coding
  • test suites - test the expected behaviour of program

• Testing is NOT debugging, you can't debug without testing. Debugging is the process that identifies the causes of known errors to solve them. Testing occurs both before and after debugging

• Testing cannot guarantee correctness, but it can prove wrongness

• Testing is hard

  • no general formula
  • psychological barrier - don't want to find out that your program is wrong

• Ideally, developer and tester should be different people

Human testing:

• humans look over the code and find flaws (reading through your code)
• code inspection, walkthrough (explaining what happens step by step)

(Automated) Machine testing:

• run program on selected input, check against specification
• can't check everything, not possible to test every possible inputs (it's infinite sometimes), choose test cases carefully and effectively
• black/white/grey box testing - no/full/some knowledge of program implementation

Types of Software Tests

Unit tests are conducted at the lowest level, testing only one specific module/unit of the software. The goal of unit tests is to verify if the unit of code works as intended and identify any existing errors (bugs).

Integration Tests are conducted at a level above the unit tests and aim to verify if the different modules/units of the software work correctly together. These tests should perform operations that involve multiple modules or classes and check if they work as intended.

Functional Tests (or System Tests) focus on the business requirements of the application. They can be conducted by executing the application with a specific set of inputs and verifying if the application produces the correct outputs. We are not concerned with how the results are being produced (which should have already been tested in the unit and integration tests), only if they are correct.

Acceptance Tests are usually part of a formal process in which the client must verify that the produced software meets all the requirements, so it can be accepted. Client acceptance is generally a condition to move to the next phase of a project.

Sometimes, acceptance tests may be detailed into phases such as alpha testing (done at the end of the development, by a subset of users, but in the controlled environment of the developer) and beta testing (done at the end of the development after the alpha tests, by a subset of real users, in the user's environment).

Regression Tests are conducted after any modification in the software, to ensure that it continues working as intended, i.e., that the modification did not introduce new errors. Ideally, regression tests should be automated, so that they can quickly be executed after each code modification. One possible way to conduct regression tests is to create automated unit, integration, and functional tests, so they can be executed again after each modification.

• Make sure new changes to a program don't break old test cases
• test suites, testing scripts
• always add; never subtract

Performance Tests are designed to verify if the run-time performance of the system will be adequate. Some systems may work well when tested by the developer in a single-user environment but may have performance issues when tested, for example, with multiple concurrent users or with large data sets.

• is this program efficient enough?

Accessibility, Usability, and User Experience Tests The goal of these tests is to verify the quality of the experience of the user interacting with the system. Software may do what it is supposed to do (i.e., it passes all the tests above), but it may be for example hard to use or learn, or may not support users with disabilities.

White/Black Box Tests

Start with black box testing, and supplement with white box testing.

Black box testing:

• tests are created based only on the requirement specifications, without any knowledge of the internal structure of the program (i.e., without access to the source code).
• commonly used for functional and acceptance tests

White box testing:

• tests are created with knowledge of the internal structure of a program.

• generally used to verify the internal structures or workings of a program

• unit tests commonly used white-box testing to verify that each function or method is working correctly

Grey box testing:

• a mix of white and black box testing.

• test focusing on the expected outcomes, having knowledge of the code

DO NOT TEST - invalid input, unless behaviour for invalid has been prescribed

• if the input is invalid (and the behaviour for that is unspecified) then the output is unspecified meaning there is no correct input. Any such test case is meaningless.

Creating black-box tests

Check the specification and try to identify all the different types of inputs that the program should accept.

Then, identify what should be the correct output to each set of inputs according to the specification.

Useful Guidelines:

• Various classes of inputs, e.g. numeric ranges, positive numbers vs. negative numbers
• don't need to test invalid input (as is not in the specification)
• boundaries of valid ranges (edge cases), e.g. minimum and maximum values
• multiple simultaneous boundaries (corner cases) (two edge cases in same test case)
• Various classes of inputs, e.g. numeric ranges, positive numbers vs. negative numbers
• don't need to test invalid input
• boundaries of valid ranges (edge cases)
• multiple simultaneous boundaries (corner cases) (two edge cases in same test case)
• many good testing comes from intuition/experience - guess at likely error
• extreme cases (within reason) (e.g. very large numbers)
• many good testing comes from intuition/experience - guess at likely errors

Creating white-box tests

Examine the code and create enough test cases to ensure that all parts work as intended

• execute all logical paths through the program
• make sure every function runs