PC_Testing_and_CI

Some introductory things in automated ways to improve your code, and to make your The idea is to introduce them and you can try use them if you think they will be helpful to you. The examples are mostly for python but the ideas are more general.

Contents

• Linting
• Testing
• Web Services integration

To follow the examples in this document clone the repo, cd into the directory and run: pip install -r requirements.txt

Linting

A process of running a program that will analyse code for potential errors.. Essentially spell-check for your code.

• Style: Indentation, spaces, comments, doc-strings, variable names.
• Correctness: x is None rather than x == None,
• Complexity: Counts nested if-else statements.
• Unused variables
• Finds TODO's
• Type checking (mypy)

If you code is of a consistent style then it easier to read, understand and edit. For yourself and others.

Many are standalone software but there are many plugins available for your IDE or text editor.

For example flake8 is a good python linter, following the PEP8 style guide with error and complexity checks.

You can run it in this directory to get a list of errors in examples.py

flake8

• You can check your latex documents with chktex. (Who hasn't misplaced a $ or [] before.)

• Or bash scripts with shellcheck

Should be able to find plugins for the text editor you use, and programming language you need. I find having the plugins very useful as I see the errors appear as I am writing the code and don't need to manually rerun the linter.

Can turn off specific errors if you don't want them included. E.g.

• operator spacing
• comment style
• Unused variables
• Found TODO's

See the documentation for error codes, and/or how to disable them.

Maybe overwhelming on large code at first so you can start by disabling all and re-enable a few at a time.

mypy is a static type checker which uses the type-hinting from PEP484.

mypy examples.py --ignore-missing-imports

--ignore-missing-imports is to ignore the complaints about mypy not understanding numpy yet.

I have heard automated linter checks for collaborative projects can help coders self-esteem. Its less personal if the tool tells you to tidy up your code before it can be added than if a person does it.

CI Testing

From wikipedia ...

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only

Not quite suitable for research but we can make use of some of the concepts/tools.

In software testing, test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes.

Different types of testing, with differing/overlapping names. E.g. unit tests, integration test, system test, end-user tests, quality assurance tests...

Here I focus on unit-tests and property-based testing.

####Unit-tests:

• Test the functionality of one single unit of code. E.g. one function per test.
• Can have more then one check in a given test.
• Should be Small and fast. You will them running them
• Should not just be a duplication of the code.

For python:

• Unittest - built in test framework. Tests written as Classes.
• nose
• pytest
  • Scans directory for test[s]/ test_*.py or *_test.py files.
  • Runs functions that begin with test_ or end with _test.
  • Can run Unittest and nose test cases.

Some test examples are in test_examples.py. These run if you type in this directory:

pytest

pytest does magic things with python's assert function which makes it easy to use. If the assert fails it prints a breakdown of why it failed.

Some useful features of pytest shown in the examples are:

• @pytest.mark.xfail(), @pytest.mark.xpass()
  • Allows you to mark tests that are known to be failing, will always fail, or sometimes fail.
  • Still allow test suite to "pass" successfully.
• @pytest.raises(ERROR)
• Test that "ERROR" was raised.
• @pytest.mark.parametrize()
• Run same test with different parameters.
• Removes duplication of code.

See the pytest documentation for much more...

Pytest Fixtures:

The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute.

Ability to pass result of long process into many tests.

• e.g. database access, web-request, load file.
• Reduces test run-time and code duplication.

I have heard software testing compared to the warning lights on a car dashboard. There is the check-engine light which is an ominous all inclusive error that does not identify what the problem is, just shows there is one which requires further investigation. The other light you should be familar with is the fuel gauge. This tests one thing only, the fuel level. You immediately know what problem is and how to resolve it. Your unit tests want to be direct like this, effectively test one thing at a time which will narrowly identify where the error is. Testing a large monolithic with a result return "something in your code is broken" isn't very helpful for quickly correcting your code.

Hypothesis - Property based testing.

Often in science there is not an exact value to test. assert x == 42 We can however test the properties of the values we expect. Type, length, shape, attributes etc.

Good examples are reversible processes:

• Encryption / Decryption
• Coordinate transformation

Hypothesis is a tool allowing you to test the properties of you code. It generates data to pass into test cases, following defined strategies.

Some strategies are.

• int, float, bool
• list, tuple, sample_from

You can also refine the strategies, such as setting min/max size values, inclusion of infinity etc.

It throws in unique cases trying to find corner cases you have not though of to test. e.g. inf, Nan or an empty list.

• Tries 200 (default) different combinations.
• Remembers failed test cases, to test again.

The new release has support for numpy although I have not used it yet.

Testing Figures

A feature I have not used yet but would like too one day is pytest-mpl. This is a matplotlib plugin for pytest, allowing you to test the generation of images. From what I have read the pixel difference between the image generated from the code and a reference "correct" image is calcualted. To pass the sum of the pixel differences nees to be below a certian threshold.

This could allow your to test that your code continues to produce the publication figures for reproducability and consistency.

Coverage

How much run able code was actually ran during the tests? Tells you percentage of coverage and which lines were not run.

pytest --cov=. --cov-report term-missing

It is only helpful metric if you have useful tests.

Should aim to always increase coverage when adding new functionality. Large project possibly won't accept new features unless they have the associated test.

Of course this metric is not vital to the success of your project, but others are more likely to contribute to a well tested and covered project.

Requires the coverage and pytest-cov packages.

Web Services

There are many web-services available that you can link to your github projects.

• Public repo's use these for free while private repo's need to pay.

When ever you push your code to github it will trigger these services to run, and email you any changed result.

Code Review:

Services that perform automated code review checks on your code. I find they complement linting, and each other. The links here are to my spectrum_overload project on each of these sites.

Examples:

• Code Climate
• Quantified code
  • Can submit pull requests on your repo to make suggested changes. e.g. this PR

Continuous Integration Testing:

Services that run your tests.

• Clone repo and install
• run tests e.g. pytest
• after success - do something?
• Email changed build result.

Travis CI but many others with different flavors (windows, phone apps).

• configure with .travs.yml file in project root dir

Coverage:

• coveralls.io
• Displays code coverage statistics, and missing lines.
• Comments coverage change on PR's

Documentation:

Sphinx is a tool that makes it easy to create intelligent and beautiful documentation

• Auto-documentation from module/class/function doc strings.

Read The Docs has become popular place to host the documentation of software.

• Builds the html documentation from your github repo and hosts it.

Pypi

Make your software available from a package manager. Make easy for others to install. e.g. pip install ...

• Easier than you think...
• How to submit a package to PyPi

Badges:

Little tokens to display on your documentation, website. Gives a quick look at the status of your project.

• tests passing
• well covered
• latest version
• etc.

You can make your own badges at shields.io/. 

Much like Pokémon, gotta catch 'em all!

Advanced for dev teams / large projects:

• pre-commit Hooks
  • Lint check, run tests have to pass before commits or pushes.
• Fail tests if linting doesn't pass
• Daily test builds?

Links

• pytest
• coverage
• Code Coverage-- General
• Test and Code Podcast
• Package Bling
• practical software testing tips