Introduce and use pytest markers

[bjlittle]

Given pull-request #5969 it seems that there might be benefit in allowing developers to target testing on specific areas of iris, as generally it takes while (>4mins) to run the entire test suite.

Example markers might include something like:

• graphics
• integration
• unit
• regrid
• merge
• concatenate
• load
  • load-pp
  • load-ff
  • load-nc
• save
  • save-pp
  • save-ff
  • save-nc

etc

We could be as granular and provide as much marker coverage of the tests as we feel appropriate.

Thoughts?

Ping @bouweandela

[bjlittle]

... Of course this would encourage wider pytest native adoption throughout iris 👍

[trexfeathers]

Ping @ESadek-MO

[rcomer]

I think you can get a long way by just pointing pytest at a particular file or directory

pytest lib/iris/tests/integration
pytest lib/iris/tests/unit/plot

Markers can get forgotten SciTools/cartopy#2369 🤦, so you need a process/infrastructure to make sure they always get updated appropriately. I think time working on this might be better spent targeting the longest running tests to see if we can speed them up, e.g. #5973.

[bouweandela]

I agree with this. Learning a new codebase is challenging, and understanding which markers to use to test what code may be as challenging as understanding a well-organized directory structure.

[bjlittle]

Yup, agreed.

I use markers sparingly on geovista, but I find them very useful from the perspective of not needing to know the file or directory structure ... particularly if the tests for a marker are peppered across several test files/directories.

For example, we have pytest -m "not image" on geovista to run all tests apart from expensive image rendering integration tests, and on the flip-side pytest -m image to only run those tests.

That said, --strict-markers (in the config) gives a wee bit of sanity checking with regards to marker use.

For iris I would imagine markers would only be use sparingly too, if at all, but we would need a legitimate reason with definite benefits to introduce them.

I just wanted to test the water on peoples opinions. I definitely don't want to introduce infrastructure for the sake of it, which then eventually turns into a maintenance burden/headache.