Skip to content

Add testing to repository#7

Merged
lispandfound merged 13 commits intomainfrom
tests
Aug 28, 2024
Merged

Add testing to repository#7
lispandfound merged 13 commits intomainfrom
tests

Conversation

@lispandfound
Copy link
Copy Markdown
Contributor

This pull request adds extensive test cases to the source modelling repository. With this pull request merged, every current module will be tested. This testing effort is a part of my overall workflow testing strategy, and part of the type-5 validation (and general code validation efforts).

You can ignore the files in the srfs and fsps directory, they just contain test files.

Testing Strategy

We currently have a very ad-hoc testing strategy. We test some code half-heartedly, and only where the testing is easy to do. This is ok for low-risk code, but ideally anything involved in Cybershake should be thoroughly tested since the cost of bugs in a Cybershake run can be really high. I want to change this as much as is feasible (recognising that some repositories are too hard to test properly). Broadly, I have identified three testing strategies that I think can enable high quality testing of the different kinds of code we write:

  1. Hypothesis Testing. This is a new kind of testing I have introduced to the source modelling repo that has wide applications for code that is especially Mathematical in nature, or where we want to test properties not process. The Hypothesis website does a really great job of extolling the virtues of hypothesis testing. The key features of this kind of testing are:
    • Automated test case generation. We do not have to write test cases manually, and instead rely on intelligent searching of the input space to test a function.
    • Automated test case simplification. If a test fails, Hypothesis will automatically simplify the test case to the simplest possible input that produces the same failure. It can even tell you if certain values of input are irrelevant (for example, that a function that incorrectly computes rrup fails regardless of hypocentre depth).
      This kind of testing is suitable where we can easily validate properties that must hold, and where generation of test cases is simple. For example, the qcore.bounding_box module could have hypothesis testing to check that (a) the bounding box contains the set of points it is supposed to bound, and (b) it is smaller than a random sampling of other boxes that contain the points.
  2. Unit Testing. This is familiar and used in a couple of places, but I think where we cannot hypothesis test and we can verify outputs are correct, we should unit test. For example, the srf and fsp parsers are not amenable to hypothesis testing because it's really hard to generate valid files.
    However we can unit test to check that certain files parse correctly/fail, and verify their properties.
  3. Regression/Integration Testing. I think this is useful to involve where we cannot know the right answer for a test, or to test across design concerns. I have outlined a strategy to implement regression and integration tests for the next workflow iteration in an issue on the workflow repo.

The Testing GitHub Action

The GitHub action included in this pull request checks that all the tests pass and that every module has 95% test coverage. Going forward, a pull request to the source modelling repo cannot be merged without the code coverage for each module staying above 95% and without passing test cases.

dbottom: float,
length: float,
width: float,
projected_width: float,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docstring should also be updated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the long-term, a better fix would be to take dip + width as an argument and then compute projected width in the function. But considering that you do make use of this code I will refrain from changing the API immediately.

Copy link
Copy Markdown
Contributor

@AndrewRidden-Harper AndrewRidden-Harper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@lispandfound lispandfound merged commit 21dd4b5 into main Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants