Future Testing Requirement Details

This page fleshes out requirement details for testing Momentum model components.

Compute-intensive testing on a cloud platform

Rationale

Many tests of simulation models require significant computational resources and run over a longer period of (wall-clock) time, making it unpractical or even impossible to integrate them in GitHub Actions or similar CI services. The public cloud is a possible choice for running such tests in an automated fashion.

As a shared system, the cloud platform should typically run tests that are of general interest and do not require private or sensitive configurations or data.

Advantages

• A given test can run on the most appropriate set of resources to satisfy its needs around processor type and capability, memory requirements, and storage system requirements.
• A large variety of architectures and compilers can be used for testing to ensure portability and improve overall test coverage.
• Developers from across the Momentum partnership can be given access, which is typically much more difficult to arrange on in-house resources at partner sites.
• Test outcomes can be made visible to developers much more easily, who can take action and gather information required to debug a problem.
• Cloud services come with sophisticated automation tools and services, which is ideal for integrating test systems with the GitHub repository.
• Developers from across the partnership can use the cloud service to access a wide range of tests, while they have to rely on their site support to enable running these tests on in-house systems otherwise.
• Efforts for developing and maintaining tests and test platforms can be shared across the partnership.

Disadvantages

• Cloud resources can be expensive, depending on the type and level of utilisation.
• Using cloud services can add a lot of complexity around security, automation, vendor-specific constraints, etc.

Requirement Details

• The cloud services should be accessible by developers from across the Momentum partnership.
• The cloud service needs to integrate with GitHub so that tests can be triggered automatically.

Compute-intensive testing on an on-premise platform

Rationale

Many partner sites are interested in running a subset of tests on their own platform to ensure ongoing support and code portability. They may also want to add highly site-specific tests that are either not of interest to other sites, or that rely on private data or sensitive configurations. In these cases, partners need to be able to run tests on their own on-premise platforms.

Advantages

• Partner platforms can be tested directly.
• Easier to restrict access to satisfy privacy, security, and regulatory requirements for data and configurations.
• Easier to integrate with some in-house systems (e.g., private GitLab CI instances).

Disadvantages

• More difficult to integrate on-premise testing with GitHub - for example, using a self-hosted GitHub runner can potentially collide with site security requirements.
• More difficult to make test results visible to the developer community.
• Setting up and maintaining site-specific testing requires additional effort.

Requirement Details

• It is not necessary for on-site testing to integrate with GitHub, but integration should be possible if that is desired.
• Sites need to be able to set up testing according to their site requirements around privacy, security, and regulatory compliance.