Testing Plan and Continuous Integration

System Testing Method

Our system is a web-based app implemented with Typescript for the frontend and python for the backend and simulation. Thus, two different tools will need to be used for each testing phase according to the new code's language.

Unit Tests and Coverage

All pull requests will be required to have unit tests covering at least 70% of the new code. Hence, the team member that created the pull request will need to create unit tests for their code before it can be approved. Frontend will be able to use Vitest for their unit tests and npm coverage to test and get the coverage report for their code, while backend/sim will use pytest for their unit tests. Codecov will be used to provide coverage reports for all tests within each branch.

Other Tests

To guarantee the quality of the system's code, various other tests will be run against it. The following extra checks will be done for both the front and back/sim.

1. Linting : Checks for programming and stylistic errors
   • Frontend: npm run lint
   • Backend/Sim: flake8 back or flake8 sim
2. Type Check : Checks if correct type of data is entered
   • Frontend: npm run typecheck
   • Backend/Sim: mypy back or mypy sim
3. Check Formatting : Checks the code's formatting
   • Frontend: npx prettier --check app/**/*.{js,jsx,ts,tsx,json,css,md} --no-error-on-unmatched-pattern
   • Backend/Sim: black --check back or black --check sim

These commands will be run automatically through the CI after each commit to a pull request. Further explanations in the next section.

Continuous Integration Strategy

Our continuous integration strategy aims to find bugs quickly and ensure code quality. It makes use of GitHub Actions to build the code, as well as run tests, on all pull requests and commits that follow.

Sub-team CIs

Since we have three different sub-teams, the continuous integration was separate into three to run different workflows for each of them: Frontend CI, Backend CI and Simulation CI. To allow GitHub Actions to decide which CI to run on a pull request, the workflow will analyze which subsystem folder was modified. It will be then run the corresponding CIs. This will additionally allow multiple CIs to be run in the case multiple subsystems were touched. Otherwise, it will only run the specific ones needed instead of running unnecessary ones.

The corresponding CI will be run after each commit in a pull request to execute all its tests, including linting, formatting and type check, and ensure the code is of quality. For the full list of checks, kindly refer to the flow diagram pictured below.

Pre-commit Hooks

Some pre-commit hooks have been setup in conjunction with the continuous integration to catch minor issues before the creation of each commit. These will check for linting issues, code formatting, merge conflicts and trailing whitespaces. In addition, it will ensure the commit message follows the Angular commit convention with an identified issue number (e.g. [type(scope)]: [description] #[issue number]). This message format must be respected to ensure commits are clear, organized, and reference the issue/comment they are aiming to resolve.

Code Review Process

Each pull request will require at least two code reviews and approvals to be considered completed before merging. Thanks to the codeowners file, as explained previously, at least one of the final approvals must be a member of the same sub-team as the branch subsystem (e.g. a PR under a branch from the sim subsystem would require at least one review/approval from a member in the sim team).

Pull requests have been separated into two categories: bug fix and regular PR. The code review process both types must be followed by the team before the pull request can be approved.

Bug Fix PRs

1. Access the bug issue referenced on the PR. Each bug fix PR should have a reference to a bug report issue in the Related Issues section. It is important to understand what problem is being fixed.
2. Follow the steps to reproduce the bug. These steps are provided in the bug report issue.
3. Ensure the steps result to the bug issue's expected behavior. The fix should have resolved the bug and resulted in the expected behavior as described.
   1. If not the case, add a comment to the PR and discuss the differences.
   2. If changes were made, repeat steps 1 to 3 to ensure the code follows the expected behavior.
4. Approve the pull request by leaving a comment. The comment should include the match in behavior between the tested one and the expected one.

General PRs

1. Access the referenced task issue and its user story issue on the PR. Each PR should have a reference to a task issue in the Related Issues section.
2. Cross-check the behavior/UI of the code against its AC/mockups. Ensure the pushed feature looks and behaves as intended through the task issue.
   • If not the case, add a comment to the PR and discuss the differences.
3. Verify the unit tests written. The written test cases should be effective in testing the new feature and cover it properly to ensure the quality of the code.
   1. Tests should cover 70% of the new code.
   2. Verify that they cover the main edge cases.
4. Approve the pull request by leaving a comment. The comment should include the validation of the code's behavior and UI, as well as an approval of the written tests.

User Story Signoff

Each user story is composed of a list of task and acceptance test sub-issues. All of these are expected to be completed before the user story can be closed. However, a certain flow should be followed for the completion of a user story to happen:

1. All tasks must be completed. Indeed, all tasks within a user story should be completed and closed by the team. Task issues automatically get closed once an associated PR is closed and merged.
2. End-to-End tests are created and cover ATs. Each user story could include multiple acceptance tests. These each need to be covered through E2E tests to validate the associated acceptance criteria. A .webm video showing the E2E test flow must then be uploaded on its associated AT sub-issue.
   • Due to some limitations, such as external activities not being recorded in E2E tests and sim initialization delays, some tests would be manually recorded or directly demo-ed to the stakeholder for approval.
3. All AT sub-issues are approved and closed by the stakeholder Once the video is uploaded to an AT sub-issue, it is ready to be reviewed by the stakeholder. The latter will then be assigned to the sub-issue and notified to review it when convenient or during a sprint meeting with him. Depending on the review given, the sub-issue could get closed or need to be reworked on.
   • If the stakeholder disapproves with the feature's implementation, he will leave a comment to request changes which will need to be addressed before re-requesting approval.
   • If the stakeholder approves the work done, he will leave an approval comment and the sub-issue will be closed.