Manual testing expectations for PRs

[xuang7]

Hi community,

I would like to start a discussion about our expectations for manual testing in PRs, especially around what should be tested and how testing responsibilities should be shared.

Currently, we rely heavily on automated checks, which are helpful and should remain the main baseline. We are also continuing to improve test coverage. At the same time, some issues may only show up in specific manual workflows or environments, such as local setup, Docker, or Kubernetes. For example, during RC2 validation, we found an environment-specific issue related to the new PVE feature (#5963). We have also seen cases during workshops or demos where some issues were only discovered during actual usage.

We probably do not need to require every PR to be tested in every environment, since that would add a lot of overhead. Instead, it may be helpful to define a lightweight standard for manual testing based on the type and risk of the PR.

Some questions I would like to discuss:

1. What should the manual testing expectations be for different types of PRs?

Examples:

• Frontend behavior changes
• Backend/API changes
• Database or schema changes
• Authentication/authorization changes
• Deployment, Docker, Kubernetes, or CI changes
• Large feature PRs

2. What should PR authors be expected to include?

Examples:

• What environment was tested
• What manual workflow was tested
• Whether screenshots, logs, or videos are useful
• Known limitations or untested areas

3. What should reviewers be expected to check?

Examples:

• Whether the testing section is clear enough
• Whether the testing level matches the scope/risk of the change
• Whether additional manual testing should be requested before merge
• Whether reviewers are expected to manually test the PR as well, and under what circumstances

4. How can we make testing across different environments more practical?

Examples:

• Are there ways to provide easier setup instructions for Docker or Kubernetes testing?
• For environments that are time-consuming to set up, can maintainers or reviewers help verify them when needed?
• Should certain high-risk PRs request help from someone who already has the relevant environment available?

The goal is not to increase overhead unnecessarily, but to develop a shared expectation for when manual testing is needed beyond automated checks. This can help clarify what information authors should provide and what reviewers should check, while improving feature reliability in real-world usage.

Please feel free to share your thoughts on what would be practical and reasonable for different types of PRs. Thanks!

[gracecluvohio]

For testing across environments, I'm not sure if it is necessary to target every single environment. We have limited time and lots of work, so I think we should only target the ones that are relevant. For instance, for frontend changes, testing across common browser; for backend, testing on similar environments to our cloud-based service/releases. Perhaps we can invest some time in creating dev containers like some other communities have done, to streamline setup for development and unify testing environments.

I think the burden of testing should fall on the PR author, rather than the reviewer. They should be responsible for testing on the necessary environments and cases and documenting it appropriately. Reviewers should not have to thoroughly test the code but should rather mostly check for general code quality. If they wish to, they can also manually test the PR, but I think that requiring manual checking could discourage reviewers from reviewing PRs and also block them from doing their own dev work. Furthermore, the author is the most familiar with their own PR, so they would have the most knowledge of what to test.

The fourth issue particularly resonates with me. #5961 definitely helps with dev work a lot, and I'd love to see more work to streamline development even more. Streamlining development would help make the feedback loop between testing and development a lot faster. Having better documentation for Docker and Kubernetes setup and keeping it updated would also help streamline the development process for everybody.

[carloea2]

1. What should the manual testing expectations be for different types of PRs? I prefer all of the PRs should be manually tested, 1 setting at least should be good enough.
2. What should PR authors be expected to include? I think current template is good but the points you mentioned will be helpful.
3. What should reviewers be expected to check? I prefer reviewer test the PR manually in 1 setting at least.
4. How can we make testing across different environments more practical? I have not proposal for this point.

Thanks