Feature request: Make building and executing tests mandatory

[HappySeaFox]

---

Is your feature request related to a problem? Please describe.

For now, it's totally up to the port maintainer to control if the port is functional or not. Most probably maintainers don't test ports on every supported platform and in debug/release mode on every platform.

This is why it's totally possible to commit partially functional ports.

For example, HappySeaFox/sail#128

Proposed solution

Make building and executing tests in ports mandatory. This way we can make sure every port is functional on every platform and build mode. At least, this is a minimal quality control we can implement.

If a port doesn't have any tests or example applications to build, it may be recommended to add one explicitly by the port maintainer.

Also, it would be great to compile a small external test application against a port with cmake -DCMAKE_TOOLCHAIN_FILE=.... This is needed because of possibly broken cmake rules in some ports. See
#21234

Describe alternatives you've considered
-

Additional context
-

[JackBoosY]

Yes, we need this.
I hope that upstream provides test code for testing after build, and vcpkg should also be tested against the generated cmake or other configurations.

Related: #15173.

[JackBoosY]

And I think we need to discuss how to implement it using a unified interface.

[HappySeaFox]

Another uncaught port issue caused by the fact that we don't execute tests: HappySeaFox/sail#190. Fixed by #29809

[HappySeaFox]

I also propose two possible ways to test the built port: 1) introduce a function to simply run cmake tests with ctest 2) allow the user to run a specific test executable

This way I can choose how to test the built port in portfile.

[HappySeaFox]

Hi! Any progress? I'm sure this is a great investment into the ports' stability.