Add "validation" pytest marker

[VeckoTheGecko]

Description

This PR:

• Adds a validation marker for the validation tests
• Moves the validation tests (so that they can now be picked up by pytest - pytest assumes a file name of test_...)
• Updates the test finding to skip flaky and validation tests by default (also updates our CI to match)
• Adds test-flaky and test-validation pixi tasks

Checklist

• Closes None (followup from Feature/verification tests #2628 )
• This PR targets the correct branch (main for normal development, v3-support for v3 support)

AI Disclosure

None used

[VeckoTheGecko]

@erikvansebille @fluidnumericsJoe do you think we should add another github action workflow which can run the validation tests?

Options:

• have it run all the time
• have it run on request (i.e., when a "run-validation" label is applied to a PR, then the workflow will be run for that PR)
• leave it to be run manually by the user

[erikvansebille]

Looks good; two small comments below

[erikvansebille]

Why is this needed? Does not seem to be used in the rest of the script?

[VeckoTheGecko]

This is used by Pytest (pytest is a mildly magical package in the sense that it looks for these sorts of specially named things). It marks the whole module as validation

You can see that by doing pytest -m validation

[erikvansebille]

@erikvansebille @fluidnumericsJoe do you think we should add another github action workflow which can run the validation tests?

Options:

• have it run all the time
• have it run on request (i.e., when a "run-validation" label is applied to a PR, then the workflow will be run for that PR)
• leave it to be run manually by the user

We could run the validation before we do a release? Perhaps simply by adding it to our own workflow notes?

[VeckoTheGecko]

We could run the validation before we do a release? Perhaps simply by adding it to our own workflow notes?

I think that sounds good

[fluidnumericsJoe]

Why would we skip them when other tests are run? Feels like a recipe to potentially miss things until we want to release and there's "one more thing" to do at release. If there's tests that fail then it could kill predictability here on time to release

[fluidnumericsJoe]

IMO the validation tests should be run on PRs and pushes to main. We previously had code that was "correct" but wasn't found out as problematic until these examples were run out of cycle in python notebooks; hence why they were brought in.

[VeckoTheGecko]

Gotcha. Thanks for the input @fluidnumericsJoe . Since CI times were a concern last time we chatted, would it be useful to have it as another required workflow in GHA thats separate to the unit tests? Would that be a good solution?

[VeckoTheGecko]

And I think you're right @fluidnumericsJoe - having this in the iteration cycle in CI is a good idea.