Behavior of PARALLEL_RUN

[antonio-gg-dev]

Both @Chemaclass and I found it odd that if we left PARALLEL_RUN empty in the .env file, it would run in parallel. This is because in the code it is evaluated as PARALLEL_RUN = true to run in parallel. However, the env_configuration.sh defines PARALLEL as true if it's empty.

Both @Chemaclass and I found it more intuitive that if it's empty, it should run synchronously (in the vast majority of languages, the absence of value/null/undefined is evaluated as false).

If we wanted to change this behavior, I believe we should invert the conditions in the code (i.e., use PARALLEL_RUN != false instead of PARALLEL_RUN = true) so that the code expresses a more realistic behavior.

For now, we've changed it so that the GitHub actions work correctly due to issue #39.

Another distinct matter is whether we want the GitHub actions to run in parallel (once the issue #39 is resolved). In that case, I think it would be interesting to have a separate .env for GitHub actions from the .env.example.

To summarize, I want to discuss three points in this issue.

1. What behavior do we expect when PARALLEL_RUN is empty?
   • My opinion is that it should be evaluated as false.
2. If we want empty PARALLEL_RUN evaluates as true, do we invert the code checks so they don't lead to confusion when reading them?
   • My opinion is that they should be inverted if we decide that an empty PARALLEL_RUN evaluates as true.
3. How do we want it to launch in GitHub actions and how do we proceed to do it?
   • My opinion here is that they should run in parallel as soon issue Total assertions count not working when parallel run enabled #39 is fixed, and we should have a separate .env from the .env.example for GitHub actions (i.e. .env.github).

What is your opinion?

[antonio-gg-dev]

To clarify one thing, this behavior was prior to pull request #40. @Chemaclass and I have changed how it behaves, so now empty PARALLEL_RUN run tests synchronously, but we want to know your opinion on how to proceed (and also regarding the GitHub actions).

[fabriziofs]

1. What behaviour do we expect when PARALLEL_RUN is empty?
   • My opinion is that by default tests should run synchronously and let the user change this behaviour with an env variable
2. If we want empty PARALLEL_RUN evaluates as true, do we invert the code checks so they don't lead to confusion when reading them?
   • Same as @Tito-Kati
3. How do we want it to launch in GitHub actions and how do we proceed to do it?
   • Maybe we can use this option of Github Actions to customize how everything works on CI environment

[antonio-gg-dev]

I'm closing this issue as it seems we all agree that we like how it currently works.

Aside from fixing the errors (covered in another issue) and deciding whether we will run it synchronously or asynchronously in the CI.