CI Pipeline Detection Issues

[macdjord]

Currently, PyTest detects when it is running in a CI pipeline by checking if either of the environment variables $CI or $BUILD_NUMBER are defined. This approach has multiple issues, all of which combine to complicate any attempt to work around the others:

1. There is no way to change what env variables it checks. Your CI/CD system uses a different variable name, or worse your non-CI testing environment needs $CI or $BUILD_NUMBER defined for some other reason? You're out of luck.
2. CI mode will be triggered if either of those variables is defined, even if the value is the empty string. Try to work around the above issue by adding CI=${MY_CI_ENV_VAR} to, e.g., the environment: section of your docker-compose file? Nope, even if $MY_CI_ENV_VAR is undef, $CI will now be defined-but-empty, thus triggering CI mode.
3. There's no way to override CI detection manually. Want to just ignore the whole mess and add --ci to the command-line your CI/CD system executes, or --no-ci in your non-CI test environment? Sorry, no such option.

The fixes to these problems are simple; implementing any one of them would break the triangle of annoyance, but all of them would be independently useful:

1. Add a command-line or config option to select which env variable(s) are used for CI detection
   • A command-line option can be specified in the config with addopts and a config option can be specified on the command-line with --override-ini, so there's no need to add both; in this case, a config option seems the more natural choice
   • Be sure to allow some way for the user to specify no variables at all, thus disabling automatic CI detection
2. Change CI detection to only activate if the selected env variable is a non-empty string
   • Stretch goal: Add an option to select what detection mode to use. Besides the existing options of 'defined' and 'nonempty', you could offer choices such as 'value equal to this given specific string' and 'anything but this given specific string'
3. Add command-line or config options to manually enable or suppress CI mode, which supersede the automatic detection when present
   • Again, either a command-line or config option will do; in this case, the natural choice seems to be command-line options --ci and --no-ci

[bluetech]

Add a command-line or config option to select which env variable(s) are used for CI detection

Would like to avoid adding complexity unless absolutely necessary.

Change CI detection to only activate if the selected env variable is a non-empty string

This seems like a reasonable and easy change to me. Would you like to send a PR changing this? I will approve it, but others may raise objections if there's something I didn't think about.

Add command-line or config options to manually enable or suppress CI mode, which supersede the automatic detection when present

This also seems reasonable, but previous change is preferable if sufficient.

[The-Compiler]

Using CI as a marker for being a CI system is somewhat of an informal standard, so the goal here should be that things "just work" essentially. That BUILD_NUMBER is needed in addition for Jenkins is somewhat unfortunate, but maybe due to Jenkins being older than many other CI systems (and thus this "standard").

Thus, I think any changes to this need to be driven by real-world situations. Does the "CI/CD system [that] uses a different variable name" actually exist, or is that purely hypothetical? Likewise, why would you need to define $CI or $BUILD_NUMBER in a non-CI system in the first place?

I'm mostly fine with treating an empty value the same as the variable being absent, as that's a common pattern when reading environment variables (though, annoyingly, simply using set/unset is too). It would be good to confirm that the common CI systems actually set a value though, and/or look at how other tools implement this. From a quick look at docs, CI is set to an actual value by:

• GitHub Actions
• GitLab CI
• CircleCI
• Travis CI
• AppVeyor

So 👍 from me for that part.

All in all, I'm also very much against over-engineering this and agree with what @bluetech said above.

[macdjord]

Change CI detection to only activate if the selected env variable is a non-empty string

This seems like a reasonable and easy change to me. Would you like to send a PR changing this? I will approve it, but others may raise objections if there's something I didn't think about.

Hrm. I believe I should be able to handle that.

Add command-line or config options to manually enable or suppress CI mode, which supersede the automatic detection when present

This also seems reasonable, but previous change is preferable if sufficient.

IMO, this would be a very good idea even with the previous change. I'll see if I can manage to add this, too, though I suspect doing so will be rather more complicated. Honestly, I'm surprised it doesn't already exist; the natural progression would seem to be add a CI mode controlled by command-line args and then add automatic detection based on env vars.

Thus, I think any changes to this need to be driven by real-world situations. Does the "CI/CD system [that] uses a different variable name" actually exist, or is that purely hypothetical? Likewise, why would you need to define $CI or $BUILD_NUMBER in a non-CI system in the first place?

Ah, mean culpa. We use Gitlab CI, and, due to a decision made long before I joined the team, we use $GITLAB_CI as our standard way of detecting when we're running in a CI/CD pipeline, and I wasn't aware that Gitlab defined $CI as well.

This rather thoroughly takes the wind out my argument for supporting alternate CI variables. I still think that ignoring blank vars and supporting manual overrides is a good idea, though.

As for why those env variables might be defined in a non-CI environment? Well, there are only 676 2-letter acronyms, so it's not all that unlikely that someone could use $CI to refer to something that has nothing to do with 'Continuous Integration'. As for $BUILD_NUMBER, one possibility that I can see would be an app which checks that variable in order to determine its own version number (e.g. to display a 'running version x.y.z' message in the corner) and thus the developer needs to have it set when doing manual testing.

[RonnyPfannschmidt]

there is a common standard - IMHO requesting adherence to it is better for cohesion and robustness than supporting a expansive group of standards/options

in fact now that the big forges all standardised on CI - i'm questioning id we should keep using build number as indication - id much rather see jenkins follow the standard than keeping supporting its special little thing

[macdjord]

I'm currently working on a patch to make only non-empty values for the env vars trigger CI mode. I will submit that patch under this ticket.

I have created a spin-off ticket, #13769, to handle the issue of adding command-line args for CI mode.