[ENH][MNT] testing version bumps, e.g., dependabot upgrades - how to ensure objets affected by soft dependency changes are tested

[fkiraly]

question: how do we test, between releases, compatibility of an estimator with a soft dependency if we/dependabot bump the upper bound of said dependency?

problem:

• we do diff testing, test estimators only if their module has changed
• so when bumping the soft dep only, the estimator will not change

Currently, I have been manually merging to main and then running the "run all estimators" diagnostic #5083, but that is highly manual.

Some solutions that we have discussed on the discord:

• have a separate branch for version bumps and merge dependabot in that. Merge triggers full CI. That is basically DIAGNOSTIC - run all tests on all estimators #5083 with automation.
• run regular CRONs with all estimators, similar to this workflow by @romanlutz [BUG] pmdarima dependency requirement should be increased #5081 (comment) - and adopt a workflow that ensures it runs at least once before release. Could be based on the workflow @romanlutz posted.
• some python-based test machinery that can retrieve estimators whose dependency versions have changed since the last tag. Seems doable but may be a lot of work from scratch.

There is not just the tech side, but also the "release manager instructions" side, and the "developer instructions" side - we do not want to ovecomplicate the instructions, or introduce too many things that one has to think of at the right time, or we introduce risk through less streamlined process.

FYI @yarnabrina, @romanlutz, @benHeid. Also @tarpas as possibly interested.

[fkiraly]

Here's an idea: in GHA, can we check whether a PR comes from dependabot, and in this case turn off the "differential testing" flag? That would be only_changed_modules.

[yarnabrina]

I am not sure if "PR created by dependabot" can be detected directly or not, but PR title/label/etc. can be detected. And we are already controlling the labels, so that can be the easy way forward. We can create special tags for manual/automated dependency updates.

sktime/.github/dependabot.yml

Line 11 in c5ea83a

- "maintenance"

[fkiraly]

We can create special tags for manual/automated dependency updates.

Sounds good! And these would trigger full CI?

[yarnabrina]

I don't think so, at least not automatically. I was just answering the "can" question, assuming you know how to do the "turn off" part.

I suppose once we detect the label, there seems to be existing actions for that, we can set some environment variables or pass specific CLI flags (never tried myself, but don't see why it can't be done). Will that be enough to control pytest_generate_tests?