[KED-2691] Link kedro-starters CI exclusively to kedro main branch

[merelcht]

(transfer from Jira, created by @ignacioparicio)

Description

kedro-starters CI will fail whenever we update requirements.txt of kedro. This is because:

• CI for kedro-starters first _pip install_s test_requirements.txt (https://github.com/kedro-org/kedro-starters/blob/master/features/environment.py#L70). This file refers to the main branch from kedro (https://github.com/kedro-org/kedro-starters/blob/master/test_requirements.txt#L8), which is ahead of the latest release
• CI for kedro-starters then installs project specific dependencies (https://github.com/kedro-org/kedro-starters/blob/master/features/run.feature#L6). In many cases, these project specific dependencies contain (part of) kedro itself (e.g. _kedro[pandas.CSVDataSet]==\{{ cookiecutter.kedro_version }}_ https://github.com/kedro-org/kedro-starters/blob/master/pandas-iris/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/requirements.txt for pandas-iris). Unlike before, this points to the latest kedro release
• Whenever kedro's requirement.txt are updated, this will lead to a conflict

For now we are avoiding to do pip compile in order to deal with non-breaking changes in requirements.txt (see #36). A better solution:

• Would have kedro-starters CI linked just to kedro's main branch, not the latest release
• Would install just the requirements needed for each starter (status quo)

Possible implementation of this solution would be to patch requirements.txt in the kedro-starters CI to change any kedro[*]=={{ cookiecutter.kedro_version }} (which points to latest kedro release) to point to main instead. This would then enable a pure kedro install command to be run during CI (without the --no-build-reqs flag).

If a solution is found, we might consider reverting the changes made https://github.com/kedro-org/kedro-starters/pull/36/files, as they shouldn't be required anymore.

[merelcht]

Outdated ticket. We have already revamped the CI setup to use Github Actions and requirements have been moved to pyproject.toml. We can create a new task if something like this is still needed.