Automate Python dependency update

[ArBridgeman]

Summary

Updating the dependencies in Python projects is currently a manual process, which occurs for (1) security vulnerabilities, (2) ensuring our code works with the latest versions of dependencies, & (3) coordinate our shared tooling versions. We want to automate the process of updating dependencies, in a manner similar to our Java-based repos, to reduce the burden on the developer & expedite the remediation process.

Relevant comparison link: Roadmap #216

Details

Background & Context

From project-keeper, the two critical workflows to emulate are:

• https://github.com/exasol/project-keeper/blob/main/.github/workflows/dependencies_check.yml
• https://github.com/exasol/project-keeper/blob/main/.github/workflows/dependencies_update.yml

In the Python projects:

• we currently have dependabot enacted for dependency management. The pull requests generated by the dependabot updates the poetry.lock file, which is primarily used by developers. In order to communicate with the users of a package, the pyproject.toml and unrelease.md would need to be adapted, and in certain cases, a release should be created.
• additionally, in certain projects, we build SLCs for an extension, which rely on the pinned dependencies. These projects require a release with the newly built SLC & an updated requirements file:
  • transformers-extensions
  • sageMaker-extensions
  • advanced-analytics-framework
  • text-ai-extension
  • ai-lab
    • Special case that will require its own procedure, as it depends on all python projects & would need the new SLCs.

For tracking purposes, we need to create security issues to align with the PRs that would be opened.

Tasks

The 2 workflows we would create would check for security vulnerabilities, create issues, & open PRs

1. Use pip-audit to detect security vulnerabilities
2. Use the output from pip-audit to create security issues
   with https://github.com/exasol/python-toolbox/blob/main/.github/actions/security-issues/action.yml
3. Create structured PRs to update the poetry.lock file
4. Create structured PRs to update the pyproject.toml
   • This may require fine-tuning so that we specify which dependencies can be updated automatically based on poetry groups.
   • https://github.com/MousaZeidBaker/poetry-plugin-up
5. Once the system is setup, we should also consider & remove duplicated efforts, i.e. dependabot & determine if we want to have pip-audit in the CI workflow check or rather not.
6. Propagate to the python repos (without SLCs).
7. Propagate to python with SLCs.

6 &/|| 7, we might want as separate focuses