Update Dependabot for python virtual envs

[jalaziz]

Proposed Changes

1. Simplify virtualenv installation to use multiple requirements files in
   a single directory. This eliminates the packages.txt file and provides
   an easier path to adding new tools in the future. It also allows us to
   simplify the dependenabot configuration as all requirements files are
   in a single directory.
2. Update dependabot.yml to point to the new Python dependencies folder.

Readiness Checklist

Author/Contributor

• If documentation is needed for this change, has that been included in this pull request

Reviewing Maintainer

• Label as breaking if this is a large fundamental change
• Label as either automation, bug, documentation, enhancement, infrastructure, or performance

[jalaziz]

@lindluni I did some testing with Dependabot and it seems like Dependabot is not smart enough to handle conflicting dependencies (seems like it expects a build failure to catch that), but it is smart enough to handle multiple files in the same directory.

What this gets us is a slightly simplified Dependabot configuration. Since it recognizes any .txt file as a python requirements file, we can have one per tool. Also, since Dependabot doesn't actually try to figure out if the versions conflict across files, it will go ahead and put up PRs to update any single dependency across all files that list it.

The only downside I can see here is it may eagerly update dependencies with versions that don't meet the transitive requirements, but that would fail on installation (which should fail the PR checks). We'd then have to update each the requirements independently if we want to unblock an upgrade, but the tradeoff seems worth it for fewer Dependabot PRs and config entries.

[admiralAwkbar]

Seems like a solid update

[admiralAwkbar]

@lindluni Any issues?

[lindluni]

LGTM: I see now, dependabot doesn't support recursive wildcards for the directory to scan 😢

[jalaziz]

LGTM: I see now, dependabot doesn't support recursive wildcards for the directory to scan 😢

Yeah, sorry, I could've clarified that more on the original PR. One of the more annoying dependabot limitations.