Supporting instrumented package version bumps in our instrumentation packages

[NathanielRN]

Description

Following the SIG meeting on June 24th, 2021, we recognized the need to come up with a plan for when instrumented packages bump their version in a way that breaks our existing instrumentation packages.

Auto instrumentation relies our packages indicating both the package and the version of the package they supported which is why we need PRs like #545.

In the SIG we outlined the following options with their PROs and CONs

Option 1: Release a new instrumentation package for every breaking new instrumented version bump

This is what Java does already.

E.g. if Flask version 2.0 doesn't break opentelemetry-instrumentation-flask-v1 we can keep using it? And then if Flask version 3.0 does then we only create opentelemetry-instrumentation-flask-v3.

PROs:

• Tests/specialties for different versions separated into their own packages

CONs:

• It is confusing to have multiple opentelemetry-instrumentation-<pkgs>-v* packages
• We should probably deprecate the original opentelemetry-instrumentation-<pkgs>- package
• Every time names/fixes/features are added we need to apply it to all these different packages instead of just 1 package at a time

Option 2: Use 1 instrumentation package for all versions of the instrumented package

PROs:

• No confusion regarding which package we need to use

CONs:

• Testing is more complicated as we need to download 2+ versions of the same package to test different tests?
• Requires potentially very ugly logic branching to accommodate the different versions

Additional noteworthy points

• Our CI already runs quite long, and now we are adding tests for every version...
  • What if we only run tests for the latest version?
  • @aabmass mentioned that we could try running only certain package tests when a PR modifies an instrumentation package code path using GitHub Actions
• We only need to consider major version bumps, we should fix instrumentation for bug introduced with minor version bumps

[owais]

Option 2 makes a lot of sense to me. Most importantly, it's very obvious for end users. I think in general we should not make things harder for users just to make maintenance a bit easier for ourselves unless of course it gets very hard to do.

Testing is more complicated as we need to download 2+ versions of the same package to test different tests?

I think this is a test matrix / tox issue and we should solve it there instead of splitting packages. We already do this for elasticsearch: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/tox.ini#L172-L179

Requires potentially very ugly logic branching to accommodate the different versions

This is a very valid point but I think we can make it work with some re-organization and split the code into re-usable functions/classes.

I think we can still create dedicated instrumentation packages for different versions of libraries if things get too bad but I think this should be an exception rather than a rule and should be evaluated on a per package basis.

[codeboten]

Please vote with:
Option 1 🚢
Option 2 🚀

[lzchen]

From SIG discussions: since major version updates are not common, the cons listed for option 2 should be manageable/minimal.