Document supported Python versions for a plugin

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/129

Originally created by @edgarrmondragon on 2021-10-04 15:39:55

---

In a recent Slack thread a user came across an issue with pipelinewise-target-bigquery (not yet in the Hub) caused by a target's dependency not being compatible with Python ≥ 3.8.

It would help if discovery.yml and the Hub documented Python versions supported by the plugin, so Meltano can eventually warn or fail when an incompatible version is being used (https://gitlab.com/meltano/meltano/-/issues/2854).

Gathering this information for plugins based on the SDK will be easier, but for the rest of the ecosystem will be harder since a lot of taps and targets don't explicitly mention supported versions in their readmes.

[MeltyBot]

View 4 previous comments from the original issue on GitLab

[tayloramurphy]

@aaronsteers what are your thoughts on this issue? With the work Florian is doing for the Plugin SDK, I think we'd have a clear path there to declare Python version support. I think for Singer plugins it would make sense to have some declaration in the metadata when we can't find one in the plugin itself.

Ideally we could also test them with the different python versions as well...

[aaronsteers]

@tayloramurphy - This apparently got lost in the GitHub notification flood a while back.

To answer your question, yes, I would support a python_version: {comma-separated-semvar-constraints} annotation for plugin definitions. For instance, Airflow might be constrained to python:version: >3.6,<3.11.

In theory, we could evaluate this at runtime and provide a warning during meltano add and/or during invoke or run.

This is back on my mind with Python 3.11 coming out this week. If we want to pursue this, probably the first two things would be:

1. Add python_version as option in plugin definition JSON Schema.
2. Add a default python_version constraint matching Meltano's own (e.g. something like >3.6.4,<3.11) to all plugins on the hub.
3. Add logic in Meltano that at leasts print a warning before attempting installation (if not failing the installation entirely) if the constraint is not met.

Logistical challenges then become:

1. When and how can we do rolling updates for all plugins on the hub?
2. At what point to we force update or hide plugins that haven't had their compatibility bumped to the latest version of Python?

A related mitigation which would allow users to pick a different Python interpreter if needed:

• Allow python interpreter overrides for specific plugins meltano#3333

Given the challenge with keeping up with the support profile of 500+ plugin variants on the hub, I think I'd prefer this as a warning message and not as a hard failure. 😅

cc @edgarrmondragon, @WillDaSilva, @pnadolny13

[tayloramurphy]

@aaronsteers I like the proposal and having it as a warn and not hard failure makes sense. We can pick this up as it starts to become more urgent. I was thinking we'd originally start with the EDK-based plugins and go from there.