meltano lock -> variant 'original' is not known to Meltano

[pnadolny13]

I'm getting the following error when I'm trying to run a meltano lock --all on the squared project (note, I did first manually updated to remove executable and namespace which were defined in my project). When I commented out the inherit_from children it works, the plugin is locked, then uncommenting the children again makes it fail again so I'm pretty sure its related to the inheriting.

File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 521, in get_parent
    parent, source = self.find_parent(plugin)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 510, in find_parent
    raise error
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 505, in find_parent
    return (self._get_parent_from_hub(plugin), DefinitionSource.HUB)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 441, in _get_parent_from_hub
    return self.hub_service.get_base_plugin(plugin, variant_name=plugin.variant)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/plugin_discovery_service.py", line 136, in get_base_plugin
    plugin = project_plugin.custom_definition or self.find_definition(
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/hub/client.py", line 189, in find_definition
    raise HubPluginVariantNotFound(
meltano.core.hub.client.HubPluginVariantNotFound: Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/cli/__init__.py", line 74, in _run_cli
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']
Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']

[pnadolny13]

Additionally I'm getting Orchestrator 'airflow' variant 'original' is not known to Meltano. Variants: ['apache (default)'] from a default installation of Airflow. I know that its hard because when I originally installed it there wasnt a variant name for airflow but now there is. Is there a way for us to resolve that its the default variant? Maybe that wont work because the default could change, maybe using the discovery.yml when we think were migrating to >2.0.0?

Or we could log it better and say something like "I notice youre migrating and you dont have a variant, go to the Migration guide and learn how to define a variant" and we write up the steps for locking post migration and adding a variant name:

• remove executable/namespace if defined
• add variant using one available on MeltanoHub. We can list the mapping like airflow -> apache, dbt -> dbt-labs, etc. for plugins that just got variant names
• run meltano lock --all

[edgarrmondragon]

Thanks for logging @pnadolny13!

So the inherit_from behavior is a more or less serious bug.

For the other one related to plugins added with the "original" variant pre-2.0, I like your suggestion to improve the error message with instructions and the new default variant names.

[pnadolny13]

@edgarrmondragon ok cool, I didnt know if they were going to end up being related but should I spin the messaging/docs one off as a separate issue?

[edgarrmondragon]

@pnadolny13 yeah, please do. If only just to keep the respective changesets smaller.

[pnadolny13]

@edgarrmondragon new issue created for the log message/migration guide entry #6360.

[DouweM]

@pnadolny13 Just to confirm, you're saying that meltano lock --all worked with only this:

plugins:
  extractors:
  - name: tap-github
    pip_url: git+https://github.com/MeltanoLabs/tap-github@d99378778c0cebc446c12b552ee4fd386fdc2610
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: auth_token
      kind: password

But not when this was (re)added:

plugins:
  extractors:
  # ...
  - name: tap-github-meltano
    inherit_from: tap-github
    config:
      organizations:
      - MeltanoLabs
      - meltano
      stream_maps:
        issues:
          __filter__: record['type'] = 'issue'

I wonder why in that first case with only tap-github, we didn't hit the same error as #6360, which I'd expect any time there's a name but no variant (and no namespace to indicate a custom plugin, or inherit_from to indicate an explicitly inherited one).

remove executable/namespace if defined

I don't think we should recommend this: namespace is still supported and valuable, as it indicates custom plugins that shouldn't be looked up on the Hub (see also #3296 (comment) about why we can't just remove that property).

[edgarrmondragon]

@DouweM The second one is failing because the plugin is inherited, and thus not requiring a variant. In this case, the locking should altogether be skipped according to the lockfile spec, i.e. not adding lockfiles for inherited plugins, only the parent, so that's the bug i.m.o.: meltano lock should skip inherited plugins.

To be more clear: my naive implementation of meltano lock first tried to pull all plugin definitions from the Hub to make sure the ProjectPlugin.parent instance is actually the latest definition when passed to the lockfile manager. I think I got a quick fix for that.

[tayloramurphy]

@edgarrmondragon thanks for hopping on this - assigning to you for the current iteration.

Just to be clear, the scope of this is to:

• update meltano lock to skip inherited plugins
• update https://docs.meltano.com/concepts/plugins#lock-artifacts to indicate as much

That seem accurate?

[edgarrmondragon]

Just to be clear, the scope of this is to:

• update meltano lock to skip inherited plugins
• update https://docs.meltano.com/concepts/plugins#lock-artifacts to indicate as much

That seem accurate?

@tayloramurphy yup, that is right