Make namespace optional in plugin definitions, auto-generate if missing

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3374

Originally created by @aaronsteers on 2022-04-04 21:28:56

---

As of now, it feels like we could confidently calculate namespace as plugin_name.replace("-", "_").lower() or similar.

For reference: https://gitlab.com/meltano/meltano/blob/master/src%2Fmeltano%2Fcore%2Fbundle%2Fdiscovery.yml

Do we still need this as a required field in every plugin definition? When missing it seems to result in difficult to diagnose failure messages such as https://gitlab.com/meltano/meltano/-/issues/3357#note_900121643.

cc @tayloramurphy and @DouweM, since I think each of you may have more history on this functionality and when/if we'd still want namespace as a distinct field.

cc @pandemicsyn - Logged here as discussed in our 1:1.

cc @pnadolny13 since you've been working on discovery.yml plugin definition aspects in meltano and the hub.

[MeltyBot]

View 13 previous comments from the original issue on GitLab

[DouweM]

@tayloramurphy I don't think the behavior described in the GitLab comments accounts for the scenario where someone adds a custom plugin to their project today, and someone else (perhaps the same person) adds an identically named plugin to the Hub tomorrow. The existing plugin should not suddenly start inheriting the properties on the Hub in this case, as they may be incompatible, even if the variant is the same (or if the project definition has no variant). I think that's what would happen if we start ignoring namespace entirely.

Since a lockfile is only created if a definition on the Hub is found, the lack of a matching lockfile could indicate that a plugin definition in meltano.yml is custom, but that would not work for plugins added before 2.0 (that don't have lockfiles until meltano lock is called). And if meltano lock can't tell the difference from meltano.yml between a custom and Hub-inherited plugin, we run the same risk described above where a custom plugin suddenly turns inherited when it is added to the Hub and someone locks their plugins.

We could say that Hub-inherited plugins need a variant, and custom ones can't have one, but that's backward-incompatible with Meltano projects created before variants were added, that fall back on the original variant in discovery.yml (not always the same as the current default). We could require people to update their project again of course, but that's not ideal. Additionally, today a custom plugin can have both a namespace and a variant, so if we start ignoring namespace, some of these custom plugins would turn Hub-inherited and would fail if there's actually no such plugin/namespace on the Hub.

Finally, the principle of least surprise also suggests that a user should be able to tell at a glance whether the definition in meltano.yml is complete, and all there is to know to debug the plugin, or whether there may be some invisible "magic" properties inherited from somewhere else that may make things more difficult.

I don't think can (or should) get around some way of indicating custom vs Hub-inherited plugins, but it doesn't need to be namespace. Would a simple custom: true do the trick, that we'd treat the same for these purposes as namespace: <anything>? That means the more common Hub-inherited plugin definitions are still as compact as can be (name, variant, pip_url), plugin definitions explicitly inheriting from either a project-relative plugin or a Hub plugin with a different name are easily identifiable with inherit_from, and custom plugins with custom: true or namespace: foo are as well.

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[WillDaSilva]

Still relevant