New paradigm for bundling plugin dependencies and files bundles

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3433

Originally created by @aaronsteers on 2022-04-23 23:11:08

---

Today:

transformers:
  - name: dbt
    namespace: dbt
    docs: https://docs.meltano.com/guide/transformation
    repo: https://github.com/dbt-labs/dbt-core
    pip_url: dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0
    settings:
    # ...
orchestrators:
  - name: airflow
    namespace: airflow
    docs: https://docs.meltano.com/guide/orchestration
    repo: https://github.com/apache/airflow
    pip_url: 'apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt'
    settings:
    # ...
# ...
files:
  - name: airflow
    namespace: airflow # << auto-installed with `dbt` due to matching `namespace`
    repo: https://gitlab.com/meltano/files-airflow
    pip_url: git+https://gitlab.com/meltano/files-airflow.git
    update:
      orchestrate/dags/meltano.py: true
  - name: dbt
    namespace: dbt # << auto-installed with `dbt` due to matching `namespace`
    repo: https://gitlab.com/meltano/files-dbt
    pip_url: git+https://gitlab.com/meltano/files-dbt.git@config-version-2

The challenge

The number of collisions on namespace will potentially skyrocket one we complete #3283. We can no longer reasonably scan all the plugins in the hub or in discovery.yml to identify matching namespaces and then have confidence that the matching plugins should be installed together. Instead, we need a more explicit mapping of files resources to the plugins that need those resources.

Proposed refactor:

• The namespace field can be removed (optionally auto-calculated internally per #3374).
• Files resources are now defined inline with the plugins themselves.

transformers:
  - name: dbt
    docs: https://docs.meltano.com/guide/transformation
    repo: https://github.com/dbt-labs/dbt-core
    pip_url: dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0
    bundle: # or `depends_on:`
      files:
      - name: files-dbt  # Name might be optional, since the plugin has a name in its API definition.
        # This is the API endpoint
        ref: https://hub.meltano.com/meltano/api/v1/plugins/extractors/tap-zendesk--twilio-labs
orchestrators:
  - name: airflow
    docs: https://docs.meltano.com/guide/orchestration
    repo: https://github.com/apache/airflow
    pip_url: 'apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt'
    bundle:
      files:
      - name: airflow-files
        variant: meltanolabs
    settings:
    # ...

The files needed for the repo would then be self-contained within the plugin definition and are self-sufficient for installing the plugin, without needing external plugin to be installed.

Moving to copier

Our current paradigm does not facilitate supporting changes over time to the files (the "day 2" problem). A related proposal would be to #3432, but that can be delivered separately from this change.

Standalone files plugins

We can still support standalone files plugins for use cases where it still makes sense to do so apart from auto-installed resources for a utility, transformer, or orchestrator plugin.

However, we would not post separate definitions to the hub for those files plugins which are best combined with their particular plugin.

[MeltyBot]

View 23 previous comments from the original issue on GitLab