Skip to content

Commit

Permalink
feat(cli): Support installing multiple plugins of any type (meltano#8355
Browse files Browse the repository at this point in the history
)

* Install multiple plugins of any type

* Add/update tests

* Document `meltano install -`

* Undo bad change

---------

Co-authored-by: Edgar Ramírez Mondragón <16805946+edgarrmondragon@users.noreply.github.com>
  • Loading branch information
ReubenFrankel and edgarrmondragon committed Apr 3, 2024
1 parent f77789f commit 0c39970
Show file tree
Hide file tree
Showing 4 changed files with 48 additions and 8 deletions.
8 changes: 4 additions & 4 deletions docs/docs/guide/plugin-management.md
Expand Up @@ -75,7 +75,7 @@ is repeated here for convenience, since you may want to update it to
If this property is omitted, it is inherited from the discoverable [base plugin description](/concepts/plugins#project-plugins) identified by the `name` (and `variant`) instead.

As mentioned above, directly adding a plugin to your [`meltano.yml` project file](/concepts/project#plugins)
and [installing it](#installing-your-projects-plugins) using [`meltano install <type> <name>`](/reference/command-line-interface#install)
and [installing it](#installing-your-projects-plugins) using [`meltano install [<type>|-] <name>`](/reference/command-line-interface#install)
has the same effect as adding it using [`meltano add`](/reference/command-line-interface#add).

#### Variants
Expand Down Expand Up @@ -367,7 +367,7 @@ by default, you'll need to explicitly run [`meltano install`](/reference/command
before any other `meltano` commands whenever you clone or pull an existing Meltano project from version control,
to install (or update) all plugins specified in your [`meltano.yml` project file](/concepts/project#meltano-yml-project-file).

To install a specific plugin in your project, use [`meltano install <type> <name>`](/reference/command-line-interface#install), e.g. `meltano install extractor tap-gitlab`. Subsequent calls to `meltano install` will upgrade a plugin to its latest version, if any. To completely uninstall and reinstall a plugin, use `--clean`.
To install a specific plugin in your project, use [`meltano install [<type>|-] <name>`](/reference/command-line-interface#install), e.g. `meltano install extractor tap-gitlab` or `meltano install - tap-gitlab`. Subsequent calls to `meltano install` will upgrade a plugin to its latest version, if any. To completely uninstall and reinstall a plugin, use `--clean`.

## Pinning a plugin to a specific version

Expand Down Expand Up @@ -513,7 +513,7 @@ If you've forked a plugin's repository and made changes to it, you can update yo
1. Reinstall the plugin from the new `pip_url` using [`meltano install`](/reference/command-line-interface#install):

```bash
meltano install <type> <name>
meltano install [<type>|-] <name>

# For example:
meltano install extractor tap-gitlab
Expand Down Expand Up @@ -553,7 +553,7 @@ you can [add the new variant as a separate plugin](#multiple-variants) or switch
1. Reinstall the plugin from the new `pip_url` using [`meltano install`](/reference/command-line-interface#install):

```bash
meltano install <type> <name>
meltano install [<type>|-] <name>

# For example:
meltano install loader target-postgres
Expand Down
4 changes: 4 additions & 0 deletions docs/docs/reference/command-line-interface.md
Expand Up @@ -768,6 +768,8 @@ Installs dependencies of your project based on the **meltano.yml** file.
Optionally, provide a plugin type argument to only (re)install plugins of a certain type.
Additionally, plugin names can be provided to only (re)install those specific plugins.

To install a plugin without knowing its type, or to install multiple plugins of varying types, use the special `-` (any) character as the plugin type argument, followed by plugin name(s). `meltano install -` with no additional positional arguments has the same effect as `meltano install` (i.e. install all plugins).

To only install plugins for a particular schedule specify the `--schedule` argument.
This can be useful in CI test workflows or for deployments that need to install plugins before every run.

Expand All @@ -787,10 +789,12 @@ Meltano stores package installation logs in `.meltano/logs/pip/{plugin_type}/{pl

```bash
meltano install
meltano install -

meltano install extractors
meltano install extractor tap-gitlab
meltano install extractors tap-gitlab tap-adwords
meltano install - tap-gitlab target-postgres
meltano install --schedule=<schedule_name>

meltano install --parallelism=16
Expand Down
11 changes: 7 additions & 4 deletions src/meltano/cli/install.py
Expand Up @@ -18,13 +18,15 @@
from meltano.core.project import Project
from meltano.core.tracking import Tracker

ANY = "-"

logger = structlog.getLogger(__name__)


@click.command(cls=PartialInstrumentedCmd, short_help="Install project dependencies.")
@click.argument(
"plugin_type",
type=click.Choice(PluginType.cli_arguments()),
type=click.Choice((*PluginType.cli_arguments(), ANY)),
required=False,
)
@click.argument("plugin_name", nargs=-1, required=False)
Expand Down Expand Up @@ -74,14 +76,15 @@ def install( # noqa: C901
"""
tracker: Tracker = ctx.obj["tracker"]
try:
if plugin_type:
if plugin_type and plugin_type != ANY:
plugin_type = PluginType.from_cli_argument(plugin_type)
plugins = project.plugins.get_plugins_of_type(plugin_type)
if plugin_name:
plugins = [plugin for plugin in plugins if plugin.name in plugin_name]
else:
plugins = list(project.plugins.plugins())

if plugin_name:
plugins = [plugin for plugin in plugins if plugin.name in plugin_name]

if schedule_name:
schedule_plugins = _get_schedule_plugins(
ctx.obj["project"],
Expand Down
33 changes: 33 additions & 0 deletions tests/meltano/cli/test_install.py
Expand Up @@ -36,6 +36,20 @@ def test_install(self, project, tap, tap_gitlab, target, dbt, cli_runner):
force=False,
)

with mock.patch("meltano.cli.install.install_plugins") as install_plugin_mock:
install_plugin_mock.return_value = True

result = cli_runner.invoke(cli, ["install", "-"])
assert_cli_runner(result)

install_plugin_mock.assert_called_once_with(
project,
[tap, tap_gitlab, target, dbt],
parallelism=None,
clean=False,
force=False,
)

@pytest.mark.usefixtures("dbt")
def test_install_type(
self,
Expand Down Expand Up @@ -164,6 +178,25 @@ def test_install_multiple(self, project, tap, tap_gitlab, cli_runner):
force=False,
)

@pytest.mark.usefixtures("dbt")
def test_install_multiple_any_type(self, project, tap, target, dbt, cli_runner):
with mock.patch("meltano.cli.install.install_plugins") as install_plugin_mock:
install_plugin_mock.return_value = True

result = cli_runner.invoke(
cli,
["install", "-", tap.name, target.name, dbt.name],
)
assert_cli_runner(result)

install_plugin_mock.assert_called_once_with(
project,
[tap, target, dbt],
parallelism=None,
clean=False,
force=False,
)

def test_install_parallel(
self,
project,
Expand Down

0 comments on commit 0c39970

Please sign in to comment.