Airflow install fails on fresh Meltano 3.0 install #48
Comments
|
Can probably be fixed by updating the - pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3
- --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt
+ pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.1
+ --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txtIf that fixes the problem, then the plugin should be updated: https://github.com/meltano/hub/blob/0fde97d2d899a7e1127259304d890097715a4b5a/_data/meltano/utilities/airflow/apache.yml#L54-L55 |
|
Thank you for the suggestion. No luck meltano --log-level=debug install 57.9s Tue Sep 26 04:30:25 2023
2023-09-26T09:31:30.217656Z [debug ] /etc/localtime found
2023-09-26T09:31:30.219564Z [debug ] 1 found:
{'/etc/localtime is a symlink to': 'America/Chicago'}
2023-09-26T09:31:30.228482Z [debug ] Creating DB engine for project at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt' with DB URI 'sqlite://Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/meltano.db'
2023-09-26T09:31:30.309150Z [debug ] Found plugin parent parent=tap-spreadsheets-anywhere plugin=tap-spreadsheets-anywhere source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.310576Z [debug ] Found plugin parent parent=tap-shopify plugin=tap-shopify source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.312584Z [debug ] Found plugin parent parent=target-postgres plugin=target-postgres source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.313481Z [debug ] Found plugin parent parent=target-jsonl plugin=target-jsonl source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.314407Z [debug ] Found plugin parent parent=target-csv plugin=target-csv source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.316423Z [debug ] Found plugin parent parent=airflow plugin=airflow source=<DefinitionSource.LOCKFILE: 4>
Installing 6 plugins...
Installing extractor 'tap-spreadsheets-anywhere'...
2023-09-26T09:31:30.401483Z [debug ] Installing with args 'git+https://github.com/ets/tap-spreadsheets-anywhere.git' into virtual environment for 'extractors/tap-spreadsheets-anywhere'
Installing extractor 'tap-shopify'...
2023-09-26T09:31:30.677982Z [debug ] Installing with args 'git+https://github.com/sehnem/tap-shopify.git' into virtual environment for 'extractors/tap-shopify'
Installing loader 'target-postgres'...
2023-09-26T09:31:30.889848Z [debug ] Installing with args 'meltanolabs-target-postgres~=0.0.7' into virtual environment for 'loaders/target-postgres'
Installing loader 'target-jsonl'...
2023-09-26T09:31:30.974898Z [debug ] Installing with args 'target-jsonl' into virtual environment for 'loaders/target-jsonl'
Installing loader 'target-csv'...
2023-09-26T09:31:31.061081Z [debug ] Installing with args 'git+https://github.com/hotgluexyz/target-csv.git' into virtual environment for 'loaders/target-csv'
Installing utility 'airflow'...
2023-09-26T09:31:31.182350Z [debug ] Packages for 'utilities/airflow' have changed so performing a clean install.
2023-09-26T09:31:31.349110Z [debug ] Removed old virtual environment for 'utilities/airflow'
2023-09-26T09:31:31.349200Z [debug ] Creating virtual environment for 'utilities/airflow'
2023-09-26T09:31:32.068699Z [debug ] Upgrading pip for 'utilities/airflow'
2023-09-26T09:31:32.068854Z [debug ] Upgrading with args '--upgrade pip' in existing virtual environment for 'utilities/airflow'
Installed loader 'target-jsonl'
Installed loader 'target-postgres'
2023-09-26T09:31:34.076076Z [debug ] Installing with args 'git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-no-providers-3.11.txt' into virtual environment for 'utilities/airflow'
Installed loader 'target-csv'
Installed extractor 'tap-spreadsheets-anywhere'
Installed extractor 'tap-shopify'
Utility 'airflow' could not be installed: failed to install plugin 'airflow'.
Installed 5/6 plugins
2023-09-26T09:32:10.328108Z [debug ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:102 │
│ in _run_cli │
│ │
│ 99 │ """ │
│ 100 │ try: │
│ 101 │ │ try: # noqa: WPS225, WPS505 │
│ ❱ 102 │ │ │ cli(obj={"project": None}) │
│ 103 │ │ except ProjectReadonly as err: │
│ 104 │ │ │ raise CliError( │
│ 105 │ │ │ │ f"The requested action could not be completed: {err}", │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1157 in │
│ __call__ │
│ │
│ 1154 │ │
│ 1155 │ def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any: │
│ 1156 │ │ """Alias for :meth:`main`.""" │
│ ❱ 1157 │ │ return self.main(*args, **kwargs) │
│ 1158 │
│ 1159 │
│ 1160 class Command(BaseCommand): │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/cli.py:42 in │
│ main │
│ │
│ 39 │ │ │ args: Positional arguments for the Click group. │
│ 40 │ │ │ kwargs: Keyword arguments for the Click group. │
│ 41 │ │ """ │
│ ❱ 42 │ │ return super().main(*args, windows_expand_args=False, **kwargs) │
│ 43 │
│ 44 │
│ 45 @click.group( │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1078 in main │
│ │
│ 1075 │ │ try: │
│ 1076 │ │ │ try: │
│ 1077 │ │ │ │ with self.make_context(prog_name, args, **extra) as ctx: │
│ ❱ 1078 │ │ │ │ │ rv = self.invoke(ctx) │
│ 1079 │ │ │ │ │ if not standalone_mode: │
│ 1080 │ │ │ │ │ │ return rv │
│ 1081 │ │ │ │ │ # it's not safe to `ctx.exit(rv)` here! │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:612 in │
│ invoke │
│ │
│ 609 │ │ enact_environment_behavior(self.environment_behavior, ctx) │
│ 610 │ │ if ctx.obj.get("tracker"): │
│ 611 │ │ │ ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx)) │
│ ❱ 612 │ │ super().invoke(ctx) │
│ 613 │
│ 614 │
│ 615 class InstrumentedDefaultGroup(InstrumentedGroupMixin, DefaultGroup, DYMGroup): │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1688 in │
│ invoke │
│ │
│ 1685 │ │ │ │ super().invoke(ctx) │
│ 1686 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1687 │ │ │ │ with sub_ctx: │
│ ❱ 1688 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1689 │ │ │
│ 1690 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1691 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:661 in │
│ invoke │
│ │
│ 658 │ │ if ctx.obj.get("tracker"): │
│ 659 │ │ │ ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx)) │
│ 660 │ │ │ ctx.obj["tracker"].track_command_event(CliEvent.started) │
│ ❱ 661 │ │ super().invoke(ctx) │
│ 662 │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1434 in │
│ invoke │
│ │
│ 1431 │ │ │ echo(style(message, fg="red"), err=True) │
│ 1432 │ │ │
│ 1433 │ │ if self.callback is not None: │
│ ❱ 1434 │ │ │ return ctx.invoke(self.callback, **ctx.params) │
│ 1435 │ │
│ 1436 │ def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]: │
│ 1437 │ │ """Return a list of completions for the incomplete value. Looks │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:783 in invoke │
│ │
│ 780 │ │ │
│ 781 │ │ with augment_usage_errors(__self): │
│ 782 │ │ │ with ctx: │
│ ❱ 783 │ │ │ │ return __callback(*args, **kwargs) │
│ 784 │ │
│ 785 │ def forward( │
│ 786 │ │ __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any # noqa: B902 │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/decorators.py:33 in │
│ new_func │
│ │
│ 30 │ """ │
│ 31 │ │
│ 32 │ def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R": │
│ ❱ 33 │ │ return f(get_current_context(), *args, **kwargs) │
│ 34 │ │
│ 35 │ return update_wrapper(new_func, f) │
│ 36 │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:27 in │
│ decorate │
│ │
│ 24 │ │ if database_uri: │
│ 25 │ │ │ ProjectSettingsService.config_override["database_uri"] = database_uri │
│ 26 │ │ │
│ ❱ 27 │ │ return func(*args, **kwargs) │
│ 28 │ │
│ 29 │ return functools.update_wrapper(decorate, func) │
│ 30 │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:75 in │
│ decorate │
│ │
│ 72 │ │ │ │ except MigrationError as err: │
│ 73 │ │ │ │ │ raise CliError(str(err)) from err │
│ 74 │ │ │ │
│ ❱ 75 │ │ │ func(project, *args, **kwargs) │
│ 76 │ │ │
│ 77 │ │ return functools.update_wrapper(decorate, func) │
│ 78 │
│ │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/install.py:110 │
│ in install │
│ │
│ 107 │ ) │
│ 108 │ if not success: │
│ 109 │ │ tracker.track_command_event(CliEvent.failed) │
│ ❱ 110 │ │ raise CliError("Failed to install plugin(s)") │
│ 111 │ tracker.track_command_event(CliEvent.completed) │
│ 112 │
│ 113 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Failed to install plugin(s)
The above exception was the direct cause of the following exception:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:112 │
│ in _run_cli │
│ │
│ 109 │ │ except MeltanoError as err: │
│ 110 │ │ │ handle_meltano_error(err) │
│ 111 │ │ except Exception as err: │
│ ❱ 112 │ │ │ raise CliError(f"{troubleshooting_message}\n{err}") from err │
│ 113 │ except CliError as cli_error: │
│ 114 │ │ cli_error.print() │
│ 115 │ │ sys.exit(1) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s)
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s) |
|
This might be a Mac only issue. I was able to install airflow utility plugin -v 2.3.3 inside Meltano docker project. |
Thanks. This works for me. |
Yes. I change to apache-airflow==2.8.0. works fine in docker container. Failed on MacOS. on Mac I used anaconda virual env. Python=3.11. |
I guess it's a problem for newish Apple Silicon macs, probably because there aren't any compatibles wheels for some package, and the build step is failing. |
|
For ref: meltano/hub#1647 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi.
Trying out Airflow utility, ran into problems on mac.
This a new Meltano 3.0 install on osx 13.6
Fresh new Python 3.11.5 using asdf.
Able to install a few other utilities, extractors and loaders, but not airflow.
Here is the error message:
The text was updated successfully, but these errors were encountered: