fix: meltano run no longer creates empty venv plugin directories …

…for inherited plugins (#8447) * Add failing test * Fix venv path being created for inherited plugins * Fix TypeAlias annotation * Use explicit `make_dirs` kwarg * Fix `plugin_lock_path` call * Fix tests * ci: bump the actions group with 3 updates (#8471) Bumps the actions group with 3 updates: [actions/setup-python](https://github.com/actions/setup-python), [actions/dependency-review-action](https://github.com/actions/dependency-review-action) and [codecov/codecov-action](https://github.com/codecov/codecov-action). Updates `actions/setup-python` from 5.0.0 to 5.1.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5.0.0...v5.1.0) Updates `actions/dependency-review-action` from 4.2.4 to 4.2.5 - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@v4.2.4...v4.2.5) Updates `codecov/codecov-action` from 4.1.0 to 4.1.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v4.1.0...v4.1.1) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: actions/dependency-review-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump the development-dependencies group with 5 updates (#8472) Bumps the development-dependencies group with 5 updates: | Package | From | To | | --- | --- | --- | | [boto3-stubs](https://github.com/youtype/mypy_boto3_builder) | `1.34.69` | `1.34.74` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.99.13` | `6.100.0` | | [moto](https://github.com/getmoto/moto) | `5.0.3` | `5.0.4` | | [requests-mock](https://github.com/jamielennox/requests-mock) | `1.11.0` | `1.12.1` | | [types-jsonschema](https://github.com/python/typeshed) | `4.21.0.20240311` | `4.21.0.20240331` | Updates `boto3-stubs` from 1.34.69 to 1.34.74 - [Release notes](https://github.com/youtype/mypy_boto3_builder/releases) - [Commits](https://github.com/youtype/mypy_boto3_builder/commits) Updates `hypothesis` from 6.99.13 to 6.100.0 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@hypothesis-python-6.99.13...hypothesis-python-6.100.0) Updates `moto` from 5.0.3 to 5.0.4 - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.0.3...5.0.4) Updates `requests-mock` from 1.11.0 to 1.12.1 - [Release notes](https://github.com/jamielennox/requests-mock/releases) - [Commits](jamielennox/requests-mock@1.11.0...1.12.1) Updates `types-jsonschema` from 4.21.0.20240311 to 4.21.0.20240331 - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: boto3-stubs dependency-type: direct:development update-type: version-update:semver-patch dependency-group: development-dependencies - dependency-name: hypothesis dependency-type: direct:development update-type: version-update:semver-minor dependency-group: development-dependencies - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch dependency-group: development-dependencies - dependency-name: requests-mock dependency-type: direct:development update-type: version-update:semver-minor dependency-group: development-dependencies - dependency-name: types-jsonschema dependency-type: direct:development update-type: version-update:semver-patch dependency-group: development-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the runtime-dependencies group with 7 updates (#8473) * chore(deps): bump the runtime-dependencies group with 7 updates Bumps the runtime-dependencies group with 7 updates: | Package | From | To | | --- | --- | --- | | [boto3](https://github.com/boto/boto3) | `1.34.64` | `1.34.74` | | [click-didyoumean](https://github.com/click-contrib/click-didyoumean) | `0.3.0` | `0.3.1` | | [croniter](https://github.com/kiorky/croniter) | `2.0.2` | `2.0.3` | | [google-cloud-storage](https://github.com/googleapis/python-storage) | `2.15.0` | `2.16.0` | | [importlib-resources](https://github.com/python/importlib_resources) | `6.3.1` | `6.4.0` | | [smart-open](https://github.com/piskvorky/smart_open) | `7.0.1` | `7.0.4` | | [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) | `2.0.28` | `2.0.29` | Updates `boto3` from 1.34.64 to 1.34.74 - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.34.64...1.34.74) Updates `click-didyoumean` from 0.3.0 to 0.3.1 - [Release notes](https://github.com/click-contrib/click-didyoumean/releases) - [Commits](click-contrib/click-didyoumean@v0.3.0...v0.3.1) Updates `croniter` from 2.0.2 to 2.0.3 - [Changelog](https://github.com/kiorky/croniter/blob/master/CHANGELOG.rst) - [Commits](https://github.com/kiorky/croniter/commits) Updates `google-cloud-storage` from 2.15.0 to 2.16.0 - [Release notes](https://github.com/googleapis/python-storage/releases) - [Changelog](https://github.com/googleapis/python-storage/blob/main/CHANGELOG.md) - [Commits](googleapis/python-storage@v2.15.0...v2.16.0) Updates `importlib-resources` from 6.3.1 to 6.4.0 - [Release notes](https://github.com/python/importlib_resources/releases) - [Changelog](https://github.com/python/importlib_resources/blob/main/NEWS.rst) - [Commits](python/importlib_resources@v6.3.1...v6.4.0) Updates `smart-open` from 7.0.1 to 7.0.4 - [Release notes](https://github.com/piskvorky/smart_open/releases) - [Changelog](https://github.com/piskvorky/smart_open/blob/develop/CHANGELOG.md) - [Commits](piskvorky/smart_open@v7.0.1...v7.0.4) Updates `sqlalchemy` from 2.0.28 to 2.0.29 - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: runtime-dependencies - dependency-name: click-didyoumean dependency-type: direct:production update-type: version-update:semver-patch dependency-group: runtime-dependencies - dependency-name: croniter dependency-type: direct:production update-type: version-update:semver-patch dependency-group: runtime-dependencies - dependency-name: google-cloud-storage dependency-type: direct:production update-type: version-update:semver-minor dependency-group: runtime-dependencies - dependency-name: importlib-resources dependency-type: direct:production update-type: version-update:semver-minor dependency-group: runtime-dependencies - dependency-name: smart-open dependency-type: direct:production update-type: version-update:semver-patch dependency-group: runtime-dependencies - dependency-name: sqlalchemy dependency-type: direct:production update-type: version-update:semver-patch dependency-group: runtime-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * Fix tests --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Edgar Ramírez-Mondragón <edgar@meltano.com> * Type check `meltano.core.project` --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
meltano · Apr 2, 2024 · dcf91e6 · dcf91e6
1 parent f2607b6
commit dcf91e6
Show file tree

Hide file tree

Showing 11 changed files with 137 additions and 40 deletions.
diff --git a/pyproject.toml b/pyproject.toml
@@ -530,7 +530,6 @@ module = [
   "meltano.core.plugin_install_service",
   "meltano.core.plugin_invoker",
   "meltano.core.plugin_test_service",
-  "meltano.core.project",
   "meltano.core.project_add_service",
   "meltano.core.project_files",
   "meltano.core.project_plugins_service",

diff --git a/src/meltano/core/locked_definition_service.py b/src/meltano/core/locked_definition_service.py
@@ -45,7 +45,11 @@ def find_definition(
         Raises:
             PluginNotFoundError: If the plugin definition could not be found.
         """
-        path = self.project.plugin_lock_path(plugin_type, plugin_name, variant_name)
+        path = self.project.plugin_lock_path(
+            plugin_type,
+            plugin_name,
+            variant_name=variant_name,
+        )
         try:
             standalone = StandalonePlugin.parse_json_file(path)
         except FileNotFoundError as err:

diff --git a/src/meltano/core/logging/job_logging_service.py b/src/meltano/core/logging/job_logging_service.py
@@ -8,6 +8,8 @@
 from meltano.core.utils import makedirs, slugify
 
 if t.TYPE_CHECKING:
+    from pathlib import Path
+
     from meltano.core.project import Project
 
 MAX_FILE_SIZE = 2097152  # 2MB max
@@ -26,16 +28,35 @@ def __init__(self, project: Project):
         self.project = project
 
     @makedirs
-    def logs_dir(self, state_id, *joinpaths):
-        return self.project.job_logs_dir(state_id, *joinpaths)
+    def logs_dir(self, state_id, *joinpaths, make_dirs: bool = True):  # noqa: ARG002
+        """Return the logs directory for a given state_id.
+
+        Args:
+            state_id: The state ID for the logs.
+            joinpaths: Additional paths to join to the logs directory.
+            make_dirs: Whether to create the directory if it does not exist.
+
+        Returns:
+            The logs directory for the given state ID.
+        """
+        return self.project.job_logs_dir(state_id, *joinpaths, make_dirs=make_dirs)
 
     def generate_log_name(
         self,
         state_id: str,
         run_id: str,
         file_name: str = "elt.log",
-    ) -> str:
-        """Generate an internal etl log path and name."""
+    ) -> Path:
+        """Generate an internal etl log path and name.
+
+        Args:
+            state_id: The state ID for the log.
+            run_id: The run ID for the log.
+            file_name: The name of the log file.
+
+        Returns:
+            The full path to the log file.
+        """
         return self.logs_dir(state_id, str(run_id), file_name)
 
     @contextmanager
@@ -52,9 +73,9 @@ def create_log(self, state_id, run_id, file_name="elt.log"):
                 yield log_file
         except OSError:
             # Don't stop the Job running if you can not open the log file
-            # for writting: just return /dev/null
+            # for writing: just return /dev/null
             logging.error(
-                f"Could open log file {log_file_name!r} for writting. "  # noqa: G004
+                f"Could open log file {log_file_name!r} for writing. "  # noqa: G004
                 "Using `/dev/null`",
             )
             with open(os.devnull, "w") as log_file:

diff --git a/src/meltano/core/plugin_invoker.py b/src/meltano/core/plugin_invoker.py
@@ -390,7 +390,11 @@ def env(self):  # noqa: WPS210
         if self.venv_service:
             # Switch to plugin-specific venv
             venv = VirtualEnv(
-                self.project.venvs_dir(self.plugin.type, self.plugin.name),
+                self.project.venvs_dir(
+                    self.plugin.type,
+                    self.plugin.name,
+                    make_dirs=False,
+                ),
             )
             venv_dir = str(venv.bin_dir)
             env["VIRTUAL_ENV"] = str(venv.root)

diff --git a/src/meltano/core/project.py b/src/meltano/core/project.py
@@ -12,7 +12,7 @@
 from functools import cached_property
 from pathlib import Path
 
-import fasteners
+import fasteners  # type: ignore[import-untyped]
 from dotenv import dotenv_values
 
 from meltano.core import yaml
@@ -34,6 +34,13 @@
     from meltano.core.meltano_file import MeltanoFile as MeltanoFileTypeHint
     from meltano.core.plugin.base import PluginRef
 
+    if sys.version_info < (3, 10):
+        from typing import TypeAlias
+    else:
+        from typing_extensions import TypeAlias
+
+
+StrPath: TypeAlias = t.Union[str, os.PathLike]
 
 logger = logging.getLogger(__name__)
 
@@ -71,7 +78,7 @@ class Project(Versioned):  # noqa: WPS214
 
     def __init__(
         self,
-        root: os.PathLike,
+        root: StrPath,
         environment: Environment | None = None,
         readonly: bool = False,
     ):
@@ -201,7 +208,7 @@ def activate(cls, project: Project):
             if os.name == "nt":
                 executable = Path(sys.executable).parent / "meltano.exe"
                 # Admin privileges are required to create symlinks on Windows
-                if ctypes.windll.shell32.IsUserAnAdmin():
+                if ctypes.windll.shell32.IsUserAnAdmin():  # type: ignore[attr-defined]
                     if executable.is_file():
                         project.run_dir().joinpath("bin").symlink_to(executable)
                     else:
@@ -342,7 +349,7 @@ def meltano_update(self):
 
         self.refresh()
 
-    def root_dir(self, *joinpaths):
+    def root_dir(self, *joinpaths: StrPath) -> Path:  # noqa: ARG002
         """Return the root directory of this project, optionally joined with path.
 
         Args:
@@ -354,7 +361,7 @@ def root_dir(self, *joinpaths):
         return self.root.joinpath(*joinpaths)
 
     @property
-    def meltanofile(self):
+    def meltanofile(self) -> Path:
         """Get the path to this project's meltano.yml.
 
         Returns:
@@ -363,7 +370,7 @@ def meltanofile(self):
         return self.root.joinpath("meltano.yml")
 
     @property
-    def dotenv(self):
+    def dotenv(self) -> Path:
         """Get the path to this project's .env file.
 
         Returns:
@@ -372,7 +379,7 @@ def dotenv(self):
         return self.root.joinpath(".env")
 
     @cached_property
-    def dotenv_env(self):
+    def dotenv_env(self) -> dict[str, str | None]:
         """Get values from this project's .env file.
 
         Returns:
@@ -416,122 +423,157 @@ def dotenv_update(self):
         self.refresh()
 
     @makedirs
-    def meltano_dir(self, *joinpaths):
+    def meltano_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:  # noqa: ARG002
         """Path to the project `.meltano` directory.
 
         Args:
             joinpaths: Paths to join to the `.meltano` directory.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `.meltano` dir optionally joined to given paths.
         """
         return self.sys_dir_root.joinpath(*joinpaths)
 
     @makedirs
-    def analyze_dir(self, *joinpaths):
+    def analyze_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:  # noqa: ARG002
         """Path to the project `analyze` directory.
 
         Args:
             joinpaths: Paths to join to the `analyze` directory.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `analyze` dir optionally joined to given paths.
         """
         return self.root_dir("analyze", *joinpaths)
 
     @makedirs
-    def extract_dir(self, *joinpaths):
+    def extract_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:  # noqa: ARG002
         """Path to the project `extract` directory.
 
         Args:
             joinpaths: Paths to join to the `extract` directory.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `extract` dir optionally joined to given paths.
         """
         return self.root_dir("extract", *joinpaths)
 
     @makedirs
-    def venvs_dir(self, *prefixes):
+    def venvs_dir(self, *prefixes: StrPath, make_dirs: bool = True) -> Path:
         """Path to a `venv` directory in `.meltano`.
 
         Args:
             prefixes: Paths to prepend to the `venv` directory in `.meltano`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `venv` dir optionally prepended with given prefixes.
         """
-        return self.meltano_dir(*prefixes, "venv")
+        return self.meltano_dir(*prefixes, "venv", make_dirs=make_dirs)
 
     @makedirs
-    def run_dir(self, *joinpaths):
+    def run_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:
         """Path to the `run` directory in `.meltano`.
 
         Args:
             joinpaths: Paths to join to the `run` directory in `.meltano`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `run` dir optionally joined to given paths.
         """
-        return self.meltano_dir("run", *joinpaths)
+        return self.meltano_dir("run", *joinpaths, make_dirs=make_dirs)
 
     @makedirs
-    def logs_dir(self, *joinpaths):
+    def logs_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:
         """Path to the `logs` directory in `.meltano`.
 
         Args:
             joinpaths: Paths to join to the `logs` directory in `.meltano`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `logs` dir optionally joined to given paths.
         """
-        return self.meltano_dir("logs", *joinpaths)
+        return self.meltano_dir("logs", *joinpaths, make_dirs=make_dirs)
 
     @makedirs
-    def job_dir(self, state_id, *joinpaths):
+    def job_dir(self, state_id, *joinpaths: StrPath, make_dirs: bool = True) -> Path:
         """Path to the `elt` directory in `.meltano/run`.
 
         Args:
             state_id: State ID of `run` dir.
             joinpaths: Paths to join to the `elt` directory in `.meltano`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `elt` dir optionally joined to given paths.
         """
-        return self.run_dir("elt", sanitize_filename(state_id), *joinpaths)
+        return self.run_dir(
+            "elt",
+            sanitize_filename(state_id),
+            *joinpaths,
+            make_dirs=make_dirs,
+        )
 
     @makedirs
-    def job_logs_dir(self, state_id, *joinpaths):
+    def job_logs_dir(
+        self,
+        state_id,
+        *joinpaths: StrPath,
+        make_dirs: bool = True,
+    ) -> Path:
         """Path to the `elt` directory in `.meltano/logs`.
 
         Args:
             state_id: State ID of `logs` dir.
             joinpaths: Paths to join to the `elt` directory in `.meltano/logs`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to `elt` dir optionally joined to given paths.
         """
-        return self.logs_dir("elt", sanitize_filename(state_id), *joinpaths)
+        return self.logs_dir(
+            "elt",
+            sanitize_filename(state_id),
+            *joinpaths,
+            make_dirs=make_dirs,
+        )
 
     @makedirs
-    def plugin_dir(self, plugin: PluginRef, *joinpaths):
+    def plugin_dir(
+        self,
+        plugin: PluginRef,
+        *joinpaths: StrPath,
+        make_dirs: bool = True,
+    ) -> Path:
         """Path to the plugin installation directory in `.meltano`.
 
         Args:
             plugin: Plugin to retrieve or create directory for.
             joinpaths: Paths to join to the plugin installation directory in `.meltano`.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Resolved path to plugin installation dir optionally joined to given paths.
         """
-        return self.meltano_dir(plugin.type, plugin.name, *joinpaths)
+        return self.meltano_dir(
+            plugin.type,
+            plugin.name,
+            *joinpaths,
+            make_dirs=make_dirs,
+        )
 
     @makedirs
-    def root_plugins_dir(self, *joinpaths: str):
+    def root_plugins_dir(self, *joinpaths: StrPath, make_dirs: bool = True) -> Path:  # noqa: ARG002
         """Path to the project `plugins` directory.
 
         Args:
             joinpaths: Paths to join with the project `plugins` directory.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Path to the project `plugins` directory.
@@ -543,14 +585,17 @@ def plugin_lock_path(
         self,
         plugin_type: str,
         plugin_name: str,
+        *,
         variant_name: str | None = None,
+        make_dirs: bool = True,  # noqa: ARG002
     ):
         """Path to the project lock file.
 
         Args:
             plugin_type: The plugin type.
             plugin_name: The plugin name.
             variant_name: The plugin variant name.
+            make_dirs: Whether to create the directory hierarchy if it doesn't exist.
 
         Returns:
             Path to the plugin lock file.
@@ -560,7 +605,11 @@ def plugin_lock_path(
         if variant_name:
             filename = f"{filename}--{variant_name}"
 
-        return self.root_plugins_dir(plugin_type, f"{filename}.lock")
+        return self.root_plugins_dir(
+            plugin_type,
+            f"{filename}.lock",
+            make_dirs=make_dirs,
+        )
 
     def __eq__(self, other):
         """Project equivalence check.

diff --git a/src/meltano/core/utils/__init__.py b/src/meltano/core/utils/__init__.py
@@ -390,7 +390,7 @@ def makedirs(func):
     def decorate(*args, **kwargs):
         enabled = kwargs.pop("make_dirs", True)
 
-        path = func(*args, **kwargs)
+        path = func(*args, **kwargs, make_dirs=enabled)
 
         if not enabled:
             return path

diff --git a/src/meltano/core/venv_service.py b/src/meltano/core/venv_service.py
@@ -236,7 +236,7 @@ def __init__(
         self.namespace = namespace
         self.name = name
         self.venv = VirtualEnv(
-            self.project.venvs_dir(namespace, name),
+            self.project.venvs_dir(namespace, name, make_dirs=False),
             python or project.settings.get("python"),
         )
         self.plugin_fingerprint_path = self.venv.root / ".meltano_plugin_fingerprint"