Skip to content

fix: gather engine_adapters from all projects#4977

Merged
themisvaltinos merged 1 commit intoSQLMesh:mainfrom
etonlels:multi-project-engine-adapters
Jul 17, 2025
Merged

fix: gather engine_adapters from all projects#4977
themisvaltinos merged 1 commit intoSQLMesh:mainfrom
etonlels:multi-project-engine-adapters

Conversation

@etonlels
Copy link
Contributor

@etonlels etonlels commented Jul 16, 2025

Trying to use sqlmesh render with multiple projects can fail with Gateway '{gateway}' not found in the available engine adapters. This is because the Context.engine_adapters only looks at self.config instead of self.configs. This PR modifies the definition of engine_adapters to look at all projects, thereby avoiding this issue.

@etonlels etonlels force-pushed the multi-project-engine-adapters branch from 338f642 to 381e55a Compare July 16, 2025 18:03
@etonlels etonlels marked this pull request as ready for review July 16, 2025 18:25
Copy link
Contributor

@themisvaltinos themisvaltinos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution it looks good, can you also add a test for this?

For example in tests/core/test_integration.py something like:

def test_engine_adapters_multi_repo_all_gateways_gathered(copy_to_temp_path):

    paths = copy_to_temp_path("examples/multi")
    repo_1_path = f"{paths[0]}/repo_1"
    repo_2_path = f"{paths[0]}/repo_2"

    # Add an extra gateway to repo_2's config
    repo_2_config_path = f"{repo_2_path}/config.yaml"
    with open(repo_2_config_path, "r") as f:
        config_content = f.read()

    modified_config = config_content.replace(
          "default_gateway: local",
          """  extra:
    connection:
      type: duckdb
      database: extra.duckdb

default_gateway: local"""
      )

    with open(repo_2_config_path, "w") as f:
        f.write(modified_config)

    # Create context with both repos but using the repo_1 path first
    context = Context(paths=[repo_1_path, repo_2_path], gateway="memory")

    # Verify all gateways from both repos are present
    adapters = context.engine_adapters
    expected_gateways = {"local", "memory", "extra"}
    missing = expected_gateways - set(adapters.keys())
    assert not missing

@etonlels etonlels force-pushed the multi-project-engine-adapters branch from 381e55a to 38b3299 Compare July 17, 2025 00:27
@etonlels etonlels force-pushed the multi-project-engine-adapters branch from 38b3299 to 4604122 Compare July 17, 2025 00:36
@etonlels
Copy link
Contributor Author

can you also add a test for this?

Certainly. Thank you for the starting point 🙏

@themisvaltinos themisvaltinos merged commit 9a954d7 into SQLMesh:main Jul 17, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants