Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple LoadMode.AUTOMATIC from load() method in DbtGraph #1001

Open
dwreeves opened this issue May 27, 2024 · 0 comments
Open

Decouple LoadMode.AUTOMATIC from load() method in DbtGraph #1001

dwreeves opened this issue May 27, 2024 · 0 comments
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing parsing:dbt_manifest Issues, questions, or features related to dbt_manifest parsing

Comments

@dwreeves
Copy link
Collaborator

This PR is an offshoot of #895. I'm introducing it here as it is something that can be implemented reasonably well in Cosmos 1.x.

Overview

TLDR-- DbtGraph.load() actually is two things:

  • It loads the dbt graph by dispatching the load implementation
  • It implements the LoadMode.AUTOMATIC implementation

My proposal is that it should actually only do one of these things, i.e. loads the dbt graph. This means that automatic should be its own separate implementation.

See the proposed code for DbtGraph inside #895 for a full implementation of the 2.x proposal. Note that some of the suggestions for DbtGraph in #895 are proposed deprecations that are out of scope for this issue, but we'd certainly want to move in this direction. Here it is a little slimmed down:

class DbtGraph:

    load_method_mapping: dict[LoadMode, Callable[[], None]] = {}

    def __init__(
        ...
    ):
        default_load_method_mapping = {
            LoadMode.AUTOMATIC: self.load_with_automatic_method,
            LoadMode.CUSTOM: self.load_via_custom_parser,
            LoadMode.DBT_LS: self.load_via_dbt_ls,
            LoadMode.DBT_LS_FILE: self.load_via_dbt_ls_file,
            LoadMode.DBT_MANIFEST: self.load_from_dbt_manifest,
        }
        # It looks like this in case load methods are set at the class level,
        # i.e. `default_load_method_mapping` sets defaults, but does not override
        # anything a user might set at the class level.
        default_load_method_mapping.update(self.load_method_mapping)
        self.load_method_mapping = default_load_method_mapping

        self.is_loaded: bool = False

        self.nodes: dict[str, DbtNode] = {}
        self.filtered_nodes: dict[str, DbtNode] = {

    def load(
            self,
            method: LoadMode | None = None,
            reload: bool = False
    ) -> dict[str, DbtNode]:
        if not reload and self.is_loaded:
            return self.filtered_nodes
        # Load only dispatches;
        # does not implicitly have the automatic load mode inside.
        if method is None:
            method = self.render_config.load_method
        callback = self.load_method_mapping[method]
        # Before the load() method only mutated DbtGraph's state.
        # It's possible that DbtGraph should not rely on mutated state.
        return callback()

    def load_with_automatic_method(self) -> None: ...
    def load_via_custom_parser(self) -> None: ...
    def load_via_dbt_ls(self) -> None: ...
    def load_via_dbt_ls_file(self) -> None: ...
    def load_from_dbt_manifest(self) -> None: ...
@dosubot dosubot bot added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing parsing:dbt_manifest Issues, questions, or features related to dbt_manifest parsing labels May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes parsing:custom Related to custom parsing, like custom DAG parsing, custom DBT parsing, etc parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing parsing:dbt_manifest Issues, questions, or features related to dbt_manifest parsing
Projects
None yet
Development

No branches or pull requests

1 participant