add exposures in load_assets_from_dbt_manifest

[sowusu-ba]

Summary & Motivation

load_assets_from_dbt_manifest fails to load dbt assets when the provided manifest.json is from a dbt project that uses exposures.
When load_assets_from_dbt_manifest is used to select nodes from dbt, exposures are not added in the Manifest. In dagster_dbt.asset_defs here, only nodes, sources and metrics are selected and then passed to dbt core's selector here. The thing is that the selector function's _is_graph_member, defined as:

def _is_graph_member(self, unique_id: UniqueId) -> bool:
        if unique_id in self.manifest.sources:
            source = self.manifest.sources[unique_id]
            return source.config.enabled
        elif unique_id in self.manifest.exposures:
            return True
        elif unique_id in self.manifest.metrics:
            metric = self.manifest.metrics[unique_id]
            return metric.config.enabled
        node = self.manifest.nodes[unique_id]
        return not node.empty and node.config.enabled

expects the manifest to have exposures, otherwise, it will try to find the exposure keys from the nodes in node = self.manifest.nodes[unique_id], which of course will lead to a keyError, like I got in my project:

KeyError: 'exposure.my_project.tableau_dash_model_a'

File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_grpc/server.py", line 227, in __init__
   self._loaded_repositories = LoadedRepositories(
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_grpc/server.py", line 101, in __init__
   loadable_targets = get_loadable_targets(
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
   else loadable_targets_from_python_file(python_file, working_directory)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
   loaded_module = load_python_file(python_file, working_directory)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
   return import_module_from_path(module_name, python_file)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path
   spec.loader.exec_module(module)
 File "<frozen importlib._bootstrap_external>", line 850, in exec_module
 File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
 File "/Users/dagster/dagster-test/repos.py", line 1, in <module>
   from repositories.bia_repo import rdmart_repository
 File "/Users/dagster/dagster-test/repositories/bia_repo/__init__.py", line 5, in <module>
   from repositories.bia_repo.assets import dbt_project_assets, dbt_resource_defs
 File "/Users/dagster/dagster-test/repositories/bia_repo/assets.py", line 7, in <module>
   dbt_project_assets = load_assets_from_dbt_manifest(
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 545, in load_assets_from_dbt_manifest
   selected_unique_ids = _select_unique_ids_from_manifest_json(manifest_json, select, exclude)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dagster_dbt/asset_defs.py", line 109, in _select_unique_ids_from_manifest_json
   selector = graph_selector.NodeSelector(graph, manifest)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dbt/graph/selector.py", line 58, in __init__
   graph_members = {
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dbt/graph/selector.py", line 59, in <setcomp>
   unique_id for unique_id in self.full_graph.nodes() if self._is_graph_member(unique_id)
 File "/Users/dagster/dagster-env/lib/python3.9/site-packages/dbt/graph/selector.py", line 169, in _is_graph_member
   node = self.manifest.nodes[unique_id]

This adds a simple fix by defining the exposures to the Manifest

How I Tested These Changes

The keyError disappeared after I added this change to my project. Also I have run make isort and make black

[vercel]

@sowusu-ba is attempting to deploy a commit to the Elementl Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
dagit-storybook	⬜️ Ignored (Inspect)		Nov 5, 2022 at 9:08AM (UTC)

[sowusu-ba]

@OwenKephart

[OwenKephart]

Hi @sowusu-ba ! Thanks for the contribution, this looks great :) Merging it in now, so it'll be in this week's release