Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ele 1684 create an unlinked tables macro #523

Closed
wants to merge 4 commits into from
Closed
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions macros/commands/get_tables_without_model.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
{% macro get_tables_without_models(deprecated_models_path=none) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this macro you could infer that it's somewhat related to sourced, snapshots, seeds, etc.
Maybe call it something in the area of leftover or dangling?
Not super critical I suppose.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like dangling, on it

{% set model_schemas, model_tables = elementary.get_model_schemas_and_tables(deprecated_models_path) %}
{% set db_tables = elementary.get_db_tables_in_schemas(model_schemas) %}
{% for db_table in db_tables %}
{% if db_table not in model_tables %}
{% do print(db_table) %}
{% endif %}
{% endfor %}
{% endmacro %}


{% macro get_db_tables_in_schemas(schemas) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about calling it get_tables_in_schemas?
The db confused me a bit 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the thought was to separate model_tables - tables defined by models, and db_tables - tables that exist in the db, but looks like its more confusing that way

{% set tables = [] %}
{% for schema in schemas %}
{% set db_name, schema_name = schema.split('.') %}
{% set schema_relation = api.Relation.create(db_name, schema_name).without_identifier() %}
{% set relations = dbt.list_relations_without_caching(schema_relation) %}
{# list_relations_without_caching can return either a list of Relation objects or an agate depending on the adapter #}
{% if relations.append is defined %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not do relations is list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is pretty hacky, but there isnt an is list test in jinja, only an is iterable, and an agate is also iterable.
i tried to find a less hacky solution but this is the best i found

{# relations is a list of Relation objects #}
{% for relation in relations %}
{% do tables.append(schema ~ "." ~ relation.identifier) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I usually prefer using api.Relation.create as it takes care of stringifying the relation in the safest way (quoting, etc.).

{% endfor %}
{% else %}
{# relations is an agate #}
{% for relation in elementary.agate_to_dicts(relations) %}
{% do tables.append(schema ~ "." ~ relation.name) %}
{% endfor %}
{% endif %}
{% endfor %}
{% do return(tables) %}
{% endmacro %}


{% macro get_model_schemas_and_tables(deprecated_models_path=none) %}
{% set model_schemas = [] %}
{% set model_tables = [] %}

{% set relevant_nodes = [] %}
{% for model_node in graph.nodes.values() | selectattr('resource_type', '==', 'model') %}
{% if not deprecated_models_path %}
{% do relevant_nodes.append(model_node) %}
{% endif %}
{% if deprecated_models_path and not model_node.original_file_path.startswith(deprecated_models_path) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{% if deprecated_models_path and not model_node.original_file_path.startswith(deprecated_models_path) %}
{% elif not model_node.original_file_path.startswith(deprecated_models_path) %}

{% do relevant_nodes.append(model_node) %}
{% endif %}
{% endfor %}
{% for model_node in relevant_nodes %}
{% set model_schema = model_node.database ~ "." ~ model_node.schema %}
{% set model_table = model_schema ~ '.' ~ model_node.name %}
{% if model_schema not in model_schemas %}
{% do model_schemas.append(model_schema) %}
{% endif %}
{% if model_table not in model_tables %}
{% do model_tables.append(model_table) %}
{% endif %}
{% endfor %}
{% do return([model_schemas, model_tables]) %}
{% endmacro %}
Loading