Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ele 1684 create an unlinked tables macro #523

Closed
wants to merge 4 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
60 changes: 60 additions & 0 deletions macros/commands/get_dangling_tables.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{% macro get_dangling_tables(deprecated_models_path=none) %}
{% set model_schemas, model_tables = elementary.get_model_schemas_and_tables(deprecated_models_path) %}
{% set db_tables = elementary.get_tables_in_schemas(model_schemas) %}
{% set rendered_model_tables = [] %}
{% for model_table in model_tables %}
{% do rendered_model_tables.append(model_table.render()) %}
{% endfor %}
{% for db_table in db_tables %}
{% if db_table.render() not in rendered_model_tables %}
{% do print(db_table) %}
{% endif %}
{% endfor %}
{% endmacro %}


{% macro get_tables_in_schemas(schemas) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the nit-picking, but maybe call this get_all_tables_in_schemas in order to clarify that this one includes tables that are not in dbt compared to the macro below?

{% set tables = [] %}
{% for schema_relation in schemas %}
{% set relations = dbt.list_relations_without_caching(schema_relation) %}
{# list_relations_without_caching can return either a list of Relation objects or an agate depending on the adapter #}
{# Jinja has no way for checking if a variable is a list, so we check if it has the append method (method of lists) #}
{% if relations.append is defined %}
{# relations is a list of Relation objects #}
{% do tables.extend(relations) %}
{% else %}
{# relations is an agate #}
{% for relation in elementary.agate_to_dicts(relations) %}
{% set relation = api.Relation.create(schema_relation.database, schema_relation.schema, relation.name) %}
{% do tables.append(relation) %}
{% endfor %}
{% endif %}
{% endfor %}
{% do return(tables) %}
{% endmacro %}


{% macro get_model_schemas_and_tables(deprecated_models_path=none) %}
{% set model_schemas = [] %}
{% set model_tables = [] %}

{% set relevant_nodes = [] %}
{% for model_node in graph.nodes.values() | selectattr('resource_type', '==', 'model') %}
{% if not deprecated_models_path %}
{% do relevant_nodes.append(model_node) %}
{% elif not model_node.original_file_path.startswith(deprecated_models_path) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just add this as an or above?

    {% if not deprecated_models_path or not model_node.original_file_path.startswith(deprecated_models_path) %}

{% do relevant_nodes.append(model_node) %}
{% endif %}
{% endfor %}
{% for model_node in relevant_nodes %}
{% set model_schema = api.Relation.create(model_node.database, model_node.schema).without_identifier() %}
{% set model_table = api.Relation.create(model_node.database, model_node.schema, model_node.name) %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to use alias, not name.

{% if model_schema not in model_schemas %}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use a set() if you want but not critical.

{% do model_schemas.append(model_schema) %}
{% endif %}
{% if model_table not in model_tables %}
{% do model_tables.append(model_table) %}
{% endif %}
{% endfor %}
{% do return([model_schemas, model_tables]) %}
{% endmacro %}