The following insights are available in DataPilot:
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| source_staging_model_integrity | Ensures each source has a dedicated
staging model and is not directly
joined to downstream models.
|
Manifest | None |
| downstream_source_dependence | Evaluates if downstream models
(marts or intermediates) are improperly
dependent directly on a source. This
check ensures that all downstream
models depend on staging models,
not directly on the source nodes.
|
Manifest | None |
| Duplicate_Sources | Identifies cases where multiple source
nodes in a dbt project refer to the
same database object. Ensures that each
database object is represented by a single,
unique source node.
|
Manifest | None |
| hard_coded_references | Identifies instances where SQL code
within models contains hard-coded references,
which can obscure data lineage and complicate
project maintenance.
|
Manifest | None |
| rejoining_upstream_concepts | Detects scenarios where a parent’s direct
child is also a direct child of another
one of the parent’s direct children, indicating
potential loops or unnecessary complexity
in the DAG.
|
Manifest | None |
| model_fanout | Assesses parent models to identify
high fanout scenarios, which may
indicate opportunities for more
efficient transformations in the
BI layer or better positioning
of common business logic upstream
in the data pipeline.
|
Manifest | max_fanout |
| multiple_sources_joined | Checks if a model directly joins
multiple source tables, encouraging
the use of a single staging model
per source for downstream models
to enhance data consistency
and maintainability.
|
Manifest | None |
| root_model | Identifies models without direct
parents, either sources or other
models within the dbt project.
Ensures all models can be traced
back to a source or interconnected
within the project, which is crucial
for clear data lineage and project
integrity.
|
Manifest | None |
| source_fanout | Evaluates sources for high fanout,
identifying when a single source
has a large number of direct child
models. High fanout may indicate
an overly complex or source reliant
data model, potentially introducing
risks and complicating maintenance
and scalability.
|
Manifest | max_fanout |
| staging_models_dependency | Checks whether staging models depend
on downstream models, rather than
on source or raw data models. Staging
models should ideally depend on
upstream data sources to maintain
a clear and logical data flow.
|
Manifest | None |
| staging_models_on_staging | Checks if staging models are dependent
on other staging models instead of
on source or raw data models, ensuring
that staging models are used
appropriately to maintain a clear
and logical data flow from sources
to staging.
|
Manifest | None |
| unused_sources | Identifies sources that are defined
in the project’s YML files but not
used in any models or sources. They
may have become redundant due to
model deprecation, contributing to
unnecessary complexity and clutter
in the dbt project.
|
Manifest | None |
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| chain_view_linking | Analyzes the dbt project to identify
long chains of non materialized
models (views and ephemerals).
Such long chains can result in increased
runtime for models built on top of them
due to extended computation and
memory usage.
|
Manifest | None |
| exposure_parent_bad_materialization | Evaluates the materialization types of
parent models of exposures to ensure
they rely on transformed dbt models
or metrics rather than raw sources,
and checks if these parent models are
materialized efficiently for performance
|
Manifest | None |
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| documentation_on_stale_columns | Checks for columns that are documented
in the dbt project but have been removed
from their respective models.
|
Manifest, Catalog | None |
| exposures_dependent_on_private_models | Detects if exposures in the dbt project
are dependent on private models. Recommends
using public, well documented, and
contracted models as trusted data
sources for downstream consumption.
|
Manifest | None |
| public_models_without_contracts | Identifies public models in the dbt project
that are accessible to all downstream
consumers but lack contracts specifying
data types and columns.
|
Manifest | None |
| missing_documentation | Detects columns and models that don’t
have documentation.
|
Manifest, Catalog | None |
| undocumented_public_models | Identifies models in the dbt project
that are marked as public but don’t
have documentation.
|
Manifest | None |
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| missing_primary_key_tests | Identifies dbt models in the project
that lack primary key tests, which are
crucial for ensuring data integrity
and correctness.
|
Manifest | None |
| dbt_low_test_coverage | Identifies dbt models in the project
that have tests coverage percentage
below the required threshold.
|
Manifest | min_test_coverage_percent |
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| model_directory_structure | Checks for correct placement of models
in their designated directories. Proper
directory structure is essential for ,
organization, discoverability, and maintenance
within the dbt project.
|
Manifest | None |
| model_naming_convention_check | Ensures all models adhere to a predefined
naming convention. A consistent naming
convention is crucial for clarity,
understanding of the model's purpose, and
enhancing navigation within the dbt project.
|
Manifest | None |
| source_directory_structure | Verifies if sources are correctly placed in
their designated directories. Proper directory
placement for sources is important for
organizationand easy searchability.
|
Manifest | None |
| test_directory_structure | Checks if tests are correctly placed in the
same directories as their corresponding models.
Co locating tests with models aids in
maintainability and clarity.
|
Manifest | None |
| Name | Description | Files Required | Overrides |
|---|---|---|---|
| column_descriptions_are_same | Checks if the column descriptions in the dbt
project are consistent across the project.
|
Manifest | None |
| column_name_contract | Checks if the column names in the dbt project
abide by the column name contract which
consists of a regex pattern and a series
of data types.
|
Manifest, Catalog | None |
| check_macro_args_have_desc | Checks if the macro arguments in the dbt
project have descriptions.
|
Manifest | None |
| check_macro_has_desc | Checks if the macros in the dbt project
have descriptions.
|
Manifest | None |
| check_model_has_all_columns | Checks if the models in the dbt project
have all the columns that are present in
the data catalog.
|
Manifest, Catalog | None |
| check_model_has_valid_meta_keys | Checks if the models in the dbt project
have meta keys.
|
Manifest | None |
| check_model_has_properties_file | Checks if the models in the dbt project
have a properties file.
|
Manifest | None |
| check_model_has_tests_by_name | Checks if the models in the dbt project
have tests by name.
|
Manifest | None |
| check_model_has_tests_by_type | Checks if the models in the dbt project
have tests by type.
|
Manifest | None |
| check_model_has_tests_by_group | Checks if the models in the dbt project
have tests by group.
|
Manifest | None |
| check_model_materialization_by_childs | Checks if the models in the dbt project
have materialization by a given threshold
of child models.
|
Manifest | None |
| model_name_by_folder | Checks if the models in the dbt project
abide by the model name contract which
consists of a regex pattern.
|
Manifest | None |
| check_model_parents_and_childs | Checks if the model has min/max parents
and childs.
|
Manifest | None |
| check_model_parents_database | Checks if the models in the dbt project
has parent database in whitelist and
not in blacklist.
|
Manifest | None |
| check_model_parents_schema | Checks if the models in the dbt project
has parent schema in whitelist and
not in blacklist.
|
Manifest | None |
| check_model_tags | Checks if the models in the dbt project
have tags in provided list of tags.
|
Manifest | None |
| check_source_childs | Checks if the source has min/max childs
|
Manifest | None |
| check_source_columns_have_desc | Checks if the source columns have descriptions
in the dbt project.
|
Manifest, Catalog | None |
| check_source_has_all_columns | Checks if the source has all columns
present in the data catalog.
|
Manifest, Catalog | None |
| check_source_has_freshness | Checks if the source has freshness
options.
|
Manifest | None |
| check_source_has_loader | Checks if the source has loader
|
Manifest | None |
| check_source_has_meta_keys | Checks if the source has meta keys
|
Manifest | None |
| check_source_has_tests_by_name | Checks if the source has tests by name
|
Manifest | None |
| check_source_has_tests_by_type | Checks if the source has tests by type
|
Manifest | None |
| check_source_has_tests_by_group | Checks if the source has tests by group
|
Manifest | None |
| check_source_has_tests | Checks if the source has tests
|
Manifest | None |
| check_source_table_has_desc | Checks if the source table has description
|
Manifest | None |
| check_source_tags | Checks if the source has tags
|
Manifest | None |