Skip to content

Fix tests for data-detective-etl #27

Merged
merged 2 commits into from
Nov 29, 2021
Merged

Fix tests for data-detective-etl #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a3baae
Regenerate data for root
mebelousov Nov 29, 2021
f4520ee
Add cache for etl in github-pages-publish workflow
mebelousov Nov 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions .github/workflows/github-pages-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,27 +36,38 @@ jobs:
cd data-detective-airflow
poetry config virtualenvs.in-project true

- name: Set up cache
- name: Set up cache for airflow
uses: actions/cache@v2
id: cache
id: cache-airflow
with:
path: data-detective-airflow/.venv
key: data-detective-airflow-${{ runner.os }}-${{ hashFiles('**/data-detective-airflow/poetry.lock') }}

- name: Ensure cache is healthy
if: steps.cache.outputs.cache-hit == 'true'
- name: Ensure airflow cache is healthy
if: steps.cache-airflow.outputs.cache-hit == 'true'
run: poetry run pip --version >/dev/null 2>&1 || rm -rf data-detective-airflow/.venv

- name: Install Dependencies and parse Docstrings for data-detective-airflow
working-directory: data-detective-airflow
run: |-
cd data-detective-airflow
poetry install --no-dev
pipx install 'pydoc-markdown>=4.0.0,<5.0.0'
pydoc-markdown

- name: Set up cache for etl
uses: actions/cache@v2
id: cache-etl
with:
path: data-detective-etl/.venv
key: data-detective-etl-${{ runner.os }}-${{ hashFiles('**/data-detective-etl/poetry.lock') }}

- name: Ensure etl cache is healthy
if: steps.cache-etl.outputs.cache-hit == 'true'
run: poetry run pip --version >/dev/null 2>&1 || rm -rf data-detective-etl/.venv

- name: Install Dependencies and parse Docstrings for data-detective-etl
working-directory: data-detective-etl
run: |-
cd data-detective-etl
poetry install --no-dev
pipx install 'pydoc-markdown>=4.0.0,<5.0.0'
pydoc-markdown
Expand Down
4 changes: 2 additions & 2 deletions data-detective-etl/tests_data/dags/dd_load_dds_root/dump_root_nodes_entities.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@
"entity_name_short":null,
"entity_type":"TREE_NODE",
"json_data":{
"info":"This section contains all the available information on Data Detective\n"
"info":"This section contains all available information about Data Detective\n"
},
"loaded_by":"dd_load_dds_root",
"search_data":"urn:tree_node:root:data_detective data detective",
Expand All @@ -100,4 +100,4 @@
"urn":"urn:tree_node:root"
}
]
}
}
16 changes: 8 additions & 8 deletions data-detective-etl/tests_data/dags/dd_load_dds_root/dump_root_nodes_entities.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
| urn | entity_name | loaded_by | entity_type | json_data | entity_name_short | search_data |
|:-------------------------------------------------|:---------------|:-----------------|:--------------|:------------------------------------------------------------------------------|:--------------------|:----------------------------------------------------------------|
| urn:tree_node:root:documentation | Documentation | dd_load_dds_root | TREE_NODE | {'info': 'Service documentation\n'} | | urn:tree_node:root:documentation documentation |
| urn:tree_node:root:data_detective:logical_model | Logical Model | dd_load_dds_root | TREE_NODE | {'info': 'Description of tables in the Data Detective service\n'} | | urn:tree_node:root:data_detective:logical_model logical model |
| urn:tree_node:root:data_detective:physical_model | Physical Model | dd_load_dds_root | TREE_NODE | {'info': 'This section contains technical information on Data Detective tables'} | | urn:tree_node:root:data_detective:physical_model physical model |
| urn:tree_node:root:data_detective:dags | DAGs | dd_load_dds_root | TREE_NODE | {'info': 'This section contains information on all ETL processes in the Data Platform'} | | urn:tree_node:root:data_detective:dags dags |
| urn:tree_node:root:data_detective | Data Detective | dd_load_dds_root | TREE_NODE | {'info': 'This section contains all the available information on Data Detective\n'} | | urn:tree_node:root:data_detective data detective |
| urn:tree_node:root | root | dd_load_dds_root | TREE_NODE | {'info': 'The root entity'} | | urn:tree_node:root root |
| urn | entity_name | loaded_by | entity_type | json_data | entity_name_short | search_data |
|:-------------------------------------------------|:---------------|:-----------------|:--------------|:----------------------------------------------------------------------------------------|:--------------------|:----------------------------------------------------------------|
| urn:tree_node:root:documentation | Documentation | dd_load_dds_root | TREE_NODE | {'info': 'Service documentation\n'} | | urn:tree_node:root:documentation documentation |
| urn:tree_node:root:data_detective:logical_model | Logical Model | dd_load_dds_root | TREE_NODE | {'info': 'Description of tables in the Data Detective service\n'} | | urn:tree_node:root:data_detective:logical_model logical model |
| urn:tree_node:root:data_detective:physical_model | Physical Model | dd_load_dds_root | TREE_NODE | {'info': 'This section contains technical information on Data Detective tables'} | | urn:tree_node:root:data_detective:physical_model physical model |
| urn:tree_node:root:data_detective:dags | DAGs | dd_load_dds_root | TREE_NODE | {'info': 'This section contains information on all ETL processes in the Data Platform'} | | urn:tree_node:root:data_detective:dags dags |
| urn:tree_node:root:data_detective | Data Detective | dd_load_dds_root | TREE_NODE | {'info': 'This section contains all available information about Data Detective\n'} | | urn:tree_node:root:data_detective data detective |
| urn:tree_node:root | root | dd_load_dds_root | TREE_NODE | {'info': 'The root entity'} | | urn:tree_node:root root |
14 changes: 12 additions & 2 deletions data-detective-etl/tests_data/dags/dd_load_dds_root/upload_dds_entity.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@
"name":"tables",
"type":"string"
},
{
"name":"tags",
"type":"string"
},
{
"name":"urn",
"type":"string"
Expand All @@ -87,6 +91,7 @@
"notifications":null,
"search_data":"urn:tree_node:root:documentation documentation",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root:documentation"
},
{
Expand All @@ -107,6 +112,7 @@
"notifications":null,
"search_data":"urn:tree_node:root:data_detective:logical_model logical model",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root:data_detective:logical_model"
},
{
Expand All @@ -127,6 +133,7 @@
"notifications":null,
"search_data":"urn:tree_node:root:data_detective:physical_model physical model",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root:data_detective:physical_model"
},
{
Expand All @@ -147,6 +154,7 @@
"notifications":null,
"search_data":"urn:tree_node:root:data_detective:dags dags",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root:data_detective:dags"
},
{
Expand All @@ -158,7 +166,7 @@
"htmls":null,
"info":null,
"json_data":{
"info":"This section contains all the available information on Data Detective\n"
"info":"This section contains all available information about Data Detective\n"
},
"json_data_ui":null,
"json_system":null,
Expand All @@ -167,6 +175,7 @@
"notifications":null,
"search_data":"urn:tree_node:root:data_detective data detective",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root:data_detective"
},
{
Expand All @@ -187,7 +196,8 @@
"notifications":null,
"search_data":"urn:tree_node:root root",
"tables":null,
"tags":null,
"urn":"urn:tree_node:root"
}
]
}
}
16 changes: 8 additions & 8 deletions data-detective-etl/tests_data/dags/dd_load_dds_root/upload_dds_entity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
| urn | entity_name | loaded_by | entity_type | json_data | json_system | codes | htmls | tables | notifications | grid | json_data_ui | entity_name_short | search_data | links | info |
|:-------------------------------------------------|:---------------|:-----------------|:--------------|:------------------------------------------------------------------------------|:--------------|:--------|:--------|:---------|:----------------|:-------|:---------------|:--------------------|:----------------------------------------------------------------|:--------|:-------|
| urn:tree_node:root:documentation | Documentation | dd_load_dds_root | TREE_NODE | {'info': 'Service documentation\n'} | | | | | | | | | urn:tree_node:root:documentation documentation | | |
| urn:tree_node:root:data_detective:logical_model | Logical Model | dd_load_dds_root | TREE_NODE | {'info': 'Description of tables in the Data Detective service\n'} | | | | | | | | | urn:tree_node:root:data_detective:logical_model logical model | | |
| urn:tree_node:root:data_detective:physical_model | Physical Model | dd_load_dds_root | TREE_NODE | {'info': 'This section contains technical information on Data Detective tables'} | | | | | | | | | urn:tree_node:root:data_detective:physical_model physical model | | |
| urn:tree_node:root:data_detective:dags | DAGs | dd_load_dds_root | TREE_NODE | {'info': 'This section contains information on all ETL processes in the Data Platform'} | | | | | | | | | urn:tree_node:root:data_detective:dags dags | | |
| urn:tree_node:root:data_detective | Data Detective | dd_load_dds_root | TREE_NODE | {'info': 'This section contains all the available information on Data Detective\n'} | | | | | | | | | urn:tree_node:root:data_detective data detective | | |
| urn:tree_node:root | root | dd_load_dds_root | TREE_NODE | {'info': 'The root entity'} | | | | | | | | | urn:tree_node:root root | | |
| urn | loaded_by | entity_name | entity_type | entity_name_short | info | search_data | codes | grid | json_data | json_system | json_data_ui | htmls | links | notifications | tables | tags |
|:-------------------------------------------------|:-----------------|:---------------|:--------------|:--------------------|:-------|:----------------------------------------------------------------|:--------|:-------|:----------------------------------------------------------------------------------------|:--------------|:---------------|:--------|:--------|:----------------|:---------|:-------|
| urn:tree_node:root:documentation | dd_load_dds_root | Documentation | TREE_NODE | | | urn:tree_node:root:documentation documentation | | | {'info': 'Service documentation\n'} | | | | | | | |
| urn:tree_node:root:data_detective:logical_model | dd_load_dds_root | Logical Model | TREE_NODE | | | urn:tree_node:root:data_detective:logical_model logical model | | | {'info': 'Description of tables in the Data Detective service\n'} | | | | | | | |
| urn:tree_node:root:data_detective:physical_model | dd_load_dds_root | Physical Model | TREE_NODE | | | urn:tree_node:root:data_detective:physical_model physical model | | | {'info': 'This section contains technical information on Data Detective tables'} | | | | | | | |
| urn:tree_node:root:data_detective:dags | dd_load_dds_root | DAGs | TREE_NODE | | | urn:tree_node:root:data_detective:dags dags | | | {'info': 'This section contains information on all ETL processes in the Data Platform'} | | | | | | | |
| urn:tree_node:root:data_detective | dd_load_dds_root | Data Detective | TREE_NODE | | | urn:tree_node:root:data_detective data detective | | | {'info': 'This section contains all available information about Data Detective\n'} | | | | | | | |
| urn:tree_node:root | dd_load_dds_root | root | TREE_NODE | | | urn:tree_node:root root | | | {'info': 'The root entity'} | | | | | | | |