feat(#1328) add guide for creating dbt models #1394

witash · 2024-05-17T08:57:41Z

Closes #1328
Closes #1447

njuguna-n · 2024-05-20T15:00:23Z

I'll complete my review tomorrow. Time got away from me a bit today.

njuguna-n

Looks good but left some minor comments.

njuguna-n · 2024-05-21T08:24:41Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+
+The url is set in the `CHT_PIPELINE_BRANCH_URL` environment variable, either in .env if using docker compose, or in values.yaml if useing kubernetes.
+
+When models are changed, CHT Sync needs to be restarted for the change to take effect; when it restarts, it automatically downloads the latest version of cht pipeline and applies than changes; for models materialized as views, they are are updated instantly, for models materialized as incmrenetal tables, the table is dropped and will be unavailable until it is rebuilt from scratch.


Suggested change

When models are changed, CHT Sync needs to be restarted for the change to take effect; when it restarts, it automatically downloads the latest version of cht pipeline and applies than changes; for models materialized as views, they are are updated instantly, for models materialized as incmrenetal tables, the table is dropped and will be unavailable until it is rebuilt from scratch.

When models are changed, CHT Sync needs to be restarted for the change to take effect; when it restarts, it automatically downloads the latest version of cht pipeline and applies the changes; for models materialized as views, they are are updated instantly, for models materialized as incremental tables, the table is dropped and will be unavailable until it is rebuilt from scratch.

njuguna-n · 2024-05-21T08:27:24Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+
+The url is set in the `CHT_PIPELINE_BRANCH_URL` environment variable, either in .env if using docker compose, or in values.yaml if useing kubernetes.
+
+When models are changed, CHT Sync needs to be restarted for the change to take effect; when it restarts, it automatically downloads the latest version of cht pipeline and applies than changes; for models materialized as views, they are are updated instantly, for models materialized as incmrenetal tables, the table is dropped and will be unavailable until it is rebuilt from scratch.


It might be worth it to add a sentence discouraging editing of the base models given how long it would take to rebuild the tables if there is already a lot of data in the database.

njuguna-n · 2024-05-21T08:31:09Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+|--|--|
+|`uuid`|Unique identifier of the record. Note this is both the primary key for this table, and a foreign key to the couchdb table.|
+|`contact_uuid`| uuid of the `contact` who made the report|
+|`parent_contact_uuid`| uuid of the parent of `contact` who made the report (usually a place)|


Having this named as contact_parent_uuid as we have it in the model currently makes it clearer.

content/en/apps/guides/data/analytics/building-dbt-models.md

njuguna-n · 2024-05-21T08:36:21Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+ - Models to contain aggregates that may be useful for dashboards or analysis
+
+### Form models
+for each form in the CHT app, create one model which selects from `data_record where form = 'theformyouwant'`, moves the fields from the `fields` json into columns, applies any transformations that would be convenient, and if necessary, indexes them.


Suggested change

for each form in the CHT app, create one model which selects from `data_record where form = 'theformyouwant'`, moves the fields from the `fields` json into columns, applies any transformations that would be convenient, and if necessary, indexes them.

for each form in the CHT app, create one model which selects from `data_record where form = 'theformyouwant'`, moves the fields from the `fields` json into columns, applies any transformations that would be convenient, and if necessary, add indexes to them.

andrablaj

@witash this is an excellent start to the DBT model documentation. Thank you for the detailed steps and the example queries.

My suggestions are mainly typo and style correction.

Considering @njuguna-n's latest findings about lambda views, we will need a separate PR for those changes. I suggest:

fixing the current styles and typos
creating a separate PR to be merged into this branch with the latests changes from Njuguna
re-doing a review.

content/en/apps/guides/data/analytics/building-dbt-models.md

Co-authored-by: Andra Blaj <andra@medic.org> Co-authored-by: njuguna-n <141340177+njuguna-n@users.noreply.github.com>

andrablaj

Some small style changes.

content/en/apps/guides/data/analytics/building-dbt-models.md

andrablaj · 2024-06-03T11:42:35Z

@njuguna-n, how do you plan to proceed with updating this doc to include the latest changes (e.g. lambda views)?

njuguna-n · 2024-06-05T09:12:05Z

@andrablaj I have added a section about the base lambda view here.

* docs: add a description about the couchdb lambda view * Update content/en/apps/guides/data/analytics/building-dbt-models.md Co-authored-by: Andra Blaj <andra@medic.org> --------- Co-authored-by: Andra Blaj <andra@medic.org>

njuguna-n

Looking good so far but we need to update some sections that have changed since this was first written. I think we should also add a section describing how to handle deleted documents using FK relationships and cascade deletes.

njuguna-n · 2024-08-21T06:51:19Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+```yml
+packages:
+  - git: "https://github.com/medic/cht-pipeline"
+    revision: "1.0.0"


Seeing as we do not have versioning on cht-pipeline yet we should suggest using the main branch and updating this once this issue is done.

yea the thing about using the main branch is they might get changes that they don't want or are not prepared for. Maybe we wait until that issue is done boefre merging this PR

@witash we now have versions in cht-pipeline.

njuguna-n · 2024-08-21T06:52:14Z

content/en/apps/guides/data/analytics/building-dbt-models.md

+
+## Setup
+
+To create application specifc models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init)  and a Github repository (it may be public or private).


Suggested change

To create application specifc models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init) and a Github repository (it may be public or private).

To create application specific models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init) and a Github repository (it may be public or private).

content/en/apps/guides/data/analytics/building-dbt-models.md

njuguna-n · 2024-08-21T07:30:11Z

content/en/apps/guides/data/analytics/building-dbt-models/cht-pipeline-er.png

This needs to be updated to include the new tables e.g. document_metadata and update the columns in the existing tables.

content/en/apps/guides/data/analytics/building-dbt-models.md

mrjones-plip

I was wondering why we need a new GH repo for dbt and Andra pointed me to these nascent docs. I understand now!

I'm suggesting some minor re-wording for clarity

content/en/apps/guides/data/analytics/building-dbt-models.md

Co-authored-by: mrjones <8253488+mrjones-plip@users.noreply.github.com> Co-authored-by: Andra Blaj <andra@medic.org> Co-authored-by: njuguna-n <141340177+njuguna-n@users.noreply.github.com>

witash · 2024-09-04T13:45:09Z

this now depends on medic/cht-pipeline#155 which contains the contact_type models, changes to person and place models, and macros

if there's no changes from review on that, this is ready for re-review

mrjones-plip

Keeping my review very narrowly scoped, trusting others are reviewing this PR more widely. As such, my two minor feedback points were addressed - ship it!

andrablaj

My comments are mostly typos and formatting-related. Good job at having this done!

content/en/apps/guides/data/analytics/building-dbt-models.md

Co-authored-by: Andra Blaj <andra@medic.org>

njuguna-n

This is looking very good. left a couple of additional comments.

content/en/apps/guides/data/analytics/building-dbt-models.md

content/en/apps/guides/data/analytics/building-dbt-models/cht-pipeline-er.png

Co-authored-by: Andra Blaj <andra@medic.org>

njuguna-n

Looks good. Great job!

andrablaj

feat(1328) add guide fro creating dbt models

ee4a940

witash changed the title ~~feat(#1328) add guide fro creating dbt models~~ feat(#1328) add guide for creating dbt models May 20, 2024

witash requested review from njuguna-n and dianabarsan May 20, 2024 12:35

njuguna-n reviewed May 21, 2024

View reviewed changes

andrablaj requested changes May 30, 2024

View reviewed changes

Apply suggestions from code review

761a2a4

Co-authored-by: Andra Blaj <andra@medic.org> Co-authored-by: njuguna-n <141340177+njuguna-n@users.noreply.github.com>

andrablaj requested changes Jun 3, 2024

View reviewed changes

andrablaj mentioned this pull request Jun 6, 2024

docs: add a description about the couchdb lambda view #1418

Merged

njuguna-n and others added 2 commits June 6, 2024 10:47

feat(#1328): changing to dbt models guide

9815b07

This was referenced Aug 20, 2024

Document a migration path from couch2pg to cht-sync and cht-pipeline medic/cht-sync#136

Closed

Update documentation for model deployment #1447

Closed

njuguna-n requested changes Aug 21, 2024

View reviewed changes

feat(#1328): typos, field changes, and add docuemnt_metadata

416cbca

njuguna-n mentioned this pull request Aug 22, 2024

docs: add couch2pg to cht-sync migration instructions #1494

Merged

mrjones-plip requested changes Aug 22, 2024

View reviewed changes

content/en/apps/guides/data/analytics/building-dbt-models.md Outdated Show resolved Hide resolved

content/en/apps/guides/data/analytics/building-dbt-models.md Outdated Show resolved Hide resolved

witash and others added 5 commits September 2, 2024 10:27

Apply suggestions from code review

fd4b6c1

Co-authored-by: mrjones <8253488+mrjones-plip@users.noreply.github.com> Co-authored-by: Andra Blaj <andra@medic.org> Co-authored-by: njuguna-n <141340177+njuguna-n@users.noreply.github.com>

feat(#1328): wording changes, note on materialization, lowercase dbt

73f624d

feat(#1328): adding contact_type and place table

4416ee7

feat(#1328): adding cht_contact_model macro example

24f6159

fix(#1328): fixing link

cb2c30a

witash requested review from andrablaj, mrjones-plip and njuguna-n September 4, 2024 13:42

mrjones-plip approved these changes Sep 4, 2024

View reviewed changes

andrablaj requested changes Sep 4, 2024

View reviewed changes

Apply suggestions from code review

c56f9ac

Co-authored-by: Andra Blaj <andra@medic.org>

njuguna-n requested changes Sep 5, 2024

View reviewed changes

witash and others added 5 commits September 6, 2024 11:35

feat(#1328): updating er png and adding dot

0e9e03a

feat(#1328): add _deleted temporary field

ae9ae43

Apply suggestions from code review

4786ca2

Co-authored-by: Andra Blaj <andra@medic.org>

Merge branch 'main' into 1328-building-dbt-models

7b59568

feat(#1328): removing outdated aggregates section and wording changes

bdceb86

njuguna-n approved these changes Sep 6, 2024

View reviewed changes

andrablaj approved these changes Sep 6, 2024

View reviewed changes

witash merged commit d903430 into main Sep 9, 2024
2 checks passed

witash deleted the 1328-building-dbt-models branch September 9, 2024 06:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(#1328) add guide for creating dbt models #1394

feat(#1328) add guide for creating dbt models #1394

witash commented May 17, 2024 •

edited by andrablaj

Loading

njuguna-n commented May 20, 2024

njuguna-n left a comment

njuguna-n May 21, 2024

njuguna-n May 21, 2024

njuguna-n May 21, 2024

njuguna-n May 21, 2024

andrablaj left a comment

andrablaj left a comment

andrablaj commented Jun 3, 2024

njuguna-n commented Jun 5, 2024

njuguna-n left a comment

njuguna-n Aug 21, 2024

witash Aug 21, 2024

njuguna-n Aug 27, 2024

njuguna-n Aug 21, 2024

njuguna-n Aug 21, 2024

mrjones-plip left a comment

witash commented Sep 4, 2024

mrjones-plip left a comment

andrablaj left a comment

njuguna-n left a comment

njuguna-n left a comment

andrablaj left a comment


		The url is set in the `CHT_PIPELINE_BRANCH_URL` environment variable, either in .env if using docker compose, or in values.yaml if useing kubernetes.

		When models are changed, CHT Sync needs to be restarted for the change to take effect; when it restarts, it automatically downloads the latest version of cht pipeline and applies than changes; for models materialized as views, they are are updated instantly, for models materialized as incmrenetal tables, the table is dropped and will be unavailable until it is rebuilt from scratch.

	for each form in the CHT app, create one model which selects from `data_record where form = 'theformyouwant'`, moves the fields from the `fields` json into columns, applies any transformations that would be convenient, and if necessary, indexes them.
	for each form in the CHT app, create one model which selects from `data_record where form = 'theformyouwant'`, moves the fields from the `fields` json into columns, applies any transformations that would be convenient, and if necessary, add indexes to them.


		## Setup

		To create application specifc models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init) and a Github repository (it may be public or private).

	To create application specifc models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init) and a Github repository (it may be public or private).
	To create application specific models, create a new dbt project as described [here](https://docs.getdbt.com/reference/commands/init) and a Github repository (it may be public or private).

feat(#1328) add guide for creating dbt models #1394

feat(#1328) add guide for creating dbt models #1394

Conversation

witash commented May 17, 2024 • edited by andrablaj Loading

njuguna-n commented May 20, 2024

njuguna-n left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrablaj left a comment

Choose a reason for hiding this comment

andrablaj left a comment

Choose a reason for hiding this comment

andrablaj commented Jun 3, 2024

njuguna-n commented Jun 5, 2024

njuguna-n left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrjones-plip left a comment

Choose a reason for hiding this comment

witash commented Sep 4, 2024

mrjones-plip left a comment

Choose a reason for hiding this comment

andrablaj left a comment

Choose a reason for hiding this comment

njuguna-n left a comment

Choose a reason for hiding this comment

njuguna-n left a comment

Choose a reason for hiding this comment

andrablaj left a comment

Choose a reason for hiding this comment

witash commented May 17, 2024 •

edited by andrablaj

Loading