Update docs

.pre-commit-config.yaml

@@ -0,0 +1,9 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 24.2.0
+    hooks:
+      - id: black
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort

examples/advanced_project.md

@@ -47,7 +47,7 @@ Essentially, we will create a project that has this structure:
 
 We use one `dbt_project.yml` file and one `profiles.yml` file to manage the different etl services. ETL service is a
 logically separated part of the project that has its own dbt models, seeds, and tests. Each ETL service must be run
-independently of the others, maybe event in separate Airflow instance.
+independently of the others, maybe even in separate Airflow instance.
 
 ## Cross-domain dependencies
 
@@ -144,3 +144,11 @@ config:
   dbt_target: large_dev
 ```
 
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/basic_project.md

@@ -37,4 +37,13 @@ config = Config(
     include_single_model_manual_dag=False,
     # ...
 )
-```
+```
+
+## List of Examples
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/dependencies_management.md

@@ -57,4 +57,14 @@ config:
       wait_policy: all
 ```
 
-> :warning: This setting could generate a lot of tasks in your DAG. Be cautious here! If you still want to use this, consider updating number of slots in `dbt_sensor_pool` pool.
+> :warning: This setting could generate a lot of tasks in your DAG. Be cautious here! If you still want to use this, consider updating number of slots in `dbt_sensor_pool` pool.
+
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/extras_and_scripts.md

@@ -52,3 +52,12 @@ To use it, run
 ```bash
 mini_dbt_project_generator --help
 ```
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.

examples/integration_with_other_tools.md

@@ -26,3 +26,12 @@ config = Config(
     # ...
 )
 ```
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/kubernetes_tasks.md

@@ -57,13 +57,16 @@ The possible template of the script could be like this:
 ```python
 import os
 import typer
+import pandas as pd
 from typing import Optional
 
 
 # HACK: we need to define this function so that dbt can parse the model correctly
 # here we need to define all refs that are used in the model
 def model(dbt, session):
     dbt.ref('your.model.name')
+
+    return pd.DataFrame()  # it's required to return DataFrame in dbt python models
 
 def main(
     model_config_b64: Optional[str] = os.getenv('MODEL_CONFIG_B64'),
@@ -97,3 +100,13 @@ The only difference is that you have to set up `dbt_target` parameter in config
 1. The pod will be created in the same k8s cluster.
 2. Only resource limits for pods are supported.
 3. For now, only azure authentication is supported. If you want to pass `aadpodidbinding` to the pod, use `K8sConfig` in the main `dbt-af` config (see [K8sConfig](../dbt_af/conf/config.py)).
+
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/maintenance_and_source_freshness.md

@@ -124,4 +124,13 @@ config:
 config:
   maintenance:
     persist_docs: true
-```
+```
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+4. [Manual scheduling](manual_scheduling.md): domains with manual scheduling.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.

examples/manual_scheduling.md

@@ -5,3 +5,12 @@ But in some cases, you may want to have a _manual_ scheduling for your ad-hoc do
 To enable manual scheduling, you need to use `@manual` scheduling in your dbt models.
 
 All models with `@manual` scheduling will appear in the `manual` DAG in Airflow with suffix `__manual`.
+
+## List of Examples
+1. [Basic Project](basic_project.md): a single domain, small tests, and a single target.
+2. [Advanced Project](advanced_project.md): several domains, medium and large tests, and different targets.
+3. [Dependencies management](dependencies_management.md): how to manage dependencies between models in different domains.
+5. [Maintenance and source freshness](maintenance_and_source_freshness.md): how to manage maintenance tasks and source freshness.
+6. [Kubernetes tasks](kubernetes_tasks.md): how to run dbt models in Kubernetes.
+7. [Integration with other tools](integration_with_other_tools.md): how to integrate dbt-af with other tools.
+8. [\[Preview\] Extras and scripts](extras_and_scripts.md): available extras and scripts.