Initial setup of the repository - #1
Conversation
| config-version: 2 | ||
| profile: 'oppia_analytics' | ||
|
|
||
| model-paths: ["models"] |
There was a problem hiding this comment.
@seanlip Can you please update the dbt model structure based on https://docs.google.com/document/d/1Xv8t0TPYS57iZ95HekRfKpb1WNzIlBdSxQ_ul7tMGSs/edit?tab=t.0. Thanks!
There was a problem hiding this comment.
I think this is largely done, PTAL. I can't add empty folders in github though, so perhaps someone on Product Analytics would need to do that when implementing the first end-to-end Analytics flow, after this PR goes in.
| ├── macros/ # Global reusable SQL compilation modules (e.g., surrogate keys) | ||
| ├── models/ # Core transformation layers | ||
| │ ├── stg/ # Staging: Source cleaning and 1:1 type casting | ||
| │ ├── dim/ # Dimensions: Contextual master reference tables |
There was a problem hiding this comment.
@seanlip We need to update the README to reflect the proposed dbt model structure and ensure the documentation stays aligned with the new folder organization and modeling approach.
|
Hi, @seanlip ánd @maheshkarthika. |
|
@seanlip Just following up on this update! @RicMorgado The repo folder structure should be revised as proposed. |
|
@RicMorgado @maheshkarthika PTAL. This PR doesn't set up all the directories since it doesn't seem possible to commit folders that are empty. I suggest getting this in, and then you can modify the repository further as appropriate (e.g. as part of developing the first end-to-end workflow). I also left a question in https://docs.google.com/document/d/1Xv8t0TPYS57iZ95HekRfKpb1WNzIlBdSxQ_ul7tMGSs/edit?tab=t.0, PTAL. Thanks! |
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Sets up an initial dbt-based analytics transformation repository, including baseline staging models, macros, documentation, and CI/CD workflows for PR validation and scheduled production runs.
Changes:
- Added initial dbt project configuration, example profile, and a starter web staging model with source definitions + schema tests.
- Introduced a reusable macro for surrogate key generation.
- Added GitHub Actions workflows for PR validation and weekly production execution, plus repository documentation and a simplified
.gitignore.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| profiles.yml.example | Provides a template profile for local dbt BigQuery development. |
| dbt_project.yml | Defines dbt project metadata and target-based database/schema routing. |
| models/staging/web/src_web.yml | Declares the raw web source, routed by target (dev vs prod). |
| models/staging/web/stg_web_analytics__events.sql | Adds an initial staging model for GA4 events with basic parameter extraction. |
| models/staging/web/schema.yml | Adds a starter schema test for a key column. |
| macros/generate_surrogate_key.sql | Adds a macro to generate deterministic MD5-based surrogate keys. |
| README.md | Documents repo structure and local workflow setup. |
| models/README.md | Documents modeling conventions, lifecycle expectations, and performance guidance. |
| .gitignore | Replaces a broad template with dbt- and secret-focused ignores. |
| .github/workflows/pr_validation.yml | Adds CI job to build/test dbt models on PRs to develop. |
| .github/workflows/weekly_run.yml | Adds a scheduled + manual workflow to run dbt build in production. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -0,0 +1,18 @@ | |||
| {% macro generate_surrogate_key(field_list) %} | |||
| {%- set field_expressions = [] -%} | ||
|
|
||
| {%- for field in field_list -%} | ||
| {%- do field_expressions.append("COALESCE(CAST(" ~ field ~ " AS STRING), '_null_')") -%} | ||
| {%- if not loop.last -%} | ||
| {%- do field_expressions.append("'-'") -%} | ||
| {%- endif -%} | ||
| {%- endfor -%} | ||
|
|
||
| TO_HEX(MD5(CONCAT({{ field_expressions | join(', ') }}))) |
| - name: Authenticate to GCP Natively | ||
| run: | | ||
| # Write the secret JSON token safely as a raw literal string | ||
| cat << 'EOF' > ${HOME}/gcp_key.json | ||
| ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }} | ||
| EOF | ||
|
|
||
| # Activate the service account using the pre-installed gcloud CLI | ||
| gcloud auth activate-service-account --key-file=${HOME}/gcp_key.json | ||
|
|
||
| # Set the application default credentials environment variable for dbt | ||
| echo "GOOGLE_APPLICATION_CREDENTIALS=${HOME}/gcp_key.json" >> $GITHUB_ENV |
| - name: Install Dependencies | ||
| run: | | ||
| pip install dbt-bigquery sqlfluff-templater-dbt |
| timeout_seconds: 300 | ||
| location: US | ||
| EOF | ||
|
|
Sets up initial workflows for the repository