Skip to content

Commit

Permalink
Merge pull request #14 from stewartbryson/stewart
Browse files Browse the repository at this point in the history
Notebook.
  • Loading branch information
stewartbryson committed Dec 1, 2023
2 parents 3956290 + aefff1b commit 9e002c8
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 3 deletions.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ Secondly, I just couldn't stand for having the `DimTrade` table be "both a dimen
This decision was made during an era when storage and compute were constrained, so in my design, I created both `DIM_TRADE` and `FACT_TRADE` tables.
Finally, I used a Medallion Lakehouse Architecture with Bronze, Silver, and Gold zones, with the logical model above materialized in the Gold zone.

# Demo Notebook
I've included a [demo Jupyter notebook](demo.ipynb) that walks through a subset of the instructions below. This notebook pairs with a presentation I give that focuses more on how Snowpark Python was used in the CLI, and also the motivation for using Dynamic Tables with dbt. But it may still be helpful.

# Using DIGen.jar to Generate Source Files
The Java program to generate the source files is downloaded by [filling out a form on the TPC-DI website](https://www.tpc.org/TPC_Documents_Current_Versions/download_programs/tools-download-request5.asp?bm_type=TPC-DI&bm_vers=1.1.0&mode=CURRENT-ONLY) and clicking a link in an email. Once unzipped, we have to make one slight change for running on macOS:

Expand Down
9 changes: 6 additions & 3 deletions demo.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@
"import json\n",
"from snowflake.snowpark.types import *\n",
"from snowflake.snowpark.functions import *\n",
"from snowflake.snowpark import Session\n",
"\n",
"# Read the credentials.json file\n",
"with open(\"credentials.json\") as jsonfile:\n",
Expand All @@ -165,7 +166,10 @@
" .builder\n",
" .configs(credentials_dict)\n",
" .create()\n",
")"
")\n",
"\n",
"# Constants\n",
"source_path = '/Users/stewartbryson/dev/tpcdi-output/Batch1'"
]
},
{
Expand All @@ -183,8 +187,7 @@
"metadata": {},
"outputs": [],
"source": [
"# File paths\n",
"source_path = '/Users/stewartbryson/dev/tpcdi-output/Batch1'\n",
"# Constants\n",
"stage_path = \"@tpcdi/Batch1\"\n",
"\n",
"# Put the file\n",
Expand Down

0 comments on commit 9e002c8

Please sign in to comment.