Skip to content

Removed jafgen from pyproject.toml #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AivanF merged 4 commits into from
Apr 14, 2026
Merged

Removed jafgen from pyproject.toml #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
554a8a4
Removed jafgen from pyproject.toml
AivanF Apr 13, 2026
099bebf
Update .pre-commit-config.yaml
AivanF Apr 13, 2026
4d2a62e
Merge branch 'main' into aivan/2026-04-13-jafgen-dep
AivanF Apr 14, 2026
f637914
Update NB putput
AivanF Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ jobs:
run: pip install poetry

- name: Install dependencies
run: poetry install -E all
run: poetry install -E all --with dev

- name: Install jafgen (for notebook tests, not on PyPI)
run: poetry run pip install git+https://github.com/rossbowen/jaffle-shop-generator.git@09557a1118b000071f8171aa97d54d5029bf0f0b

- name: Lint with ruff
run: poetry run ruff check slayer/ tests/
Expand Down
2 changes: 2 additions & 0 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,7 @@ repos:
hooks:
- id: ruff
args: [--fix]
exclude: ^docs/
- id: ruff-format
args: [--check]
exclude: ^docs/
291 changes: 253 additions & 38 deletions docs/examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

219 changes: 163 additions & 56 deletions docs/examples/03_auto_ingest/auto_ingest_nb.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"\n",
"We use the **Jaffle Shop** dataset — a synthetic e-commerce schema with 7 tables and foreign key relationships between them.\n",
"\n",
"**Prerequisites:** `pip install motley-slayer[examples]`\n",
"**Prerequisites:** `pip install motley-slayer[examples]` (jafgen is installed by the cell below)\n",
"\n",
"See also: [Auto-Ingestion docs](../../concepts/ingestion.md) | [Models docs](../../concepts/models.md)\n",
"\n"
Expand All @@ -23,13 +23,43 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "install-deps",
"metadata": {
"execution": {
"iopub.execute_input": "2026-04-14T08:49:09.489326Z",
"iopub.status.busy": "2026-04-14T08:49:09.489039Z",
"iopub.status.idle": "2026-04-14T08:49:18.303430Z",
"shell.execute_reply": "2026-04-14T08:49:18.301320Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.1.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m26.0.1\u001b[0m\r\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
]
}
],
"source": [
"# Install jafgen (Jaffle Shop data generator) from a specific commit\n",
"# The released PyPI version has a bug; this pinned commit is the fix.\n",
"# This is only needed for running the tutorials — not a SLayer dependency.\n",
"!pip install -q git+https://github.com/rossbowen/jaffle-shop-generator.git@09557a1118b000071f8171aa97d54d5029bf0f0b"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "278b62f3-a93e-4adc-b8a4-238bb296b061",
"metadata": {
"execution": {
"iopub.execute_input": "2026-04-09T11:03:21.358495Z",
"iopub.status.busy": "2026-04-09T11:03:21.358312Z",
"iopub.status.idle": "2026-04-09T11:03:21.774597Z",
"shell.execute_reply": "2026-04-09T11:03:21.773684Z"
"iopub.execute_input": "2026-04-14T08:49:18.306389Z",
"iopub.status.busy": "2026-04-14T08:49:18.306088Z",
"iopub.status.idle": "2026-04-14T08:49:18.837403Z",
"shell.execute_reply": "2026-04-14T08:49:18.836436Z"
}
},
"outputs": [],
Expand Down Expand Up @@ -70,32 +100,32 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 3,
"id": "be74d939",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:33.671022570Z",
"start_time": "2026-04-09T07:24:33.577643574Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:21.777345Z",
"iopub.status.busy": "2026-04-09T11:03:21.777170Z",
"iopub.status.idle": "2026-04-09T11:03:21.803711Z",
"shell.execute_reply": "2026-04-09T11:03:21.802914Z"
"iopub.execute_input": "2026-04-14T08:49:18.839417Z",
"iopub.status.busy": "2026-04-14T08:49:18.839216Z",
"iopub.status.idle": "2026-04-14T08:49:18.859336Z",
"shell.execute_reply": "2026-04-14T08:49:18.858493Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" customers: 1,937 rows\n",
" customers: 1,902 rows\n",
" stores: 6 rows\n",
" products: 10 rows\n",
" orders: 206,385 rows\n",
" order_items: 324,415 rows\n",
" orders: 206,136 rows\n",
" order_items: 326,064 rows\n",
" supplies: 65 rows\n",
" tweets: 96,663 rows\n"
" tweets: 99,231 rows\n"
]
}
],
Expand Down Expand Up @@ -123,18 +153,18 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"id": "8dbc31e7",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:43.762997780Z",
"start_time": "2026-04-09T07:24:43.465853840Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:21.805973Z",
"iopub.status.busy": "2026-04-09T11:03:21.805774Z",
"iopub.status.idle": "2026-04-09T11:03:22.618674Z",
"shell.execute_reply": "2026-04-09T11:03:22.617719Z"
"iopub.execute_input": "2026-04-14T08:49:18.861553Z",
"iopub.status.busy": "2026-04-14T08:49:18.861362Z",
"iopub.status.idle": "2026-04-14T08:49:19.131997Z",
"shell.execute_reply": "2026-04-14T08:49:19.131007Z"
}
},
"outputs": [
Expand All @@ -149,7 +179,6 @@
" tweets -> ['customers']\n",
"\n",
"Tables with no FK references (leaf tables):\n",
" customer_segments\n",
" customers\n",
" products\n",
" stores\n"
Expand Down Expand Up @@ -181,18 +210,18 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"id": "ab458329",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:48.998821658Z",
"start_time": "2026-04-09T07:24:48.952659700Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:22.621851Z",
"iopub.status.busy": "2026-04-09T11:03:22.621482Z",
"iopub.status.idle": "2026-04-09T11:03:22.633001Z",
"shell.execute_reply": "2026-04-09T11:03:22.631729Z"
"iopub.execute_input": "2026-04-14T08:49:19.133802Z",
"iopub.status.busy": "2026-04-14T08:49:19.133636Z",
"iopub.status.idle": "2026-04-14T08:49:19.139143Z",
"shell.execute_reply": "2026-04-14T08:49:19.138298Z"
}
},
"outputs": [
Expand Down Expand Up @@ -238,18 +267,18 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"id": "857ffb0f",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:52.643062223Z",
"start_time": "2026-04-09T07:24:52.610248971Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:22.636667Z",
"iopub.status.busy": "2026-04-09T11:03:22.636407Z",
"iopub.status.idle": "2026-04-09T11:03:22.642312Z",
"shell.execute_reply": "2026-04-09T11:03:22.641491Z"
"iopub.execute_input": "2026-04-14T08:49:19.140722Z",
"iopub.status.busy": "2026-04-14T08:49:19.140564Z",
"iopub.status.idle": "2026-04-14T08:49:19.144538Z",
"shell.execute_reply": "2026-04-14T08:49:19.143772Z"
}
},
"outputs": [
Expand Down Expand Up @@ -290,22 +319,28 @@
"cell_type": "markdown",
"id": "d4d8def0",
"metadata": {},
"source": "## Step 3: Dimension & Measure Generation\n\nFor each table, SLayer generates:\n- **Dimensions** for every column. See the [joins](../05_joins/joins.ipynb) on how to refer to joined dimensions and measures. \n- **Measures**: `count` always; one measure per numeric non-ID column (with `sql` set to the column name); non-numeric non-ID columns also get a measure"
"source": [
"## Step 3: Dimension & Measure Generation\n",
"\n",
"For each table, SLayer generates:\n",
"- **Dimensions** for every column. See the [joins](../05_joins/joins.ipynb) on how to refer to joined dimensions and measures. \n",
"- **Measures**: `count` always; one measure per numeric non-ID column (with `sql` set to the column name); non-numeric non-ID columns also get a measure"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 7,
"id": "a1f7133e",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:24.336258052Z",
"start_time": "2026-04-09T07:24:24.225181693Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:22.645087Z",
"iopub.status.busy": "2026-04-09T11:03:22.644886Z",
"iopub.status.idle": "2026-04-09T11:03:22.649071Z",
"shell.execute_reply": "2026-04-09T11:03:22.648179Z"
"iopub.execute_input": "2026-04-14T08:49:19.146132Z",
"iopub.status.busy": "2026-04-14T08:49:19.145974Z",
"iopub.status.idle": "2026-04-14T08:49:19.149530Z",
"shell.execute_reply": "2026-04-14T08:49:19.148813Z"
}
},
"outputs": [
Expand All @@ -319,10 +354,7 @@
"id id string [PK]\n",
"customer_id customer_id string \n",
"ordered_at ordered_at date \n",
"store_id store_id string \n",
"subtotal subtotal number \n",
"tax_paid tax_paid number \n",
"order_total order_total number \n"
"store_id store_id string \n"
]
}
],
Expand All @@ -337,22 +369,43 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"id": "798d180a",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:24.444718425Z",
"start_time": "2026-04-09T07:24:24.412200700Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:22.652391Z",
"iopub.status.busy": "2026-04-09T11:03:22.651864Z",
"iopub.status.idle": "2026-04-09T11:03:22.658715Z",
"shell.execute_reply": "2026-04-09T11:03:22.657932Z"
"iopub.execute_input": "2026-04-14T08:49:19.151050Z",
"iopub.status.busy": "2026-04-14T08:49:19.150891Z",
"iopub.status.idle": "2026-04-14T08:49:19.154125Z",
"shell.execute_reply": "2026-04-14T08:49:19.153386Z"
}
},
"outputs": [],
"source": "print(\"=== orders model measures ===\")\nprint(f\"{'Name':<30} {'SQL':<20}\")\nprint(\"-\" * 52)\nfor m in orders_model.measures:\n sql_str = m.sql or \"(COUNT(*))\"\n print(f\"{m.name:<30} {sql_str:<20}\")"
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== orders model measures ===\n",
"Name SQL \n",
"----------------------------------------------------\n",
"subtotal subtotal \n",
"tax_paid tax_paid \n",
"order_total order_total \n",
"ordered_at ordered_at \n"
]
}
],
"source": [
"print(\"=== orders model measures ===\")\n",
"print(f\"{'Name':<30} {'SQL':<20}\")\n",
"print(\"-\" * 52)\n",
"for m in orders_model.measures:\n",
" sql_str = m.sql or \"(COUNT(*))\"\n",
" print(f\"{m.name:<30} {sql_str:<20}\")"
]
},
{
"cell_type": "markdown",
Expand All @@ -366,22 +419,76 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 9,
"id": "744d7664",
"metadata": {
"ExecuteTime": {
"end_time": "2026-04-09T07:24:24.404327196Z",
"start_time": "2026-04-09T07:24:24.352857499Z"
},
"execution": {
"iopub.execute_input": "2026-04-09T11:03:22.661266Z",
"iopub.status.busy": "2026-04-09T11:03:22.661059Z",
"iopub.status.idle": "2026-04-09T11:03:22.858734Z",
"shell.execute_reply": "2026-04-09T11:03:22.857748Z"
"iopub.execute_input": "2026-04-14T08:49:19.155634Z",
"iopub.status.busy": "2026-04-14T08:49:19.155479Z",
"iopub.status.idle": "2026-04-14T08:49:19.280002Z",
"shell.execute_reply": "2026-04-14T08:49:19.279122Z"
}
},
"outputs": [],
"source": "# 1-hop: order_items -> products\nresult = engine.execute(query={\n \"source_model\": \"order_items\",\n \"fields\": [\"*:count\", \"quantity:sum\"],\n \"dimensions\": [\"products.name\"],\n \"order\": [{\"column\": {\"name\": \"quantity_sum\"}, \"direction\": \"desc\"}],\n \"limit\": 5,\n})\n\nprint(\"=== 1-hop: order_items -> products ===\")\nfor row in result.data:\n print(f\" {row['order_items.products.name']}: {row['order_items._count']} items, {row['order_items.quantity_sum']} qty\")\n\n# Multi-hop: order_items -> orders -> customers\nresult = engine.execute(query={\n \"source_model\": \"order_items\",\n \"fields\": [\"*:count\", \"quantity:sum\"],\n \"dimensions\": [\"orders.customers.name\"],\n \"order\": [{\"column\": {\"name\": \"quantity_sum\"}, \"direction\": \"desc\"}],\n \"limit\": 5,\n})\n\nprint(\"\\n=== Multi-hop: order_items -> orders -> customers ===\")\nfor row in result.data:\n print(f\" {row['order_items.orders.customers.name']}: {row['order_items._count']} items, {row['order_items.quantity_sum']} qty\")"
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== 1-hop: order_items -> products ===\n",
" adele-ade: 46265 items, 48091 qty\n",
" for richer or pourover : 46201 items, 48052 qty\n",
" chai and mighty: 46215 items, 47996 qty\n",
" tangaroo: 46064 items, 47943 qty\n",
" vanilla ice: 45813 items, 47603 qty\n",
"\n",
"=== Multi-hop: order_items -> orders -> customers ===\n",
" David Evans: 943 items, 962 qty\n",
" Ryan Duarte: 852 items, 852 qty\n",
" Andrea Thompson: 835 items, 846 qty\n",
" Robert Good: 839 items, 839 qty\n",
" Theresa Fox: 822 items, 822 qty\n"
]
}
],
"source": [
"# 1-hop: order_items -> products\n",
"result = engine.execute(\n",
" query={\n",
" \"source_model\": \"order_items\",\n",
" \"fields\": [\"*:count\", \"quantity:sum\"],\n",
" \"dimensions\": [\"products.name\"],\n",
" \"order\": [{\"column\": {\"name\": \"quantity_sum\"}, \"direction\": \"desc\"}],\n",
" \"limit\": 5,\n",
" }\n",
")\n",
"\n",
"print(\"=== 1-hop: order_items -> products ===\")\n",
"for row in result.data:\n",
" print(\n",
" f\" {row['order_items.products.name']}: {row['order_items._count']} items, {row['order_items.quantity_sum']} qty\"\n",
" )\n",
"\n",
"# Multi-hop: order_items -> orders -> customers\n",
"result = engine.execute(\n",
" query={\n",
" \"source_model\": \"order_items\",\n",
" \"fields\": [\"*:count\", \"quantity:sum\"],\n",
" \"dimensions\": [\"orders.customers.name\"],\n",
" \"order\": [{\"column\": {\"name\": \"quantity_sum\"}, \"direction\": \"desc\"}],\n",
" \"limit\": 5,\n",
" }\n",
")\n",
"\n",
"print(\"\\n=== Multi-hop: order_items -> orders -> customers ===\")\n",
"for row in result.data:\n",
" print(\n",
" f\" {row['order_items.orders.customers.name']}: {row['order_items._count']} items, {row['order_items.quantity_sum']} qty\"\n",
" )"
]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -419,9 +526,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
"version": "3.12.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
}
Loading
Loading