Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Small e-commerce tutorial update #76

Merged
merged 2 commits into from
Mar 10, 2023
Merged

Small e-commerce tutorial update #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c0c70b
update ecommerce demo
chuck-alt-delete Mar 10, 2023
584367e
remove unnecessary step
chuck-alt-delete Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions examples/ecommerce.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@
"email": "internet.exampleEmail",
"city": "address.city",
"state": "address.state",
"zipcode": "address.zipCode",
"created_at": "date.past(5)"
"zipcode": "address.zipCode"
},
{
"_meta": {
Expand Down Expand Up @@ -46,7 +45,6 @@
"name": "commerce.product",
"price": "commerce.price",
"description": "commerce.productDescription",
"material": "commerce.productMaterial",
"created_at": "date.past(5)"
"material": "commerce.productMaterial"
}
]
46 changes: 34 additions & 12 deletions examples/ecommerce.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ Here is the input schema:
"email": "internet.exampleEmail",
"city": "address.city",
"state": "address.state",
"zipcode": "address.zipCode",
"created_at": "date.past(5)"
"zipcode": "address.zipCode"
},
{
"_meta": {
Expand Down Expand Up @@ -55,8 +54,7 @@ Here is the input schema:
"name": "commerce.product",
"price": "commerce.price",
"description": "commerce.productDescription",
"material": "commerce.productMaterial",
"created_at": "date.past(5)"
"material": "commerce.productMaterial"
}
]
```
Expand Down Expand Up @@ -120,10 +118,12 @@ This tutorial will use a Confluent Cloud Basic Kafka Cluster and Schema Registry
1. Connect to the database via `psql` with the connection string provided.
1. Create a `SECRET` called `confluent_kafka_password` that is your Kafka cluster API secret.
```sql
CREATE SECRET confluent_kafka_username AS '<your kafka api key>';
CREATE SECRET confluent_kafka_password AS '<your kafka api secret>';
```
1. Create a `SECRET` called `csr_password` that is your Confluent Schema Registry API secret.
```sql
CREATE SECRET csr_username AS '<your csr api key>';
CREATE SECRET csr_password AS '<your csr api secret>';
```
1. Create a `KAFKA` connection called `confluent_kafka`.
Expand All @@ -134,7 +134,7 @@ This tutorial will use a Confluent Cloud Basic Kafka Cluster and Schema Registry
KAFKA (
BROKER = 'pkc-XXXX.XXXX.aws.confluent.cloud:9092',
SASL MECHANISMS = 'PLAIN',
SASL USERNAME = '<your kafka api key>',
SASL USERNAME = SECRET confluent_kafka_username,
SASL PASSWORD = SECRET confluent_kafka_password
);
```
Expand All @@ -145,8 +145,8 @@ This tutorial will use a Confluent Cloud Basic Kafka Cluster and Schema Registry
TO
CONFLUENT SCHEMA REGISTRY (
URL 'https://psrc-XXXX.XXXX.aws.confluent.cloud',
USERNAME = '<your csr api key>',
PASSWORD = SECRET csr
USERNAME = SECRET csr_username,
PASSWORD = SECRET csr_password
);
```
1. Create a cluster called `sources` where you will run your Kafka sources.
Expand Down Expand Up @@ -245,20 +245,46 @@ Materialize specializes in efficient, incremental view maintenance over changing
```sql
SELECT * FROM "purchases_agg" WHERE 70 = ANY ("user_ids");
```
```
id | item_ids | user_ids | total
-----+-----------------------+---------------+-------
690 | {3931} | {70} | 903
762 | {3704} | {70} | 670
401 | {1396,3966} | {19,70} | 1453
704 | {2345,2956} | {1,70} | 1699
219 | {2212,4823} | {53,70} | 1799
7 | {1793,3125,1088} | {3,70,95} | 755
174 | {57,617,4768} | {65,70,85} | 2687
983 | {1301,2906,3935,2557} | {8,21,70,100} | 2023
```
> :bulb: Notice how the results are non empty! If we were generating random records, these joins would be empty because there would likely be no matches on the join conditions.
1. Subscribe to changes in purchase history for a particular user in near-real time.
```sql
COPY
(SUBSCRIBE
(SELECT
*
item_ids, user_ids, total
FROM
"purchases_agg"
WHERE
4 = ANY ("user_ids")))
TO
STDOUT;
```
```
time | diff | item_ids | user_ids | total
1678475187360 1 {2175} {4} 839
1678475187360 1 {4675,3147} {4,20} 1369
1678475187360 1 {1423,3769,501,4860} {4,17,59,74} 1818
1678475187360 1 {530,4653,670,4767,3558,501} {2,3,4,22,38,91} 2571
1678475200000 1 {2585,2442} {4,88} 414
1678475200000 1 {539,423} {4,94} 1453
1678475200000 1 {869,944} {4,29} 1087
1678475200000 1 {3678,2426,4481} {4,45,47} 789
1678475293000 -1 {2585,2442} {4,88} 414
1678475293000 1 {2585,2442} {4,88} 943
...and so on
```

## Clean up

Expand All @@ -268,10 +294,6 @@ Materialize specializes in efficient, incremental view maintenance over changing
DROP CLUSTER sources CASCADE;
DROP CLUSTER ecommerce CASCADE;
```
1. If you haven't already, drop the cluster replica `default.r1` to avoid accruing idle charges. The default cluster will still exist, and you can create replicas for it whenever you need to compute.
```sql
DROP CLUSTER REPLICA default.r1;
```
1. Run `datagen` again with the `--clean` option to destroy topics and schema subjects.
```bash
datagen \
Expand Down