Wasm example improvements

README.md

@@ -1,26 +1,56 @@
-[![Slack](https://img.shields.io/badge/Slack-Redpanda%20Community-blue)](https://vectorized.io/slack)
+[![Slack](https://img.shields.io/badge/Slack-Redpanda%20Community-blue)](https://redpanda.com/slack)
 
 # Redpanda Examples
 
-A collection of examples to demonstrate how to interact with Redpanda from various clients and languages. Redpanda is Kafka API compatible. Any client that works with Kafka will work, but we have tested the ones listed [here](https://vectorized.io/docs/faq#What-clients-do-you-recommend-to-use-with-Redpanda).
+A collection of examples to demonstrate how to interact with Redpanda from various clients and languages. Redpanda is Kafka API compatible. Any client that works with Kafka will work, but we have tested the ones listed [here](https://docs.redpanda.com/docs/reference/faq/#what-clients-do-you-recommend-to-use-with-redpanda).
 
-## Partner Integrations
+## Directory
+
+### Clients
+
+There are a growing number of [client](./clients) examples.
+Submit any new and/or interesting examples with a pull request!
+
+| Client | Description |
+| - | - |
+| [Javascript](./clients/js/README.md) | produce events, upload schemas to registry |
+| [Python](./clients/python/README.md) | schema registry interaction, produce events, deserialize with Avro, etc. |
+
+### Ebooks microservices payments
+
+The Ebooks folder contains a Java app built with [Quarkus](https://quarkus.io/), which is a popular Java framework that uses Redpanda to power its Kafka dev service (see [this article](https://quarkus.io/guides/kafka-dev-services) for more details).
+The Ebooks app implements a consumer which validates payments.
+
+### Spark
+
+The spark app shows how to read and write to Redpanda with Spark streaming.
+A producer writes stock activity, then a streaming app reads this data from a topic and sends it into a Spark SQL DataFrame.
+Spark then modifies the events before writing to another topic.
+A consumer app then reads these modified events.
+More details [here](./spark/scala/README.md).
+
+### WebAssembly Transforms
+
+The wasm app shows how to create a JavaScript-based module that compiles to WebAssembly, and uses an Avro schema to serialize JSON events.
+More details [here](./wasm/js/transform_avro/README.md).
+
+### Partner Integrations
 
 Below is a sample of some of our partner integrations.
 
-### [Deephaven](https://deephaven.io/)
+#### Deephaven
 
-Open-core query engine for building apps and analytics with real-time streams and batch data.  
+[Deephaven](https://deephaven.io/) is an open-core query engine for building apps and analytics with real-time streams and batch data.  
 
-#### Redpanda + Deephaven Demo
+##### Redpanda + Deephaven Demo
 
 The demo follows closely the one defined for [Materialize](https://github.com/MaterializeInc/ecommerce-demo/blob/main/README_RPM.md). We want to showcase how you can accomplish the same workflow in [Deephaven](https://github.com/deephaven-examples/deephaven-debezium-demo), with Deephaven's query engine and Redpanda's streaming capabilities.
 
 
-#### Docker-stats
+##### Docker-stats
 
 Need to monitor your local machine in real time?  This example publishes your `docker stats` to Redpanda.  Written in Python, this is an out-of-the box solution to see when your Docker containers over-use their resources. See full example here: [deephaven-examples/redpanda-docker-stats](https://github.com/deephaven-examples/redpanda-docker-stats).
 
-#### Stock Market
+##### Stock Market
 
 Want to monitor updating stock market data?  Or see how to publish multiple topics to Redpanda?  This example creates tables to see stocks from [dxFeed](https://dxfeed.com/). See full example here: [deephaven-examples/redpanda-dxfeed-financial-data](https://github.com/deephaven-examples/redpanda-dxfeed-financial-data).

clients/js/README.md

@@ -0,0 +1,27 @@
+[![Slack](https://img.shields.io/badge/Slack-Redpanda%20Community-blue)](https://redpanda.com/slack)
+
+# Javascript client
+
+This app provides scripts for various client tasks.
+
+## Install dependencies
+
+```bash
+> npm i
+```
+
+More details on each script and its associated tasks are below.
+
+## Start Redpanda
+
+```bash
+> docker-compose -f docker-compose/compose-wasm.yaml up -d
+```
+
+## producer.js
+
+The following command prints the help menu:
+
+```bash
+> node producer.js -h
+```

clients/js/package.json

@@ -4,6 +4,7 @@
     "chalk": "^5.0.1",
     "csv-parser": "^3.0.0",
     "date-fns": "^2.28.0",
-    "kafkajs": "^1.15.0"
+    "kafkajs": "^1.15.0",
+    "node-fetch": "^2.6.7"
   }
 }

wasm/compose.yaml → docker-compose/compose-wasm.yaml

@@ -20,9 +20,9 @@ services:
       redpanda_network:
         ipv4_address: 172.24.1.2
     volumes:
-      - ./redpanda.yaml:/etc/redpanda/redpanda.yaml
+      - ./redpanda-wasm.yaml:/etc/redpanda/redpanda.yaml
     ports:
-      - 18081:8081  # Schema registry port
-      - 18082:18082 # Pandaproxy port
-      - 19092:19092 # Kafka API port
-      - 19644:9644  # Prometheus and HTTP admin port
+      - 8081:8081  # Schema registry port
+      - 8082:8082 # Pandaproxy port
+      - 9092:9092 # Kafka API port
+      - 9644:9644  # Prometheus and HTTP admin port

wasm/redpanda.yaml → docker-compose/redpanda-wasm.yaml

@@ -3,28 +3,28 @@ pandaproxy:
   advertised_pandaproxy_api:
   - address: 172.24.1.2
     name: internal
-    port: 8082
+    port: 18082
   - address: 127.0.0.1
     name: external
-    port: 18082
+    port: 8082
   pandaproxy_api:
   - address: 0.0.0.0
     name: internal
-    port: 8082
+    port: 18082
   - address: 172.24.1.2
     name: external
-    port: 18082
+    port: 8082
 redpanda:
   admin:
   - address: 0.0.0.0
     port: 9644
   advertised_kafka_api:
   - address: 172.24.1.2
     name: internal
-    port: 9092
+    port: 19092
   - address: 127.0.0.1
     name: external
-    port: 19092
+    port: 9092
   advertised_rpc_api:
     address: 172.24.1.2
     port: 33145
@@ -42,10 +42,10 @@ redpanda:
   kafka_api:
   - address: 0.0.0.0
     name: internal
-    port: 9092
+    port: 19092
   - address: 172.24.1.2
     name: external
-    port: 19092
+    port: 9092
   node_id: 0
   rpc_server:
     address: 172.24.1.2

schemas/market_activity.avsc

@@ -0,0 +1,12 @@
+{
+  "name": "market_activity",
+  "type": "record",
+  "fields": [
+    { "name": "Date", "type": "string" },
+    { "name": "CloseLast", "type": "string" },
+    { "name": "Volume", "type": "string" },
+    { "name": "Open", "type": "string" },
+    { "name": "High", "type": "string" },
+    { "name": "Low", "type": "string" }
+  ]
+}

schemas/market_activity.proto

@@ -0,0 +1,10 @@
+syntax = "proto3";
+
+message market_activity {
+  string Date = 1;
+  string CloseLast = 2;
+  string Volume = 3;
+  string Open = 4;
+  string High = 5;
+  string Low = 6;
+}

wasm/js/transform_avro/README.md

@@ -1,15 +1,27 @@
 # Redpanda Data Transforms: JSON to AVRO
 
-This example JavaScript transform uses the async engine to map JSON formatted messages to an AVRO schema using `avro-js`.
+This example JavaScript transform uses the async engine to map JSON formatted messages to an AVRO schema.
+
+# Tool versions
+
+Versions for tools used in this project are listed below.
+Other versions may work, but make sure to use the following versions if you run into issues:
+
+| Tool | Version | Notes |
+| - | - | - |
+| curl | 7.77.0 |
+| docker | v20.10.13 |
+| docker-compose | v1.29.2 |
+| jq | jq-1.6 |
+| node | v17.7.2 | node >= 11.15.0 has been verified |
+| rpk | v21.11.9 | rev ad9c5af53cc2ac68170d4a4d7c6e5f25c0d17f69 |
 
 ## Start Redpanda Container
 
 To run the example locally, spin up a Redpanda node with Docker:
 
 ```bash
-cd redpanda-examples/wasm
-docker-compose up -d
-
+> docker-compose -f docker-compose/compose-wasm.yaml up -d
 [+] Running 2/2
  ⠿ Network redpanda_network  Created
  ⠿ Container redpanda        Started
@@ -20,51 +32,89 @@ docker-compose up -d
 Use `npm` to package up the JavaScript application:
 
 ```bash
-cd redpanda-examples/wasm/js/transform_avro
-
-npm install    # install js dependencies in node_modules
-npm test       # run mocha tests
-npm run build  # bundle js application with webpack 
+> cd wasm/js/transform_avro
+> npm install    # install js dependencies in node_modules
+> npm test       # run mocha tests
+> npm run build  # bundle js application with webpack
 ```
 
-Create Redpanda topic:
+## Create the topic
+
+Create the topic where the producer app will send messages.
+This step is required since the consumed topic must exist prior to deploying the wasm coprocessor.
 
 ```bash
-rpk topic create market_activity --brokers localhost:19092
-rpk topic list --brokers localhost:19092
+> rpk topic create market_activity
+TOPIC            STATUS
+market_activity  OK
 ```
 
-Deploy the transform script to Redpanda:
+List all current topics:
+```bash
+> rpk topic list
+NAME             PARTITIONS  REPLICAS
+_schemas         1           1
+market_activity  1           1
+```
+
+The `_schemas` topic is automatically generated when the schemas are uploaded to the registry.
+
+## Deploy the transform script to Redpanda
 
 ```bash
-rpk wasm deploy dist/main.js --name json2avro --description "Transforms JSON to AVRO" --brokers localhost:19092
+> rpk wasm deploy dist/main.js --name json2avro --description "Transforms JSON to AVRO"
+Deploy successful!
 ```
 
 ## Produce JSON Records and Consume AVRO Results
 
-Start consumers:
+Start two consumers (run each command below in a separate terminal):
 
 ```bash
-rpk topic consume market_activity --brokers localhost:19092
-rpk topic consume market_activity._result_ --brokers localhost:19092
+> rpk topic consume market_activity
+> rpk topic consume market_activity._avro_
 ```
 
-Start producer:
+The topic `market_activity._avro_` doesn't yet exist, but it will be automatically created once the wasm function begins consuming events from the topic `market_activity`.
+
+Start the producer (in a third terminal):
 
 ```bash
-cd redpanda-examples/clients/js
-node producer.js --brokers localhost:19092
+> cd clients/js
+> npm install
+> node producer.js -rd Date
 ```
 
+The above command will output many lines of JSON string representations of the events being sent to topic `market_activity`.
+
+Verify that you see output in the second terminal.
+This will be a string representation of the Avro-serialized events being sent to `market_activity._avro_`.
+
 ## View Coproc Log
 
+The coprocessor log on the Redpanda container shows status information and possibly errors if something went wrong.
+
 ```bash
-docker exec --user root -it redpanda /bin/bash
+> docker exec --user root -it redpanda /bin/bash
 tail -100f /var/lib/redpanda/coprocessor/logs/wasm
 ```
 
+# Cleanup
+
 ## Remove Coproc
 
+Occasionally you may want disable a deployed coprocessor, for instance when you want to deploy a new version of the transform.
+Run the following command (make sure to use the same name as your previous deploy step):
+
 ```bash
-rpk wasm remove json2avro --brokers localhost:19092
+> rpk wasm remove json2avro
+```
+
+## Stop docker compose resources
+
+```bash
+> docker-compose -f docker-compose/compose-wasm.yaml down
+[+] Running 2/2
+ ⠿ Container redpanda                       Removed   0.3s
+ ⠿ Network docker-compose_redpanda_network  Removed   0.1s
 ```

wasm/js/transform_avro/package.json

@@ -3,21 +3,26 @@
   "version": "0.0.1",
   "description": "Transforms JSON to AVRO",
   "main": "bin/index.js",
-  "bin": { "iwt": "./bin/index.js" },
+  "bin": {
+    "iwt": "./bin/index.js"
+  },
   "scripts": {
     "build": "./webpack.js",
     "test": "node_modules/mocha/bin/mocha"
   },
-  "keywords": ["inline-wasm-transform", "redpanda"],
+  "keywords": [
+    "inline-wasm-transform",
+    "redpanda"
+  ],
   "author": "",
   "license": "ISC",
   "dependencies": {
-    "@vectorizedio/wasm-api": ">=21.11.3-si-beta5",
-    "avro-js": ">=1.11.0"
+    "@vectorizedio/wasm-api": "21.12.1-wasm-beta1",
+    "avsc": "^5.7.3"
   },
   "devDependencies": {
-    "ts-loader": "8.0.4",
-    "webpack": "4.44.2",
-    "mocha": "8.1.3"
+    "mocha": "9.2.2",
+    "ts-loader": "9.2.8",
+    "webpack": "5.70.0"
   }
 }