diff --git a/README.md b/README.md index 190653c..e09c99b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# **Scramjet Transform Hub - Quick Start** +

Scramjet Transform Hub - Quick Start

GitHub license @@ -19,7 +19,7 @@ Scramjet Transform Hub Logo

-> :bulb: **Note**: This repository contains introductory documentation and code samples for Scramjet Transform Hub. +> 💡 **Note**: This repository contains introductory documentation and code samples for Scramjet Transform Hub. * Developers looking for source code repository should visit the following link [https://github.com/scramjetorg/transform-hub](https://github.com/scramjetorg/transform-hub). * You can also find our packages published in NPM: @@ -36,28 +36,28 @@ 2. [Installation](#2-Installation) * [Prepare environment](#21-prepare-environment) * [Install STH](#22-install-sth) -3. [Run your first sequence](#3-run-your-first-sequence) +3. [Run your first Sequence](#3-run-your-first-sequence) * [Review the package](#31-review-the-package) - * [Run the sequence](#32-run-the-sequence) + * [Run the Sequence](#32-run-the-sequence) 4. [Where to go next](#4-where-to-go-next) -5. [User's dictionary](#5-users-dictionary-book) +5. [User's dictionary](#5-users-dictionary) ## **1. Introduction** ### **1.1 What is Scramjet Transform Hub** -Scramjet Transform Hub (STH) can be treated both as data processing engine and execution platform for multiple sequences running on the same platform and performing various data processing tasks. +Scramjet Transform Hub (STH) can be treated both as data processing engine and execution platform for multiple Sequences running on the same platform and performing various data processing tasks. -STH allows you to deploy and run multiple data processing apps called sequences. +STH allows you to deploy and run multiple data processing apps called Sequences.
Sequences -**Sequences** are specific apps, not just any apps. They specialize in efficient data processing. +[**Sequences**](dictionary.md#sequence) are specific apps, not just any apps. They specialize in efficient data processing. -We named our apps "sequences" and that term describes well its nature, as they process data through a sequence of chained functions. Therefore, usually our sequences are concise, easy to write and powerful at the same time. +We named our apps "Sequences" and that term describes well its nature, as they process data through a Sequence of chained functions. Therefore, usually our Sequences are concise, easy to write and powerful at the same time.

@@ -68,18 +68,18 @@ The core part of our STH engine is called the "host". Host -**Host** is responsible for maintaining and deploying sequences, keeping them running and managing its lifecycle. +**Host** is responsible for maintaining and deploying Sequences, keeping them running and managing its lifecycle. -Host exposes also its own REST API to provide and receive data and manage sequences and host itself. +Host exposes also its own REST API to provide and receive data and manage Sequences and host itself. -What we also do on the host level is that we apply a set of algorithms to optimize and speed up data processing execution in sequences. +What we also do on the host level is that we apply a set of algorithms to optimize and speed up data processing execution in Sequences. > We call our processing optimization algorithms **"IFCA"** meaning "Intelligent Function Composition Algorithms". -You can interact with host using our dedicated STH CLI that will help you with sequences deployment, running it and monitoring. +You can interact with host using our dedicated STH CLI that will help you with Sequences deployment, running and monitoring.
-Our vanilla STH engine is based on Node.js and thus allows developers to benefit from rich ecosystem, numerous packages and solutions provided by this vibrant community. +Our vanilla STH engine is based on Node.js and thus allows developers to benefit from its rich ecosystem, numerous packages and solutions provided by this vibrant community. ### **1.2 Solution concept diagram** @@ -93,7 +93,7 @@ Our vanilla STH engine is based on Node.js and thus allows developers to benefit 1. STH can handle any input that can be handled by Node.js application. -2. You, as a developer, are free to process variety of inputs in your sequence applications, such as: Text, JSON, XML, SOAP, Audio, Video and more. +2. You, as a developer, are free to process variety of inputs in your Sequence applications, such as: Text, JSON, XML, SOAP, Audio, Video and more. 3. Inputs can be either: * Provided to STH via its REST API; or * Consumed from various local or remote sources by the app; such as: Stream, STDIN, File, API, URL @@ -108,11 +108,11 @@ Our vanilla STH engine is based on Node.js and thus allows developers to benefit This is a solution for the central processing and management unit with the following major components: -1. **Sequences** - these are the actual "STH" apps. It is a gzipped package (`*.tar.gz`) containing at least two files: +1. **Sequences** - these are the actual "STH" apps. It is a package containing at least two files: * **package.json** - JSON manifest file describing the app and its configuration; such as main file to run * **main file** - file such as `index.js` or `index.ts` that contains a lightweight application business logic. -2. **Instance** - once a sequence is run, the host will create a separate runtime environment for it and will execute sequence code inside this runtime entity. This is an instance. -3. **API & CLI** - our Application Programming Interface and CLI connecting to it allows both for **Data operations** (sending input data and receiving output data) and **Management operations** (manage host itself and its entities: sequences or instances) +2. **Instance** - once a Sequence is run, the host will create a separate runtime environment for it and will execute Sequence code inside this runtime entity. This is an Instance. +3. **API & CLI** - our Application Programming Interface and CLI connecting to it allows both for **Data operations** (sending input data and receiving output data) and **Management operations** (manage host itself and its entities: Sequences or Instances) @@ -125,8 +125,8 @@ Our engine outputs can be managed in several ways: * **File** - you can save your output to a local or a remote file * **STDOUT** - output can be directed to system STDOUT (STDERR is supported as well) -* **API** - output can be consumed from our STH REST API -* **URL Request** - you can write your app in a way to request URL, webhook etc +* **API** - output can be consumed from our STH REST API +* **URL Request** - you can write your app in a way to request URL, webhook, etc. * **Stream** - output can be streamed to a particular destination * you can mix multiple actions together: you can both send data to remote system/URL and save it locally. @@ -158,39 +158,43 @@ npm i -g @scramjet/sth @scramjet/cli scramjet-transform-hub ``` -> :point_up: **HINT:** There is also an alias for running STH: `sth` +> 💡 **HINT:** There is also an alias for running STH: +> +>```bash +>sth +>``` -More detailed installation instructions can be found in our [STH GitHub repository](https://github.com/scramjetorg/transform-hub/tree/main#installation-clamp). +More detailed installation instructions can be found in our [STH GitHub repository](https://github.com/scramjetorg/transform-hub//HEAD/#installation-clamp). -## **3. Run your first sequence** +## **3. Run your first Sequence** ### **3.1 Review the package** -Before running our first sequence let's have a quick look what's inside the sequence package. +Before running your first Sequence let's have a quick look what's inside the Sequence package. -We have prepared for you a simple JavaScript sample sequence "hello-snowman". This sequence is available in the directory `samples/hello-snowman` in this repository. In this directory you will find two files: +We have prepared for you a simple JavaScript sample Sequence "hello-snowman". This Sequence is available in the directory `samples/hello-snowman` in this repository. In this directory you will find two files: -* `package.json` - manifest file that describes this particular sequence +* `package.json` - manifest file that describes this particular Sequence * `index.js` - file containing main application logic. -This particular application is written in plain JavaScript to simplify this example. However, you can also write your sequences in TypeScript and build them before packaging and sending sequence to STH. +This particular application is written in plain JavaScript to simplify this example. However, you can also write your Sequences in TypeScript and build them before packaging and sending Sequence to STH. > In the [template's readme](templates/README.md) you will find a more specific descriptions of the particular file's content. -There is no need to change anything in our `hello-snowman` sequence for a first run. Let's move to the next step. +There is no need to change anything in our `hello-snowman` Sequence for a first run. Let's move to the next step. -### **3.2 Run the sequence** +### **3.2 Run the Sequence** -There are 4 steps to follow in order to run the example sequence: +There are 4 steps to follow in order to run the example Sequence:
- 1. Pack your sequence into a package + 1. Pack your Sequence into a package -Every "sequence" app needs to be packaged (compressed) before sending to the Transform Hub. Package is a simple TAR archive and our STH CLI has a special command to pack an app directory into a sequence tarball. +Every "Sequence" app needs to be packaged (compressed) before sending to the Transform Hub. Package is a simple TAR archive and our STH CLI has a special command to pack an app directory into a Sequence tarball. -> :bulb: **Note:** any time, you can display STH CLI help by issuing terminal command `si help` (for general help) or `si help` for specific command (ie. `si sequence help`) +> 💡 **Note:** any time, you can display STH CLI help by issuing terminal command `si help` (for general help) or `si help` for specific command (ie. `si sequence help`) Please open new terminal window (and keep the first one open with STH running). Then issue following commands in the root directory of this repository: @@ -206,7 +210,7 @@ There is no output shown in the terminal but you can verify with `ls` that tarba
- 2. Send the sequence package + 2. Send the Sequence package Send `hello-snowman.tar.gz` to the running host (default localhost API endpoint will be used by the CLI send command) by issuing following command: @@ -215,7 +219,7 @@ Send `hello-snowman.tar.gz` to the running host (default localhost API endpoint si sequence send ./samples/hello-snowman.tar.gz ``` -> :bulb: **Note:** if you receive reply: **Request ok: status: 422 Unprocessable Entity**, it means that STH Docker images are not yet pulled from DockerHub. Please wait 2-3 minutes and try to issue `si sequence send` command again. We are working on fixing this issue in the next STH release. Also, if you keep receiving docker errors you can start sth without docker: `scramjet-transform-hub --no-docker` +> 💡 **Note:** if you receive reply: **Request ok: status: 422 Unprocessable Entity**, it means that STH Docker images are not yet pulled from DockerHub. Please wait 2-3 minutes and try to issue `si sequence send` command again. We are working on fixing this issue in the next STH release. Also, if you keep receiving docker errors you can start STH without docker: `scramjet-transform-hub --no-docker` > If you encounter any problems or issues while using our platform, please visit our **[Troubleshooting](https://github.com/scramjetorg/transform-hub#troubleshooting-collision)** section, where some of the problems are already known and described. You can also log an issue/bug there. @@ -236,28 +240,37 @@ SequenceClient { } ``` -Now we have uploaded sequence to the host and host assigned to it a random ID (GUID), in this case our sequence ID is: +Now we have uploaded Sequence to the host and host assigned to it a random ID (GUID), in this case our Sequence ID is: `_id: 'cf775cc1-105b-473d-b929-6885a0c2182c'` - Host also exposes REST API endpoint for each sequence and this is also described in this response. + Host also exposes REST API endpoint for each Sequence and this is also described in this response. - Exposed sequence ID allows us to move to the next step where we will start the sequence. + Exposed Sequence ID allows us to move to the next step where we will start the Sequence.
- 3. Run the sequence + 3. Run the Sequence -We can now use sequence ID to start uploaded sequence. The command is `si seq start `. You can also pass arbitrary number of parameters by providing them after ``, in case of our `hello-snowman` no parameters are used. -Use the following command to start the sequence: +We can now use Sequence ID to start uploaded Sequence. The command is `si seq start `. To make our users life easier we provided an alias for Sequence ID: `si seq start -`. This CLI functionality replaces `-` argument with the last item the user interacted with or `select`ed. + +Also, an arbitrary number of parameters can be passed to a Sequence while `start`ing by providing them after `` or `-` alias. In case of our `hello-snowman` no parameters are used. + +Use the following command to start the Sequence: ```bash si sequence start cf775cc1-105b-473d-b929-6885a0c2182c ``` +or + +```bash +si sequence start - +``` + The output will look similar to this one: ```bash @@ -275,26 +288,26 @@ InstanceClient { } ``` -Sequence is an app template. Once it is up and running, it will become a new instance. Instance also receives its own ID (GUID). In this case instance ID is: +Sequence is an app template. Once it is up and running, it will become a new Instance. The Instance also receives its own ID (GUID). In this case the Instance ID is: `_id: 'e70222d1-acfc-4e00-b046-4a3a9481c53b'` -Of course, sequences can be run multiple times. Each run will create a separate instance with a distinct instance ID. +Of course, Sequences can be run multiple times. Each run will create a separate Instance with a distinct Instance ID.
- 4. Send data to the sequence + 4. Send data to the Sequence -We want to make your life easier and for this very example, we have prepared a special Node.js app that will generate a stream of simple messages and send them to our running instance of `hello-snowman`. +We want to make your life easier and for this very example, we have prepared a special Node.js app that will generate a stream of simple messages and send them to our running Instance of `hello-snowman`. For fun, our stream generator will send simple text messages containing temperature readings from artificial weather station. Temperature value will be generated randomly in range of <-50,50> degrees Celsius. Our `hello-snowman` app will read and interpret these messages and will inform us about state of our Snowman: -* if temperature will be 0 or below, sequence will return message: `Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️` -* in the other case (temperature above 0 degrees), sequence will return message: `Snowman ⛄ is melting! 🥵` +* if temperature will be 0 or below, Sequence will return message: `Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️` +* in the other case (temperature above 0 degrees), Sequence will return message: `Snowman ⛄ is melting! 🥵` To run this app, please execute the following command from the root of our directory `node ./tools/stream-gen-tool/stream-gen.js `. In our case this would look like this: @@ -328,12 +341,12 @@ OUTPUT| Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄ ---------------------------------------- ``` -Our sequence generator app does two things here: +Our Sequence generator app does two things here: * Sends stream of messages; each one containing number with temperature value -* Reads output from Host API that is generated by our `hello-snowman` sequences +* Reads output from Host API that is generated by our `hello-snowman` Sequence -Separately, you can also open a new terminal window and see log of this particular instance with command `si instance log `. In our case this would be: +Separately, you can also open a new terminal window and see log of this particular Instance with command `si instance log ` or by using alias `si instance log -`. In our case this would be: ```bash si instance log e70222d1-acfc-4e00-b046-4a3a9481c53b @@ -342,37 +355,92 @@ si instance log e70222d1-acfc-4e00-b046-4a3a9481c53b The sample output will be similar to this one: ```bash -... -2021-08-09T04:29:39.790Z log (object:Runner) Input message -2021-08-09T04:29:40.791Z log (object:Runner) Input message -2021-08-09T04:29:41.792Z log (object:Runner) Input message -2021-08-09T04:29:42.798Z log (object:Runner) Input message -2021-08-09T04:29:43.801Z log (object:Runner) Input message +Request ok: http://127.0.0.1:8000/api/v1/instance/e70222d1-acfc-4e00-b046-4a3a9481c53b/log status: 200 OK +{"level":"DEBUG","msg":"Streams initialized","ts":1647447631103,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"}} +{"level":"TRACE","msg":"Handshake sent","ts":1647447631103,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"}} +{"level":"DEBUG","msg":"Control message received","ts":1647447631113,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":[4000,{"appConfig":{},"args":[]}]} +{"level":"DEBUG","msg":"Handshake received","ts":1647447631113,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"}} +{"level":"DEBUG","msg":"Sequence","ts":1647447631115,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":[[null]]} +{"level":"INFO","msg":"Sequence loaded, functions count","ts":1647447631116,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":[1]} +{"level":"DEBUG","msg":"Processing function on index","ts":1647447631116,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":[0]} +{"level":"DEBUG","msg":"Function called","ts":1647447631116,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":[0]} +{"level":"INFO","msg":"All sequences processed.","ts":1647447631116,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"}} +{"level":"DEBUG","msg":"Stream type is","ts":1647447631116,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":["object"]} +{"level":"TRACE","msg":"Piping sequence output","ts":1647447631117,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":["object"]} +{"level":"DEBUG","msg":"Content-Type","ts":1647447645282,"from":"Runner","Runner":{"id":"e70222d1-acfc-4e00-b046-4a3a9481c53b"},"data":["application/octet-stream"]} ... ``` +
+ +
+ + 5. Get the Instance output + + +Once `hello-snowman` Sequence is up and running, we have also sent some input data to the Instance to consume. To see what the program does to this data use the command below, it will show you the Instance output after data transformation. Open one more terminal and paste: + +```bash +si inst output e70222d1-acfc-4e00-b046-4a3a9481c53b +``` + +or by using alias + +```bash +si inst output - +``` + +This is an example output that you should get: + +```bash +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +Snowman ⛄ is melting! 🥵 +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +``` +

-> Congratulations! :clap::clap::clap: You have run your first Scramjet Transform Hub sequence! +🎉 Congratulations! 🥳 You have run your first Scramjet Transform Hub Sequence! + +> To see more Sequence or Instance CLI commands use `si help` accordingly: +> +>* `si seq help` +> +>* `si inst help` ## **4. Where to go next** Here you can find more resources related to Scramjet Transform Hub: -* [Check out more samples](samples) :books: - we have prepared some ready-to-use apps, which you can either use as a starting point for creating your own sequences or simply run them just to see what they do, and how the STH works with them. -* [Start from our app templates](templates) :file_folder: - almost a blank file structure (package) and usage instructions, ready to be used as a starting point for building your own sequences. This is the simplest base we can provide for you to start with. -* [Contribute to STH development](https://github.com/scramjetorg/transform-hub) :construction_worker: - please feel free to contribute to STH development by submitting pull requests or creating issues. -* [Visit our Scramjet.org page](https://scramjet.org) :globe_with_meridians: - check out our website for more information about our Scramjet team, history and products. +* 📚 [Check out more samples](samples) - we have prepared some ready-to-use apps, which you can either use as a starting point for creating your own Sequences or simply run them just to see what they do, and how the STH works with them. +* 📂 [Start from our app templates](templates) - almost a blank file structure (package) and usage instructions, ready to be used as a starting point for building your own Sequences. This is the simplest base we can provide for you to start with. +* 🧑‍💻 [Contribute to STH development](https://github.com/scramjetorg/transform-hub) - please feel free to contribute to STH development by submitting pull requests or creating issues. +* 🌐 [Visit our Scramjet.org page](https://scramjet.org) - check out our website for more information about our Scramjet team, history and products. -## **5. User's dictionary :book:** +## **5. User's dictionary** There is a lot of terminology that we use in our project that may already be known to you. We have prepared a [dictionary](dictionary.md) of terms that you may find useful and which you will learn as you learn about Scramjet Platform. We try to keep the definitions short and simple. ---
-### Thank you for reading, we hope you enjoyed it. If not, here is a random cheer up joke, that may make you smile :grin: +### Thank you for reading, we hope you enjoyed it. If not, here is a random cheer up joke, that may make you smile 😁 +

diff --git a/dictionary.md b/dictionary.md index 1902812..bba3557 100644 --- a/dictionary.md +++ b/dictionary.md @@ -261,7 +261,7 @@ Software Development Kit. A collection of tools and libraries to create a softwa ### Sequence -It is a compressed package (`*.tar.gz`) containing file with a manifest(eg. `package.json`), describing the app and its configuration (such as main file to run); and a **main file** (eg. `index.js`, `app.ts`) that contains a developer's code that consists of chained functions with a lightweight application business logic . Minimal sequence consists of 1 [function](#function). +It is a package (application) containing a set of files. A file with a manifest(eg. `package.json`), describing the app and its configuration (such as main file to run); and a __main file__ (eg. `index.js`, `app.ts`) that contains a developer's code that consists of chained functions with a lightweight application business logic. Minimal Sequence consists of 1 [function](#function). ### Socket @@ -277,9 +277,7 @@ standard input/standard output. Application communication channels with the outs ### STH -### Supervisor - -An application supervising work of a single [instance](#instance). Communicates with [host](#host), returns health status of the sequence, runner, etc +shortcut for Scramjet Transform Hub ### Synchronous diff --git a/samples/README.md b/samples/README.md index 907c272..62a6adc 100644 --- a/samples/README.md +++ b/samples/README.md @@ -1,20 +1,22 @@ -# Samples :books: +# Samples -We have prepared some sample code for you to try out. Some is written in JavaScript and some in TypeScript. +Try out our samples prepared some samples for you. We differentiated them according to the language in which they were written. -> :bulb: **Please note that samples below require some previous installations before you start running them. Instructions are [here](../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that samples below require some previous installations before you start running them. Instructions are [here](../README.md#3-install-scramjet-transform-hub).** ## JavaScript samples -- [hello](hello) - Sequence that modifies incoming stream of strings by saying Hello :). +- [hello](hello) - Sequence that modifies incoming stream of strings by saying "Hello". - [hello-snowman](hello-snowman) - Sequence that reads incoming stream, and modifies it by adding a text message according to the incoming data. - [simple-counter-js](simple-counter-js) - Sequence, that counts to 1000, and logs the number in one-second intervals. -- [test-output](test-output) - Sequence that just writes random values to the output stream. +- [test-output](test-output) - Sequence that simply writes random values to the output stream. ## TypeScript samples -- [crypto-prices](crypto-prices) - Sequence that keeps printing current crypto prices for a provided pair of currencies every 1s. +- [crypto-prices](crypto-prices) - Sequence that keeps printing current crypto prices for a provided pair of currencies every 3s. +- [discord-slack-connection](discord-slack-connection) - a set of Sequences that enable communication between Slack and Discord. - [mediawiki](mediawiki) - Sequence that keeps printing mediawiki event stream. +- [rss](rss) - Sequence that gets a list of RSS and then retrieves each feed and passes links to scraper. - [scraping](scraping) - Sequence that scrapes web pages. - [stack-overflow](stack-overflow) - Sequence that gets a number of changes in Stack Overflow tag count. - [transform-string-stream](transform-string-stream) - Sequence that modifies incoming stream of strings by adding a prefix and a suffix. diff --git a/samples/crypto-prices/README.md b/samples/crypto-prices/README.md index 4cf6f41..fa2ae75 100644 --- a/samples/crypto-prices/README.md +++ b/samples/crypto-prices/README.md @@ -1,11 +1,10 @@ -## crypto-prices ![bitcoin](../../images/bitcoin1.png) ![zcash](../../images/zcash1.png) ![ethereum](../../images/etherum1.png) +# crypto-prices ![bitcoin](../../images/bitcoin1.png) ![zcash](../../images/zcash1.png) ![ethereum](../../images/etherum1.png) ---- -Sequence that keeps printing current crypto prices for a provided pair of currencies every 1s. +Sequence that keeps printing current crypto prices for a provided pair of currencies every 3 seconds. -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -35,15 +34,15 @@ si pack dist/ -o crypto-prices.tar.gz si seq send crypto-prices.tar.gz # start a sequence with currency parameters, this will output Instance ID -si seq start ETH USD +si seq start - ETH USD -# See output -si inst output +# see output +si inst output - ``` -### Output +## Output -Once you run `si inst output ` command you will get output like this one: +Once you run `si inst output -` command you should get an output similar to this one: ```bash Request ok: http://127.0.0.1:8000/api/v1/instance/87442a03-a8ca-451c-b89f-d5371774c2f3/output status: 200 OK diff --git a/samples/discord-slack-connection/README.md b/samples/discord-slack-connection/README.md index 8c73846..491725f 100644 --- a/samples/discord-slack-connection/README.md +++ b/samples/discord-slack-connection/README.md @@ -1,6 +1,8 @@ # Discord to Slack (and vice versa) Connection -In this project you will find 4 STH sequences: +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** + +In this project you will find 4 STH Sequences: * [discord-read](./discord-read/) * [discord-write](./discord-write/) @@ -9,21 +11,23 @@ In this project you will find 4 STH sequences: You must run at least two (discord-read, slack write or discord-write, slack-read) in order to get one directional communication or all four for bi-directional communication. -Each sequence either reads or writes to specified application respectively and uses `const TOPIC = ...` topic to exchange data between applications. You need two topics for bi-directional communication. +Each Sequence either reads or writes to specified application respectively and uses `const TOPIC = ...` topic to exchange data between applications. You need two topics for bi-directional communication. ## TODO As there is no way to map message IDs between the two applications, threads are not supported. -In order to get this working, sequences must keep a track of posted and written IDs. +In order to get this working, Sequences must keep a track of posted and written IDs. + +## Known Issues -## Know Issues + STH `v0.13` if installed via `npm` has an issue with `objectMode: true`. You will get an error like: ```bash -TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an instance of Buffer or Uint8Array. -Received an instance of Object +TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an Instance of Buffer or Uint8Array. +Received an Instance of Object ``` If that happens either use `v0.14` if available or rebuild and run STH from repo. Please check [transform-hub documentation](https://github.com/scramjetorg/transform-hub/#build-the-packages-building_construction) for details. diff --git a/samples/discord-slack-connection/discord-read/README.md b/samples/discord-slack-connection/discord-read/README.md index 1ff1b1b..50c8cde 100644 --- a/samples/discord-slack-connection/discord-read/README.md +++ b/samples/discord-slack-connection/discord-read/README.md @@ -1,6 +1,6 @@ # Discord Read -Read messages from discord channel and write to topic +Read messages from Discord channel and write to topic. [Discord Documentation](https://discord.js.org/#/docs/discord.js/stable/general/welcome) @@ -9,14 +9,14 @@ In order to read messages from Discord we need to create a bot first: * Go to Developer Portal and click on [Applications](https://discord.com/developers/applications) * Click on `New Application` button and give it a name. Then click on `Create`. * In the newly created and selected app, click on `Bot` and click on `Add Bot` button. -* Under `Build-A-Bot` either click on `Click to Reveal Token` link or on `Copy` button to get Discord Bot Token. *This is important!* -* You can uncheck Public Bot -* You can customize its name and icon -* Expand `OAuth2` and select `URL Generator` -* Select `bot` under scopes and `Read Messages/View Channels` under bot permissions. Copy URL -* Paste the URL into web browser address bar. Connect to Discord message will appear. Select your server from `Add To Server` dropdown and click on `Continue` -* Confirm permissions on the next screen by clicking `Authorise` button. -* You will get a confirmation saying: _you may now close this window or tab_. +* Under `Build-A-Bot` either click on `Click to Reveal Token` link or on `Copy` button to get Discord Bot Token. **This is important!** +* You can uncheck Public Bot. +* You can customize its name and icon. +* Expand `OAuth2` and select `URL Generator`. +* Select `bot` under scopes and `Read Messages/View Channels` under bot permissions. Copy URL. +* Paste the URL into web browser address bar. Connect to Discord message will appear. Select your server from `Add To Server` dropdown and click on `Continue`. +* Confirm permissions on the next screen by clicking `Authorize` button. +* You will get a confirmation saying: *you may now close this window or tab*. Create a file called `config.json` and add: @@ -37,14 +37,14 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. -si seq start +# start a Sequence +si seq start - # view messages in topic si topic get messages diff --git a/samples/discord-slack-connection/discord-write/README.md b/samples/discord-slack-connection/discord-write/README.md index 81639c4..3ae88bb 100644 --- a/samples/discord-slack-connection/discord-write/README.md +++ b/samples/discord-slack-connection/discord-write/README.md @@ -1,16 +1,14 @@ # Discord Write -Read messages from topic and write to discord via webhook +Read messages from topic and write to Discord via webhook. [Discord Documentation](https://discord.com/developers/docs/resources/webhook) ## Running -Before you start you need `DISCORD_WEBHOOK_URL`. In Discord go your server. Then next to server name click on the drop down menu and select `Server Settings` -> `Integrations` -> `Webhooks` -> `Create webhook` -> Select channel name and save changes. You can copy webhook URL by clicking on `Copy webhook URL` button. +Before start you need `DISCORD_WEBHOOK_URL`. In Discord go to your server. Then next to the server name click on the drop down menu and select `Server Settings` -> `Integrations` -> `Webhooks` -> `Create webhook` -> Select channel name and save changes. You can copy webhook URL by clicking on `Copy webhook URL` button. -WebHook URL has following format: `https://discord.com/api/webhooks/{webhook.id}/{webhook.token}` - -where `webhook.id` are numbers and `webhook.token` are letters and numbers. +WebHook URL has following format: `https://discord.com/api/webhooks/{webhook.id}/{webhook.token}` where `webhook.id` are numbers and `webhook.token` are letters and numbers. ```bash # install dependencies @@ -19,14 +17,14 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. Provide DISCORD_WEBHOOK_URL as the second parameter -si seq start +# start a Sequence, provide DISCORD_WEBHOOK_URL as the second parameter +si seq start - # view messages in topic si topic get messages diff --git a/samples/discord-slack-connection/slack-read/README.md b/samples/discord-slack-connection/slack-read/README.md index c01ac54..33230a6 100644 --- a/samples/discord-slack-connection/slack-read/README.md +++ b/samples/discord-slack-connection/slack-read/README.md @@ -21,14 +21,14 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. Provide SOCKET_MODE_TOKEN as the second parameter -si seq start +# start a Sequence, provide SOCKET_MODE_TOKEN as the second parameter +si seq start - # view messages in topic si topic get messages diff --git a/samples/discord-slack-connection/slack-write/README.md b/samples/discord-slack-connection/slack-write/README.md index b76a2ab..8dc819a 100644 --- a/samples/discord-slack-connection/slack-write/README.md +++ b/samples/discord-slack-connection/slack-write/README.md @@ -1,6 +1,6 @@ # Slack Write -Read messages from topic and write to slack. +Read messages from topic and write to Slack. In order to get SLACK_WEBHOOK_URL you need to create application in Slack first. Please refer to notes in [slack-read](../slack-read/) example. @@ -20,14 +20,14 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. Provide SLACK_WEBHOOK_URL as the second parameter -si seq start +# start a Sequence, provide SLACK_WEBHOOK_URL as the second parameter +si seq start - # view messages in topic si topic get messages diff --git a/samples/hello-snowman/README.md b/samples/hello-snowman/README.md index 5a0f840..0847295 100644 --- a/samples/hello-snowman/README.md +++ b/samples/hello-snowman/README.md @@ -1,18 +1,26 @@ -## hello-snowman :snowman: +# hello-snowman ⛄ ----- Sequence that reads incoming stream (input), and and modifies it by adding a text message according to the incoming data. -Stream is generated in [stream-gen.js](../tools/stream-gen-tool/stream-gen.js) file, where numbers in range of <-50,50> are randomly chosen and sent as Celsius degrees to `hello-snowman` instance API endpoint `/input`. +Stream is generated in [stream-gen.js](../tools/stream-gen-tool/stream-gen.js) file, where numbers in range of <-50,50> are randomly chosen and sent as Celsius degrees to `hello-snowman` Instance API endpoint `/input`. Our `hello-snowman` app will read and interpret these Celsius degrees, and will inform us about state of our Snowman: -- if temperature will be 0 or below, sequence will return message: `Snowman is freezing ... :)` -- in the other case (temperature above 0 degrees), sequence will return message: `Snowman is melting! :(` +- if temperature will be 0 or below, Sequence will return message: -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +```bash +Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️ +``` + +- in the other case (temperature above 0 degrees), Sequence will return message: + +```bash + Snowman ⛄ is melting! 🥵 +``` -### Running +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** + +## Running Open three terminals and run the following commands: @@ -32,27 +40,29 @@ cd samples/hello-snowman # install dependencies npm install -# make a compressed package with sequence +# make a compressed package with Sequence si pack . -o hello-snowman.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send hello-snowman.tar.gz -# start a sequence, this will output Instance ID -si seq start +# start a Sequence, this will output Instance ID +si seq start - -# See output of instance process -si inst output +# See output of Instance process +si inst output - ``` **In the third terminal** ```bash -# Start stream generator tool with instance ID as parameter +# Start stream generator tool with Instance ID as parameter node ./tools/stream-gen-tool/stream-gen.js ``` -### Now you should see something like this in output console: +## Output + +Now you should see the output in the console: ```js ---------------------------------------- diff --git a/samples/hello/README.md b/samples/hello/README.md index 3e63b65..6957606 100644 --- a/samples/hello/README.md +++ b/samples/hello/README.md @@ -1,11 +1,10 @@ -## hello 🙋‍♂️ +# hello 🙋‍♂️ ----- Sequence that modifies incoming stream of strings by saying Hello :). -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open three terminals and run the following commands: @@ -25,36 +24,34 @@ cd samples/hello # install dependencies npm install -# make a compressed package with sequence +# make a compressed package with Sequence si pack . -o hello.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send hello.tar.gz -# start a sequence, this will output Instance ID -si seq start +# start a Sequence, this will output I ID +si seq start - -# See output of instance process -si inst output +# See output +si inst output - ``` **The third terminal** ```bash -# Send file to instance input steam -si inst input name.txt +# Send file to Instance input steam +si inst input - # if file not given the data will be read from stdin +# type "John" +John ``` - - -### Output +## Output ```bash # Now you should see "Hello John" in output console $ si inst output 7a1ffd59-9d1a-4e8f-a246-020124803931 -Request ok: http://127.0.0.1:8000/api/v1/instance/7a1ffd59-9d1a-4e8f-a246-020124803931/output status: 200 OK +Request ok: http://127.0.0.1:8000/api/v1/I/7a1ffd59-9d1a-4e8f-a246-020124803931/output status: 200 OK Hello John ``` diff --git a/samples/hello/name.txt b/samples/hello/name.txt deleted file mode 100644 index 08df4e7..0000000 --- a/samples/hello/name.txt +++ /dev/null @@ -1 +0,0 @@ -John \ No newline at end of file diff --git a/samples/mediawiki/README.md b/samples/mediawiki/README.md index a1e7706..dad8b71 100644 --- a/samples/mediawiki/README.md +++ b/samples/mediawiki/README.md @@ -1,11 +1,10 @@ -## mediawiki +# mediawiki ---- Sequence that keeps printing mediawiki event stream. -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -28,29 +27,29 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist/ -o mediawiki.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send mediawiki.tar.gz -# start a sequence, this will output Instance ID. Search is optional and can be used to filter out results, e.g. "data.server_name === 'en.wikipedia.org'" -si seq start [] +# start a Sequence, this will output Instance ID. Search is optional and can be used to filter out results, e.g. "data.server_name === 'en.wikipedia.org'" +si seq start - [] # See output -si inst output +si inst output - # Check console.log messages -si inst stdout +si inst stdout - # Check console.error messages -si inst stderr +si inst stderr - # Send event, e.g. `drain`: `si inst emit drain "{}"` -si inst emit +si inst emit - ``` -#### Example Event +## Example Event ID diff --git a/samples/rss/README.md b/samples/rss/README.md index f16f603..cfc4480 100644 --- a/samples/rss/README.md +++ b/samples/rss/README.md @@ -1,13 +1,13 @@ -## RSS +# RSS This sample gets a list of RSS feed URLs from `./config/rss.json` file. Then it retrieves each feed and passes links to scraper. Scraper reads content of each URL. Next, keywords are matched and a score given for each URL. If score is grater than 0 it gets added to a list of links and short details are posted to slack channel. -### Configuration +## Configuration -Pass SLACK_WEBHOOK_URL as input parameter when starting sequence. +Pass SLACK_WEBHOOK_URL as input parameter when starting Sequence. Keywords are configured in `./config/keywords.json` file. Use `word` and `weight` as in the example: @@ -17,7 +17,7 @@ Keywords are configured in `./config/keywords.json` file. Use `word` and `weight RSS Feed URL as in `./config/rss.json` -### Running +## Running ```bash # install dependencies @@ -26,25 +26,25 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. Provide slack webhook URL as input parameter -si seq start +# start a Sequence, this will output Instance ID. Provide slack webhook URL as input parameter +si seq start - # See output - actual output will be send to slack channel -si inst output +si inst output - # Optional commands below: # Check console.log messages -si inst stdout +si inst stdout - # Check console.error messages -si inst stderr +si inst stderr - ``` As this is scraping content on regular basis `429 Too Many Requests` Error is inevitable. In order to mitigate this problem, increase pause between requests. diff --git a/samples/scraping/README.md b/samples/scraping/README.md index 8db518b..b0b3407 100644 --- a/samples/scraping/README.md +++ b/samples/scraping/README.md @@ -1,6 +1,5 @@ -## scraping +# scraping ---- This is a simple and trivial example of scraping web pages. The scraper takes URL and CSS ID selector as input parameters and returns data every second. @@ -8,9 +7,9 @@ The scraper takes URL and CSS ID selector as input parameters and returns data e To test this please use URL: and ID: `#ct`. Scraper will connect to the website and read (scrap) the current time. Next, it returns this as a stream. As URL and ID are parametrized we can use other websites too. For example, URL: and ID: `#clock` -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -36,28 +35,28 @@ npm run build # prepare standalone JS package cp -r node_modules package.json dist/ -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. As the CSS ID has # (hash) sign surround it with quotes: -si seq start https://www.timeanddate.com/worldclock/poland '#ct' +# start a Sequence, this will output Instance ID. As the CSS ID has # (hash) sign surround it with quotes: +si seq start - https://www.timeanddate.com/worldclock/poland '#ct' # See output -si inst output +si inst output - # Optional commands below: # Check console.log messages -si inst stdout +si inst stdout - # Check console.error messages -si inst stderr +si inst stderr - ``` -### Output +## Output ```bash $ si instance output 41783884-2e97-4b78-9639-aac5d7ff8447 diff --git a/samples/simple-counter-js/README.md b/samples/simple-counter-js/README.md index e26b47f..242a413 100644 --- a/samples/simple-counter-js/README.md +++ b/samples/simple-counter-js/README.md @@ -1,11 +1,10 @@ -## simple-counter +# simple-counter ---- -A simple sequence, that counts and logs the number in one-second intervals. As a default, the counter is started with 0 and ends with 1000. These values can be changed by passing the `start` and `end` parameters. +A simple Sequence, that counts and logs the number in one-second intervals. As a default, the counter is started with 0 and ends with 1000. These values can be changed by passing the `start` and `end` parameters. -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -25,20 +24,20 @@ cd samples/simple-counter-js # instal dependencies npm install -# make a compressed package with sequence +# make a compressed package with Sequence si pack . -o simple-counter-js.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send simple-counter-js.tar.gz -# start a sequence, this will output Instance ID -si seq start +# start a Sequence, this will output Instance ID +si seq start - -# See output of instance process -si inst stdout +# See output of Instance process +si inst stdout - ``` -### Output +## Output ```bash { x: 1 } @@ -58,7 +57,7 @@ si inst stdout ... ``` -### Running the same sequence but with different parameters +## Running the same Sequence but with some parameters ```bash # go to 'simple-counter-js' directory @@ -67,20 +66,20 @@ cd samples/simple-counter-js # instal node_modules npm install -# make a compressed package with sequence +# make a compressed package with Sequence si pack . -o simple-counter-js.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send simple-counter-js.tar.gz -# start a sequence with "start" and "end" parameters, this will output Instance ID -si seq start 100 2000 +# start a Sequence with "start" and "end" parameters, this will output Instance ID +si seq start - 100 2000 -# See output of instance process -si inst stdout +# See output +si inst stdout - ``` -### Output +## Output ```bash # the counter will start counting at 100 and finish at 200 diff --git a/samples/stack-overflow/README.md b/samples/stack-overflow/README.md index 3994145..b7d0f80 100644 --- a/samples/stack-overflow/README.md +++ b/samples/stack-overflow/README.md @@ -1,13 +1,12 @@ -## stack-overflow +# stack-overflow ---- Get number of changes in Stack Overflow tag count. This queries SO API every X minutes, gathers, compares and outputs result as difference. -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -30,28 +29,28 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send dist.tar.gz -# start a sequence, this will output Instance ID. Provide number of minutes that we pause between request to SO API. API key is optional -si seq start +# start a Sequence, this will output Instance ID. Provide number of minutes that we pause between request to SO API. API key is optional +si seq start - -# See output -si inst output +# See Instance output +si inst output - # Optional commands below: # Check console.log messages -si inst stdout +si inst stdout - # Check console.error messages -si inst stderr +si inst stderr - ``` -### Example output +## Example output ```bash {"diff":{"python":1,"c#":1,"android":1,"html":2,"jquery":1,"c++":1,"css":1,"node.js":1,"reactjs":1,".net":1,"swift":2,"xml":1,"vb.net":1,"amazon-web-services":1,"function":-1,"csv":1},"timestamp":1632986938780} diff --git a/samples/test-output/README.md b/samples/test-output/README.md index adb9952..2e486ff 100644 --- a/samples/test-output/README.md +++ b/samples/test-output/README.md @@ -1,11 +1,10 @@ -## test-output +# test-output ---- Sequence that just writes random values to output stream. > :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open two terminals and run the following commands: @@ -25,20 +24,20 @@ cd samples/test-output # install dependencies npm install -# make a compressed package with sequence +# make a compressed package with Sequence si pack . -o test-output.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send test-output.tar.gz -# start a sequence, this will output Instance ID -si seq start +# start a Sequence, this will output Instance ID +si seq start - -# See output of instance process -si inst output +# See output of Instance process +si inst output - ``` -### Output +## Output ```bash Test 6 diff --git a/samples/transform-string-stream/README.md b/samples/transform-string-stream/README.md index 62faa64..7f79c57 100644 --- a/samples/transform-string-stream/README.md +++ b/samples/transform-string-stream/README.md @@ -1,11 +1,10 @@ -## transform-string-stream +# transform-string-stream ----- Sequence that modifies incoming stream of strings by adding a prefix and a suffix. -> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** +> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).** -### Running +## Running Open three terminals and run the following commands: @@ -28,37 +27,37 @@ npm install # transpile TS->JS to dist/ npm run build -# make a compressed package with sequence +# make a compressed package with Sequence si pack dist/ -o transform-string-stream.tar.gz -# send sequence to transform hub, this will output Sequence ID +# send Sequence to transform hub, this will output Sequence ID si seq send transform-string-stream.tar.gz -# start a sequence, this will output Instance ID -si seq start Hello Bye +# start a Sequence, this will output Instance ID +si seq start - Hello Bye -# See output of instance process -si inst output +# See output of Instance process +si inst output - ``` **The third terminal** ```bash -# Send file to the instance input steam -si inst input name.txt +# Send file to the Instance input steam +si inst input - name.txt # the date will be read from the file 'name.txt' ``` or ```bash -# Send text to the instance input steam -si inst input +# Send text to the Instance input steam +si inst input - > John # if file not given the data will be read from stdin ``` -### Output +## Output ```bash # Now you should see "Hello John Bye" in output console diff --git a/templates/README.md b/templates/README.md index 921de56..cef6111 100644 --- a/templates/README.md +++ b/templates/README.md @@ -1,11 +1,11 @@ -# Templates :pencil: +# Templates We have prepared for you some templates. User templates allow you to apply predefined settings to start building your own applications. Using templates, you can easily set up multiple applications with similar settings. -You can use it as a base for your own samples 👉 [sample template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/main/templates). For this moment we support two variants of template in two programming languages: +You can use it as a base for your own samples 👉 [sample template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/HEAD/templates). For this moment we support two variants of template in two programming languages: -- JavaScript (Node.js) 👉 [template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/main/templates/template-js) -- TypeScript (ts-node) 👉 [template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/main/templates/template-ts) +- JavaScript (Node.js) 👉 [template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/HEAD/templates/template-js) +- TypeScript (ts-node) 👉 [template](https://github.com/scramjetorg/scramjet-cloud-docs/tree/HEAD/templates/template-ts) In the following sections we will describe both templates' contents and show you how they can be used in your own project. @@ -56,13 +56,13 @@ module.exports = function(input) { }; ``` -And this is what our template app does, it takes any input that will be sent or piped to it and simply writes it to the output stream. Of course you can add some logic to this function, but to keep the template simple we will just write the data to the output stream. +This is what our template app does, it takes any input that will be sent or piped to it and simply writes it to the output stream. Of course you can add some logic to this function, but to keep the template simple we will just write the data to the output stream. To see how this template works you can run it with a few simple commands written below: -> :bulb: **Note!** To run this template you need to have those two packages installed: `npm install -g @scramjet/sth @scramjet/cli` +> 💡 **Note!** To run this template you need to have those two packages installed: `npm install -g @scramjet/sth @scramjet/cli` -### **Open 3 terminals** and run the following commands and let the magic begin :magic_wand:: +### **Open 3 terminals** and run the following commands and let the magic begin 🪄 #### 1️⃣ terminal: @@ -73,14 +73,14 @@ To see how this template works you can run it with a few simple commands written - `cd templates` - `si pack template-js` - `si seq send template-js.tar.gz` -- `si sequence start ` -- `si instance output ` +- `si sequence start -` which is the same as `si sequence start ` +- `si q output -` which is the same as `si q output ` #### 3️⃣ terminal: -The command below will run the app in the background. The app generates random numbers from 1 to 10 and write them to instance's `/input` endpoint (to instance which will be run on sth). In this way we are sending an input stream that will be consumed by our template app. +The command below will run the app in the background. The app generates random numbers from 1 to 10 and write them to q's `/input` endpoint (to q which will be run on STH). In this way we are sending an input stream that will be consumed by our template app. -- `node ./tools/stream-gen-tool/numbers-gen.js ` +- `node ./tools/stream-gen-tool/numbers-gen.js ` ### **Expected output:** @@ -88,11 +88,11 @@ The command below will run the app in the background. The app generates random n What you can see in the attached image is 3 terminals that illustrates the template's workflow: -Terminal :one: shows the logs of running `scramjet-transform-hub` process. +Terminal 1️⃣ shows the logs of running `scramjet-transform-hub` process. -Terminal :two: shows the output of the program that we launched using STH CLI. +Terminal 2️⃣ shows the output of the program that we launched using STH CLI. -Terminal :three: shows the output of the `node` command that runs the app which generates random numbers and sends them to the instances's input. +Terminal 3️⃣ shows the output of the `node` command that runs the app which generates random numbers and sends them to the q's input. ## Work with TypeScript (ts-node) @@ -100,7 +100,7 @@ TypeScript compiles to JavaScript. It is a superset of JavaScript, which means t This application package template contains files: -- **package.json** - function of this file is similar to `package.json` file in [JavaScript template](##Work-with-JavaScript-(Node.js)) with a small but :exclamation: **VERY IMPORTANT** :exclamation: difference is that the entry file given in `"main"` property must be typed in without any extension, like in the example below. This is because this `package.jason` will be copied into the `dist` directory, which eventually will be archived into a `.tar.gz` format and sent to the STH. +- **package.json** - function of this file is similar to `package.json` file in [JavaScript template](##Work-with-JavaScript-(Node.js)) with a small but ❗**VERY IMPORTANT**❗ difference. The file given in `"main"` property must be typed in without any extension, like in the example below. This is because this `package.json` will be copied into the `dist` directory, which eventually will be archived into a `.tar.gz` format and sent to the STH. ```json { @@ -129,7 +129,7 @@ This application package template contains files: } ``` -:nerd_face: If you would like to learn more about the configuration in TypeScript projects, please refer to the documentation on [TypeScript official website](https://www.typescriptlang.org/docs/handbook/tsconfig-json.html) and [ts-node official website](https://typestrong.org/ts-node/). +🤓 If you would like to learn more about the configuration in TypeScript projects, please refer to the documentation on [TypeScript official website](https://www.typescriptlang.org/docs/handbook/tsconfig-json.html) and [ts-node official website](https://typestrong.org/ts-node/). - **dist** - this is a folder where the compiled JavaScript files will be stored. It will be created after running this script: `npm run build` together with compiled `index.js` file. @@ -140,7 +140,7 @@ This file will be compiled into manila JavaScript and stores in `index.js` file In our template we introduce you to a very straight forward application, which simply reads input stream and write it to the output stream. -> :book: In the example below we use `@scramjet/types` module. Scramjet is a very nice and functional stream programming framework. You might find it very useful while writing your app or dealing with streams in general. More info, examples and usage you can find [here](https://www.npmjs.com/package/scramjet). +> 📖 In the example below we use `@scramjet/types` module. Scramjet is a very nice and functional stream programming framework. You might find it very useful while writing your app or dealing with streams in general. More info, examples and usage you can find [here](https://www.npmjs.com/package/scramjet). ```typescript import { ReadableApp } from "@scramjet/types"; @@ -177,7 +177,7 @@ And this is what our template app does, it takes any input that will be sent or To see how this template works you can run it with a few simple commands written below: -> :bulb: **Note!** To run this template you need to have those two packages installed: `npm install -g @scramjet/sth @scramjet/cli` +> 💡 **Note!** To run this template you need to have those two packages installed: `npm install -g @scramjet/sth @scramjet/cli` ### **Open 3 terminals** and run the following commands and let the magic begin :magic_wand:: @@ -193,14 +193,14 @@ To see how this template works you can run it with a few simple commands written - `cp -r node_modules/ package.json dist/` - it copies node_modules and package.json to the dist folder - `si pack dist` - it creates a tar.gz archive of the dist folder - `si sequence send dist.tar.gz` - it sends the archive to STH -- `si sequence start ` - it starts the sequence (started sequence turns into instance) -- `si instance output `- it shows the instance's output stream in the terminal +- `si sequence start ` - it starts the Sequence (started Sequence turns into Instance) +- `si inst output `- it shows the Instance's output stream in the terminal #### 3️⃣ terminal: -The command below will run the app in the background. The app generates random numbers from 1 to 10 and write them to instance's `/input` endpoint (to instance which will be run on sth). In this way we are sending an input stream that will be consumed by our template app. +The command below will run the app in the background. The app generates random numbers from 1 to 10 and write them to q's `/input` endpoint (to q which will be run on STH). In this way we are sending an input stream that will be consumed by our template app. -- `node ./tools/stream-gen-tool/numbers-gen.js ` +- `node ./tools/stream-gen-tool/numbers-gen.js ` ### **Expected output:** @@ -208,8 +208,8 @@ The command below will run the app in the background. The app generates random n What you can see in the attached image is 3 terminals that illustrates the template's workflow: -Terminal :one: shows the logs of running `scramjet-transform-hub` process. +Terminal 1️⃣ shows the logs of running `scramjet-transform-hub` process. -Terminal :two: shows the output of the program that we launched using STH CLI. +Terminal 2️⃣ shows the output of the program that we launched using STH CLI. -Terminal :three: shows the output of the `node` command that runs the app which generates random numbers and sends them to the instances's input. +Terminal 3️⃣ shows the output of the `node` command that runs the app which generates random numbers and sends them to the q's input. diff --git a/tools/stream-gen-tool/README.md b/tools/stream-gen-tool/README.md index 1a6c7a7..02737d1 100644 --- a/tools/stream-gen-tool/README.md +++ b/tools/stream-gen-tool/README.md @@ -4,4 +4,4 @@ The stream-gen.js is a Node.js application that you can start using following co `node stream-gen.js ` -Once run, this app will start to send stream of simple text messages to the running instance ("Message 1", "Message 2") and will start to read output from STH API. +Once run, this app will start to send stream of simple text messages to the running Instance ("Message 1", "Message 2") and will start to read output from STH API.