Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 131 additions & 63 deletions README.md

Large diffs are not rendered by default.

6 changes: 2 additions & 4 deletions dictionary.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ Software Development Kit. A collection of tools and libraries to create a softwa

### Sequence

It is a compressed package (`*.tar.gz`) containing file with a manifest(eg. `package.json`), describing the app and its configuration (such as main file to run); and a **main file** (eg. `index.js`, `app.ts`) that contains a developer's code that consists of chained functions with a lightweight application business logic . Minimal sequence consists of 1 [function](#function).
It is a package (application) containing a set of files. A file with a manifest(eg. `package.json`), describing the app and its configuration (such as main file to run); and a __main file__ (eg. `index.js`, `app.ts`) that contains a developer's code that consists of chained functions with a lightweight application business logic. Minimal Sequence consists of 1 [function](#function).

### Socket

Expand All @@ -277,9 +277,7 @@ standard input/standard output. Application communication channels with the outs

### STH <!--TODO-->

### Supervisor

An application supervising work of a single [instance](#instance). Communicates with [host](#host), returns health status of the sequence, runner, etc
shortcut for Scramjet Transform Hub

### Synchronous

Expand Down
14 changes: 8 additions & 6 deletions samples/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
# Samples :books:
# Samples

We have prepared some sample code for you to try out. Some is written in JavaScript and some in TypeScript.
Try out our samples prepared some samples for you. We differentiated them according to the language in which they were written.

> :bulb: **Please note that samples below require some previous installations before you start running them. Instructions are [here](../README.md#3-install-scramjet-transform-hub).**
> 💡 **Please note that samples below require some previous installations before you start running them. Instructions are [here](../README.md#3-install-scramjet-transform-hub).**

## JavaScript samples

- [hello](hello) - Sequence that modifies incoming stream of strings by saying Hello :).
- [hello](hello) - Sequence that modifies incoming stream of strings by saying "Hello".
- [hello-snowman](hello-snowman) - Sequence that reads incoming stream, and modifies it by adding a text message according to the incoming data.
- [simple-counter-js](simple-counter-js) - Sequence, that counts to 1000, and logs the number in one-second intervals.
- [test-output](test-output) - Sequence that just writes random values to the output stream.
- [test-output](test-output) - Sequence that simply writes random values to the output stream.

## TypeScript samples

- [crypto-prices](crypto-prices) - Sequence that keeps printing current crypto prices for a provided pair of currencies every 1s.
- [crypto-prices](crypto-prices) - Sequence that keeps printing current crypto prices for a provided pair of currencies every 3s.
- [discord-slack-connection](discord-slack-connection) - a set of Sequences that enable communication between Slack and Discord.
- [mediawiki](mediawiki) - Sequence that keeps printing mediawiki event stream.
- [rss](rss) - Sequence that gets a list of RSS and then retrieves each feed and passes links to scraper.
- [scraping](scraping) - Sequence that scrapes web pages.
- [stack-overflow](stack-overflow) - Sequence that gets a number of changes in Stack Overflow tag count.
- [transform-string-stream](transform-string-stream) - Sequence that modifies incoming stream of strings by adding a prefix and a suffix.
19 changes: 9 additions & 10 deletions samples/crypto-prices/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
## crypto-prices ![bitcoin](../../images/bitcoin1.png) ![zcash](../../images/zcash1.png) ![ethereum](../../images/etherum1.png)
# crypto-prices ![bitcoin](../../images/bitcoin1.png) ![zcash](../../images/zcash1.png) ![ethereum](../../images/etherum1.png)

---
Sequence that keeps printing current crypto prices for a provided pair of currencies every 1s.
Sequence that keeps printing current crypto prices for a provided pair of currencies every 3 seconds.

> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**
> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**

### Running
## Running

Open two terminals and run the following commands:

Expand Down Expand Up @@ -35,15 +34,15 @@ si pack dist/ -o crypto-prices.tar.gz
si seq send crypto-prices.tar.gz

# start a sequence with currency parameters, this will output Instance ID
si seq start <sequence-id> ETH USD
si seq start - ETH USD

# See output
si inst output <instance-id>
# see output
si inst output -
```

### Output
## Output

Once you run `si inst output <instance-id>` command you will get output like this one:
Once you run `si inst output -` command you should get an output similar to this one:

```bash
Request ok: http://127.0.0.1:8000/api/v1/instance/87442a03-a8ca-451c-b89f-d5371774c2f3/output status: 200 OK
Expand Down
16 changes: 10 additions & 6 deletions samples/discord-slack-connection/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Discord to Slack (and vice versa) Connection

In this project you will find 4 STH sequences:
> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**

In this project you will find 4 STH Sequences:

* [discord-read](./discord-read/)
* [discord-write](./discord-write/)
Expand All @@ -9,21 +11,23 @@ In this project you will find 4 STH sequences:

You must run at least two (discord-read, slack write or discord-write, slack-read) in order to get one directional communication or all four for bi-directional communication.

Each sequence either reads or writes to specified application respectively and uses `const TOPIC = ...` topic to exchange data between applications. You need two topics for bi-directional communication.
Each Sequence either reads or writes to specified application respectively and uses `const TOPIC = ...` topic to exchange data between applications. You need two topics for bi-directional communication.

## TODO

As there is no way to map message IDs between the two applications, threads are not supported.

In order to get this working, sequences must keep a track of posted and written IDs.
In order to get this working, Sequences must keep a track of posted and written IDs.

## Known Issues

## Know Issues
<!-- TODO - check if this issues still apply -->

STH `v0.13` if installed via `npm` has an issue with `objectMode: true`. You will get an error like:

```bash
TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an instance of Buffer or Uint8Array.
Received an instance of Object
TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an Instance of Buffer or Uint8Array.
Received an Instance of Object
```

If that happens either use `v0.14` if available or rebuild and run STH from repo. Please check [transform-hub documentation](https://github.com/scramjetorg/transform-hub/#build-the-packages-building_construction) for details.
Expand Down
26 changes: 13 additions & 13 deletions samples/discord-slack-connection/discord-read/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Discord Read

Read messages from discord channel and write to topic
Read messages from Discord channel and write to topic.

[Discord Documentation](https://discord.js.org/#/docs/discord.js/stable/general/welcome)

Expand All @@ -9,14 +9,14 @@ In order to read messages from Discord we need to create a bot first:
* Go to Developer Portal and click on [Applications](https://discord.com/developers/applications)
* Click on `New Application` button and give it a name. Then click on `Create`.
* In the newly created and selected app, click on `Bot` and click on `Add Bot` button.
* Under `Build-A-Bot` either click on `Click to Reveal Token` link or on `Copy` button to get Discord Bot Token. *This is important!*
* You can uncheck Public Bot
* You can customize its name and icon
* Expand `OAuth2` and select `URL Generator`
* Select `bot` under scopes and `Read Messages/View Channels` under bot permissions. Copy URL
* Paste the URL into web browser address bar. Connect to Discord message will appear. Select your server from `Add To Server` dropdown and click on `Continue`
* Confirm permissions on the next screen by clicking `Authorise` button.
* You will get a confirmation saying: _you may now close this window or tab_.
* Under `Build-A-Bot` either click on `Click to Reveal Token` link or on `Copy` button to get Discord Bot Token. **This is important!**
* You can uncheck Public Bot.
* You can customize its name and icon.
* Expand `OAuth2` and select `URL Generator`.
* Select `bot` under scopes and `Read Messages/View Channels` under bot permissions. Copy URL.
* Paste the URL into web browser address bar. Connect to Discord message will appear. Select your server from `Add To Server` dropdown and click on `Continue`.
* Confirm permissions on the next screen by clicking `Authorize` button.
* You will get a confirmation saying: *you may now close this window or tab*.

Create a file called `config.json` and add:

Expand All @@ -37,14 +37,14 @@ npm install
# transpile TS->JS to dist/
npm run build

# make a compressed package with sequence
# make a compressed package with Sequence
si pack dist

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send dist.tar.gz

# start a sequence, this will output Instance ID.
si seq start <sequence-id>
# start a Sequence
si seq start -

# view messages in topic
si topic get messages
Expand Down
16 changes: 7 additions & 9 deletions samples/discord-slack-connection/discord-write/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
# Discord Write

Read messages from topic and write to discord via webhook
Read messages from topic and write to Discord via webhook.

[Discord Documentation](https://discord.com/developers/docs/resources/webhook)

## Running

Before you start you need `DISCORD_WEBHOOK_URL`. In Discord go your server. Then next to server name click on the drop down menu and select `Server Settings` -> `Integrations` -> `Webhooks` -> `Create webhook` -> Select channel name and save changes. You can copy webhook URL by clicking on `Copy webhook URL` button.
Before start you need `DISCORD_WEBHOOK_URL`. In Discord go to your server. Then next to the server name click on the drop down menu and select `Server Settings` -> `Integrations` -> `Webhooks` -> `Create webhook` -> Select channel name and save changes. You can copy webhook URL by clicking on `Copy webhook URL` button.

WebHook URL has following format: `https://discord.com/api/webhooks/{webhook.id}/{webhook.token}`

where `webhook.id` are numbers and `webhook.token` are letters and numbers.
WebHook URL has following format: `https://discord.com/api/webhooks/{webhook.id}/{webhook.token}` where `webhook.id` are numbers and `webhook.token` are letters and numbers.

```bash
# install dependencies
Expand All @@ -19,14 +17,14 @@ npm install
# transpile TS->JS to dist/
npm run build

# make a compressed package with sequence
# make a compressed package with Sequence
si pack dist

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send dist.tar.gz

# start a sequence, this will output Instance ID. Provide DISCORD_WEBHOOK_URL as the second parameter
si seq start <sequence-id> <DISCORD_WEBHOOK_URL>
# start a Sequence, provide DISCORD_WEBHOOK_URL as the second parameter
si seq start - <DISCORD_WEBHOOK_URL>

# view messages in topic
si topic get messages
Expand Down
8 changes: 4 additions & 4 deletions samples/discord-slack-connection/slack-read/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@ npm install
# transpile TS->JS to dist/
npm run build

# make a compressed package with sequence
# make a compressed package with Sequence
si pack dist

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send dist.tar.gz

# start a sequence, this will output Instance ID. Provide SOCKET_MODE_TOKEN as the second parameter
si seq start <sequence-id> <SOCKET_MODE_TOKEN>
# start a Sequence, provide SOCKET_MODE_TOKEN as the second parameter
si seq start - <SOCKET_MODE_TOKEN>

# view messages in topic
si topic get messages
Expand Down
10 changes: 5 additions & 5 deletions samples/discord-slack-connection/slack-write/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Slack Write

Read messages from topic and write to slack.
Read messages from topic and write to Slack.

In order to get SLACK_WEBHOOK_URL you need to create application in Slack first.
Please refer to notes in [slack-read](../slack-read/) example.
Expand All @@ -20,14 +20,14 @@ npm install
# transpile TS->JS to dist/
npm run build

# make a compressed package with sequence
# make a compressed package with Sequence
si pack dist

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send dist.tar.gz

# start a sequence, this will output Instance ID. Provide SLACK_WEBHOOK_URL as the second parameter
si seq start <sequence-id> <SLACK_WEBHOOK_URL>
# start a Sequence, provide SLACK_WEBHOOK_URL as the second parameter
si seq start - <SLACK_WEBHOOK_URL>

# view messages in topic
si topic get messages
Expand Down
40 changes: 25 additions & 15 deletions samples/hello-snowman/README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,26 @@
## hello-snowman :snowman:
# hello-snowman

----
Sequence that reads incoming stream (input), and and modifies it by adding a text message according to the incoming data.

Stream is generated in [stream-gen.js](../tools/stream-gen-tool/stream-gen.js) file, where numbers in range of <-50,50> are randomly chosen and sent as Celsius degrees to `hello-snowman` instance API endpoint `/input`.
Stream is generated in [stream-gen.js](../tools/stream-gen-tool/stream-gen.js) file, where numbers in range of <-50,50> are randomly chosen and sent as Celsius degrees to `hello-snowman` Instance API endpoint `/input`.

Our `hello-snowman` app will read and interpret these Celsius degrees, and will inform us about state of our Snowman:

- if temperature will be 0 or below, sequence will return message: `Snowman is freezing ... :)`
- in the other case (temperature above 0 degrees), sequence will return message: `Snowman is melting! :(`
- if temperature will be 0 or below, Sequence will return message:

> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**
```bash
Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️
```

- in the other case (temperature above 0 degrees), Sequence will return message:

```bash
Snowman ⛄ is melting! 🥵
```

### Running
> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**

## Running

Open three terminals and run the following commands:

Expand All @@ -32,27 +40,29 @@ cd samples/hello-snowman
# install dependencies
npm install

# make a compressed package with sequence
# make a compressed package with Sequence
si pack . -o hello-snowman.tar.gz

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send hello-snowman.tar.gz

# start a sequence, this will output Instance ID
si seq start <sequence-id>
# start a Sequence, this will output Instance ID
si seq start -

# See output of instance process
si inst output <instance-id>
# See output of Instance process
si inst output -
```

**In the third terminal**

```bash
# Start stream generator tool with instance ID as parameter
# Start stream generator tool with Instance ID as parameter
node ./tools/stream-gen-tool/stream-gen.js <instance_id>
```

### Now you should see something like this in output console:
## Output

Now you should see the output in the console:

```js
----------------------------------------
Expand Down
33 changes: 15 additions & 18 deletions samples/hello/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
## hello 🙋‍♂️
# hello 🙋‍♂️

----
Sequence that modifies incoming stream of strings by saying Hello :).

> :bulb: **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**
> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**

### Running
## Running

Open three terminals and run the following commands:

Expand All @@ -25,36 +24,34 @@ cd samples/hello
# install dependencies
npm install

# make a compressed package with sequence
# make a compressed package with Sequence
si pack . -o hello.tar.gz

# send sequence to transform hub, this will output Sequence ID
# send Sequence to transform hub, this will output Sequence ID
si seq send hello.tar.gz

# start a sequence, this will output Instance ID
si seq start <sequence-id>
# start a Sequence, this will output I ID
si seq start -

# See output of instance process
si inst output <instance-id>
# See output
si inst output -
```

**The third terminal**

```bash
# Send file to instance input steam
si inst input <instance-id> name.txt
# Send file to Instance input steam
si inst input -
# if file not given the data will be read from stdin
# type "John"
John
```

<!-- TODO Delete when the issue is solved
Issue created for reading data from stdin https://github.com/scramjetorg/transform-hub/issues/165 -------ISSUE SOLVED!!! awaits release
-->

### Output
## Output

```bash
# Now you should see "Hello John" in output console
$ si inst output 7a1ffd59-9d1a-4e8f-a246-020124803931
Request ok: http://127.0.0.1:8000/api/v1/instance/7a1ffd59-9d1a-4e8f-a246-020124803931/output status: 200 OK
Request ok: http://127.0.0.1:8000/api/v1/I/7a1ffd59-9d1a-4e8f-a246-020124803931/output status: 200 OK
Hello John
```
1 change: 0 additions & 1 deletion samples/hello/name.txt

This file was deleted.

Loading