diff --git a/README.md b/README.md index b7bb7ddc7..3e72acc00 100644 --- a/README.md +++ b/README.md @@ -1,45 +1,119 @@ # Scramjet Transform Hub -Scramjet Transform Hub is an execution system that is installed on a managed -server by the developer. Once installed, it will serve as an execution platform for -programs on this server. You will be able to quickly and simply execute -programs on this platform, which will automatically forge connections between -programs and allow for remote control of program running, managing and -termination. +Scramjet Transform Hub is a deployment and execution platform. Once installed on a server, it will allow you to start your programs and keep them running on a remote machine. You will be able to start programs in the background or connect to them and see their output directly on your terminal. You will be able to pipe your local data to the program as if it was running from your terminal. You can start your server in AWS, Google Cloud or Azure, start it on your local machine, install it on a Rasperry Pi or wherever else you'd like. -This is the STH development repo. +There's no limit what you can use it for. You want a stock checker? A chat bot? Maybe you'd like to automate your home? Retrieve sensor data? Maybe you have a lot of data and want to transfer and wrangle it? You have a database of cities and you'd like to enrich your data? You do machine learning and you want to train your set while the data is fetched in real time? Hey, you want to use it for something else and ask us if that's a good use? Ask us [via email](mailto:get@scramjet.org) or hop on our [Scramjet Slack](https://join.slack.com/t/scramjetframework/shared_invite/enQtODg2MDIyMTQ5MzUxLTVlNTIwMmFlYWU0YTg2ZTg1YmFiOTZkZTdhNzNmNjE2ZmQ3ZWQzZjI5MGQyZDAyOWM2NDc5YzdmZGQzNGI3YTU)! + +This is the STH development repo. In order to use it, you need to have linux based os, for instance [Ubuntu](https://ubuntu.com/download/server), [docker](https://www.docker.com/get-started) and [node.js v14.x](https://nodejs.org/en/) installed. We're working on development guides for Mac and Windows and node.js v16.x compatibility. A docker install will also be shortly available. + +## TL;DR + +First clone build and start the hub, copy the following commands to the terminal: + +```bash +git clone https://github.com/scramjetorg/transform-hub.git && \ + cd transform-hub && sudo gpasswd -a $USER docker && \ + yarn install && yarn build:all && npm i -g dist/cli && yarn start -P 8000 +``` + +Depending on your machine this may take some time. When it's done the Hub should be running and you should see initial logs showing that the API server has been started on port 8000, something like this: + +``` +2021-07-07T18:19:36.808Z info (object:Host) API listening on port: localhost:8000 +``` + +Now create an application, let's say you want to get the currency rates every 10 seconds and do something. In a clean folder save this as index.js: + +```js +const { DataStream } = require("scramjet"); +const fetch = require("node-fetch"); + +module.exports = function(_stream, apikey, fr, to) { + const idx = `${fr}_${to}`; + const get = () => fetch(`https://free.currconv.com/api/v7/convert?q=${idx}&compact=ultra&apiKey=${apikey}`).then(r => r.json()); + const defer = (t = 10000) => new Promise((res) => setTimeout(res, t)); + + return DataStream + .from(async function*() { + while (true) + yield await Promise.all([get(), defer()]).then(([data]) => data); + }) + .do(async x => { console.log(x[idx]); }) // add some logic here + .run(); +}; +``` + +Copy a [package.json from here](./packages/samples/currency-js/package.json). + +Open a terminal run your program on the hub: + +```bash +si pack /path/to/my/folder -o ~/package.tar.gz # compress the app to a package +SEQ_ID=$(si seq send ~/package.tar.gz) # upload the package to the server SEQ_ID is now it's id +INT_ID=$(si seq start $SEQ_ID -C "{}" $APIKEY BTC EUR) + # start the program on the host with arguments +si inst stdout $INT_ID # see the output from the program. +``` + +See `si help` for more information. Also you will need to get an [API key for this example](https://free.currencyconverterapi.com/). ## Table of contents +- [TL;DR](#tldr) - [Intro](#intro) +- [The Basics](#the-basics) - [How to start development](#how-to-start-development) -- [Basic commands](#basic-commands) -- [Start host](#start-host) -- [Lerna commands](#lerna-commands) -- [Clean build](#clean-build) -- [Docker commands](#docker-commands) -- [Install Host and execute](#install-host-and-execute) -- [Install CLI and execute](#install-cli-and-execute) + - [Development commands](#development-commands) + - [Start host](#start-host) + - [Lerna commands](#lerna-commands) + - [Clean build](#clean-build) + - [Docker commands](#docker-commands) + - [Install Host and execute](#install-host-and-execute) + - [Install CLI and execute](#install-cli-and-execute) - [Build Host on Docker](#build-host-on-docker) - [Run Transform Hub in Docker](#run-transform-hub-in-docker) - [Run components](#run-components) - [Runner](#runner) - - [HostOne](#hostone) - [Sequences and samples](#sequences-and-samples) - [Compress the package](#compress-the-package) - ["Hello Alice" sample](#hello-alice-sample) - [How to run tests](#how-to-run-tests) - [Unit tests](#unit-tests) - [BDD tests](#bdd-tests) -- [Publish](#publish) +- [Publishing](#publishing) +- [License and contributions](#license-and-contributions) ## Intro The readme file contains information about the development process of the STH. It is focused mainly on a day by day commands. Commands won't work as long as you don't set up the environment correctly. You can [find setup instructions in the docs.](docs/development-guide/README.md) +## The basics + +Scramjet Transform Hub allows you to deploy and execute programs that you build and develop. As mentioned above, you can run any program you like, but you need to know a couple of important things: + +* The program should consist of a function or an array of functions, such a program is called a **Transform Sequence**. +* The sequence will be executed within a separate docker instance (we're working on other execution environment integrations - help will be appreciated). +* The sequence function will receive a stream as input in the first argument - you can send the data to it via the command `si instance input`. +* If your sequence contains more than one function, then the output from the previous one is passed to the next one. The first function in sequence receives the input from API. +* The last (or the only) function in sequence can return a `Promise` or a `Stream` - based on this, STH will know when processing is done. +* Once the returned `Promise` is resolved, or the `Stream` is ended, STH will gracefully stop the sequence and remove its container. +* You can communicate with the server via API, command line client `si` which we wrote for your convenience. +* The sequence is called with an AppContext as `this`, a class that allows you to communicate back from the sequence: send logs, provide health info, send and receive events from the API or CLI. +* You can run your sequence multiple times with different arguments (like for instance currency tickers with different symbols or sensor data readers for each sensor) +* The program does not leave your server and doesn't use any external systems. It runs on the server you install the host on. +* Currently STH supports node.js runner only, we're working on bringing you runners for other languages, with Python and C++ as the first ones. + +Some important links: + +* Here you can find the definition of the [Transform Sequence AppContext](./docs/types/interfaces/appcontext.md) +* You can see the [Scramjet Transform Hub API docs here](./docs/development-guide/stream-and-api.md) +* You can see the [CLI documentation here](./docs/development-guide/scramjet-interface-cli.md), but `si help` should also be quite effective. +* Don't forget to `star` this repo if you like it, `subscribe` to releases and keep visiting us for new versions and updates. +* You can [open an issue - file a bug report or a feature request here](https://github.com/scramjetorg/transform-hub/issues/new/choose) + ## How to start development -Follow the below information to start development +If you want to help out, we're happy to accept your pull requests. Please follow the below information to start development. ```bash git clone git@github.com:scramjetorg/transform-hub.git # clone the repo @@ -48,39 +122,23 @@ yarn install # install dependenci yarn build:all # build all packages # -> modules, samples and docker images yarn install -g dist/cli/ # install the cli -yarn packseq +yarn packseq # packs the sequencees yarn start # start the hub ``` Now in another window: ```bash -# this will upload the program to the host -SEQ_ID=$(./scripts/_/upload-sequence packages/samples/example/) - -# this will start the sequence -INSTANCE_ID=$(curl -H "Content-Type: application/json" --data-raw '{"appConfig": {},"args": ["/package/data.json"]}' http://localhost:8000/api/v1/sequence/$SEQ_ID/start | jq ".id" -r); - -# this will show the stdout -stdbuf -o0 -e0 curl --no-progress-meter -X GET -H "Content-Type: application/octet-stream" "http://localhost:8000/api/v1/instance/$INSTANCE_ID/stdout" +si pack /path/to/my/folder -o ~/package.tar.gz # compress the app to a package +SEQ_ID=$(si seq send ~/package.tar.gz) # upload the package to the server SEQ_ID is now it's id +INT_ID=$(si seq start $SEQ_ID -C "{}" $APIKEY BTC EUR) + # start the program on the host with arguments +si inst stdout $INT_ID # see the output from the program. ``` -## Basic commands +### Start host -```bash -yarn install:clean # Removes dist directories, cleans node_modules, installs packages -yarn build # Build all packages and reference apps -yarn build:all-packages # Builds packages only -yarn build:refapps # Builds reference apps only -yarn bic # Build only the changed components -yarn lint # Check and fix syntax -yarn watch # Watch files -yarn lint:dedupe # Check if there are packages to deduplicate -yarn pack:pre # Move linked packages to dist/ (alias: yarn prepack) -yarn pack:pub # Prepare (unlink) packages for publication and move to dist/ -``` - -## Start host +Host can be started in multiple ways ```bash yarn start # Starts Host after it's been built @@ -88,59 +146,36 @@ node dist/host/bin/start # This is the same as above ts-node packages/host/src/bin/start # This starts node from source code ``` -## Lerna commands - -Add new package: - -```bash -lerna create package_name -``` - -List all of the public packages in the current Lerna repo: - -```bash -lerna ls -``` - -Run an npm script in each package that contains that script. - -```bash -lerna run [script] -``` +### Lerna commands -Run script in all packages excluding one package: +We use Lerna to control our monorepo. Here's a couple of helpful commands: ```bash +lerna create package_name # Add new package: +lerna ls # List all of the public packages in the current Lerna repo: +lerna run [script] # Run an npm script in each package that contains that script. lerna run --ignore @scramjet/ -``` - -... or run script excluding more packages - -```bash + # Run script in all packages excluding one package: lerna run --ignore @scramjet/ --ignore @scramjet/ -``` - -Run script only in one package - -```bash + # ... or run script excluding more packages lerna run --scope @scramjet/ -``` - -Run script in more packages - -```bash + # Run script only in one package lerna run --scope @scramjet/ --scope @scramjet/ + # Run script in more packages ``` -## Clean build +### Clean build + +This is how to perform a clean build ```bash -yarn install:clean # this command will perform yarn clean && yarn clean:modules && yarn install at once -yarn build:all-packages # optionally build:all if you want all dockerfiles. -yarn prepack # moves files to ./dist/ +yarn install:clean # this command will perform yarn clean && yarn clean:modules && yarn install at once +yarn build:all-packages # optionally build:all if you want all dockerfiles. ``` -## Docker commands +### Docker commands + +During development some artifact may be left over in docker, here's how to clean them ```bash docker ps # list containers @@ -151,9 +186,9 @@ docker stop $(docker ps -a -q) # stops all running containers > *(`-f`) - don't prompt confirmation -## Install Host and execute +### Install Host and execute -After built and prepack is done, install and run Host: +After build is done, you can install and run Hub globally: ```bash npm install -g ./dist/hub # installs packages globally @@ -167,9 +202,9 @@ npm install -g @scramjet/hub scramjet-transform-hub ``` -## Install CLI and execute +### Install CLI and execute -In the root folder, after building and prepacking, run the following commands: +In the root folder, after building run the following commands: ```bash npm i -g ./dist/cli # install CLI globally @@ -183,7 +218,7 @@ npm i -g @scramjet/cli # install CLI globally si help # show CLI commands ``` -> **HINT:** If something goes wrong make clean, install, and prepack. +> **HINT:** If something goes wrong make clean, install, build. ### Build Host on Docker @@ -235,16 +270,6 @@ Example of usage: node dist/runner/bin/start-runner.js sequence-file-path fifo-files-path ``` -### HostOne - -Starting `HostOne` script: `./packages/host-one/src/bin/start-host-one.ts` - -Example of usage: - -```bash -node dist/host-one/bin/start-host-one.js sequence-file-path config-file-path -``` - ## Sequences and samples To run sequence / sample (example Alice), first, you need to install all the dependencies, [install and execute host](#install-host-and-execute), compress the package, and then you're good to go and use curl commands. @@ -339,13 +364,13 @@ sequence: kill [See more about streams and curl commands =>](docs/development-guide/stream-and-api.md) -> **HINT:** If something goes wrong run clean, build and prepack. +> **HINT:** If something goes wrong run clean, build. Copy and paste 🤞 - ```bash - lerna run clean && lerna run build && lerna run prepack - ``` +```bash +yarn clean && yarn build +``` ## How to run tests @@ -393,7 +418,7 @@ BDD tests are located in a `bdd` folder, to execute them simply follow the steps - start with: ```bash -yarn clean && yarn install && yarn build:all && yarn prepack && yarn packseq +yarn clean && yarn install && yarn build:all && yarn packseq ``` Remeber if you want to test core dump file you must set ```echo '/cores/core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern``` on your linux machine. @@ -410,9 +435,21 @@ yarn test:bdd yarn test:bdd --name="Execute example HelloAlice" ``` +When you want to execute a group of tests you can do it using the substring of their name, for example, to execute all E2E tests: + +```bash +yarn test:bdd --name="E2E" +``` + +You can also execute tests based on their tag name, for example: + +```bash +yarn test:bdd --tags '@ci' +``` + Results of the performed test will be displayed in the console. There is also a report generated in `html` which illustrates the results in a very user friendly form. Html report is generated every time we run a bdd test, those html's are saved in `bdd/reports` folder. -In a result of runnung all the test, both unit and bdd (command: `yarn test`), Lerna goes through all the packages and runs unit tests and also checks the `bdd` directory and runs all bdd scenarios. +In a result of running all the test, both unit and bdd (command: `yarn test`), Lerna goes through all the packages and runs unit tests and also checks the `bdd` directory and runs all bdd scenarios. If you see the error along the way, that means some tests were not passed. @@ -439,7 +476,7 @@ lerna success - @scramjet/types Done in 26.66s. ``` -## Publish +## Publishing To perform full publishing of packages with build and install, perform the following commands: @@ -452,3 +489,23 @@ yarn build:all-packages # build all packages yarn bump:version # bump version and docker images prior to publishing yarn bump:postversion # prepare dist folder, publish packages from dist, push git tags ``` + +## License and contributions + +This project is licensed dual licensed under the AGPL-3.0 and MIT licences. Parts of the project that are linked with your programs are MIT licensed, the rest is AGPL. + +We accept valid contributions and we will be publishing a more specific project roadmap so contributors can propose features and also help us implement them. We kindly ask you that contributed commits are Signed-Off `git commit --sign-off`. + +We provide support for contributions via test cases. If you expect a certain type of workflow to be officially supported, please specify and implement a test case in `Gherkin` format in [`bdd` directory](./bdd). + +### Help wanted + +The project need's your help! There's lots of work to do and we have a lot of plans. If you want to help and be part of the Scramjet team, please reach out to us, [on slack](https://join.slack.com/t/scramjetframework/shared_invite/zt-bb16pluv-XlICrq5Khuhbq5beenP2Fg) or email us: [opensource@scramjet.org](opensource@scramjet.org). + +### Donation + +Do you like this project? It helped you to reduce time spent on delivering your solution? You are welcome to buy us a coffee ;) + +[You can sponsor us on github](https://github.com/sponsors/scramjetorg) + +* There's also a Paypal donation link if you prefer that: [![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=7F7V65C43EBMW) diff --git a/aaa/package.json b/aaa/package.json index 7ebb75cb4..ac7380416 100644 --- a/aaa/package.json +++ b/aaa/package.json @@ -37,7 +37,7 @@ "eslint-to-editorconfig": "^2.0.0", "glob": "^7.1.7", "globrex": "^0.1.2", - "husky": "^6.0.0", + "husky": "6.0.0", "lerna": "^4.0.0", "nyc": "^15.1.0", "proxyquire": "^2.1.3", diff --git a/aaa/yarn.lock b/aaa/yarn.lock index 84e0aaf3a..08adb6921 100644 --- a/aaa/yarn.lock +++ b/aaa/yarn.lock @@ -3799,7 +3799,7 @@ humanize-ms@^1.2.1: dependencies: ms "^2.0.0" -husky@^6.0.0: +husky@6.0.0: version "6.0.0" resolved "https://registry.yarnpkg.com/husky/-/husky-6.0.0.tgz#810f11869adf51604c32ea577edbc377d7f9319e" integrity sha512-SQS2gDTB7tBN486QSoKPKQItZw97BMOd+Kdb6ghfpBc0yXyzrddI0oDV5MkDAbuB4X2mO3/nj60TRMcYxwzZeQ== diff --git a/bdd/delay.test.result.txt b/bdd/delay.test.result.txt new file mode 100644 index 000000000..9914e09ff --- /dev/null +++ b/bdd/delay.test.result.txt @@ -0,0 +1,4000 @@ +229045 +101679 +57276 +51245 +54430 +62046 +86941 +56534 +64149 +55102 +62104 +56635 +56253 +57587 +55062 +57026 +56185 +57397 +57928 +60271 +55924 +57306 +57877 +53268 +49802 +48681 +49151 +48299 +50003 +51485 +49462 +49602 +48881 +48469 +49100 +48479 +48771 +54651 +49652 +49262 +48510 +48750 +48390 +49051 +49081 +48540 +48380 +48761 +48500 +48821 +49311 +49572 +49121 +48680 +49883 +48931 +48530 +49081 +48941 +48991 +48760 +48680 +49492 +51806 +50785 +49312 +49330 +49532 +50053 +49151 +49873 +49400 +48770 +49241 +49252 +49642 +51184 +50013 +48380 +49251 +49592 +49843 +49001 +49702 +49611 +48741 +49582 +49641 +50684 +49852 +49983 +49472 +48810 +49451 +49221 +49482 +51334 +49642 +49641 +57816 +49863 +49291 +49082 +49401 +49492 +49843 +51456 +50955 +48280 +48360 +48250 +48330 +48389 +50905 +48821 +48650 +48801 +48560 +48450 +48300 +48430 +49151 +68897 +50674 +69078 +49331 +50003 +48881 +48941 +48470 +48570 +48019 +49301 +48009 +48299 +48640 +49441 +48881 +48800 +48169 +51636 +53809 +51214 +49612 +66022 +48410 +48811 +48640 +49111 +48309 +49151 +33291 +31448 +32320 +39633 +29935 +28764 +28383 +28473 +30838 +33893 +28752 +28433 +28422 +28823 +28924 +30586 +30847 +32270 +29044 +28844 +34123 +31038 +29165 +27019 +29064 +32200 +30697 +32109 +30877 +32280 +29575 +33832 +27501 +31027 +26540 +25828 +26228 +26379 +25918 +32169 +34724 +30146 +29615 +29445 +29105 +28492 +26979 +30736 +29274 +27370 +28633 +28754 +26309 +33822 +26949 +36118 +48639 +32030 +28443 +26007 +27921 +30446 +30636 +32981 +31469 +31168 +31568 +32049 +32821 +32580 +34494 +31178 +29866 +26269 +25408 +28312 +26478 +29866 +29015 +28123 +28112 +27861 +28463 +28151 +28793 +28062 +28783 +35575 +28273 +28333 +30937 +29624 +27731 +27841 +27972 +30356 +28322 +30015 +27761 +28402 +27812 +28372 +28523 +29815 +28392 +28041 +28222 +29976 +31257 +28192 +28543 +30466 +28042 +28012 +28384 +27280 +28262 +28233 +29115 +29866 +28393 +28372 +28011 +27962 +29855 +29314 +30437 +26809 +30347 +27882 +28593 +28503 +27991 +28083 +28252 +28042 +28173 +29725 +27530 +28041 +27852 +27902 +27802 +27992 +28963 +28212 +28833 +28413 +27862 +27782 +27751 +28162 +28092 +27963 +28383 +27942 +35015 +28232 +28833 +27772 +28083 +33913 +28162 +28131 +27651 +28082 +28474 +36076 +28484 +27971 +41457 +28312 +27872 +27691 +29905 +28602 +28413 +28232 +27862 +27632 +28152 +28422 +28473 +27732 +27961 +27821 +27751 +37279 +31728 +34294 +31748 +33673 +31337 +30387 +30116 +31538 +32570 +32381 +31368 +31107 +30426 +30285 +29975 +32099 +32490 +37820 +34524 +31308 +30085 +33583 +30106 +31397 +31378 +32210 +32430 +31829 +30506 +50073 +35325 +30968 +30938 +32230 +31167 +39884 +30527 +31318 +29273 +28442 +42900 +35686 +46636 +29535 +29174 +26328 +33021 +32621 +30217 +32740 +41808 +48490 +34313 +28863 +27831 +28132 +33452 +28634 +26999 +28043 +28854 +27442 +28804 +27781 +27611 +27742 +26949 +27461 +27330 +26980 +41116 +27712 +37219 +28734 +27161 +27891 +27200 +27410 +30267 +27982 +2471350 +27722 +25918 +24626 +24356 +25197 +25136 +24625 +25729 +27681 +25397 +25026 +24706 +24795 +25328 +25637 +23554 +25227 +22762 +22621 +24956 +27411 +31478 +27661 +31559 +26429 +27661 +27410 +27601 +28643 +28874 +27522 +27973 +26639 +27191 +32079 +29024 +54000 +28223 +29554 +27009 +26719 +26789 +27992 +26409 +27883 +29435 +24596 +26940 +26619 +27111 +26689 +29205 +624838 +27631 +24264 +23072 +24446 +22872 +26329 +920266 +26870 +22381 +20418 +20498 +20417 +20017 +19788 +19887 +20066 +20267 +20818 +20076 +29765 +20629 +20087 +20689 +20047 +19015 +18364 +18134 +17872 +19336 +18865 +19496 +19787 +19757 +19496 +19546 +19737 +19486 +18505 +18836 +19967 +19797 +20088 +19777 +18925 +19917 +19796 +19006 +17974 +18625 +18994 +21040 +20267 +19066 +19646 +19566 +19836 +20237 +19817 +19516 +18243 +18795 +21360 +20018 +21881 +19056 +15749 +15348 +15058 +15780 +14938 +15129 +14848 +15838 +14988 +17602 +15989 +15478 +15008 +15129 +30096 +18435 +15219 +36667 +16721 +15548 +14197 +14337 +15349 +15158 +15179 +14967 +19196 +17491 +15249 +15068 +15449 +15648 +14927 +15228 +15178 +15247 +15157 +15358 +15239 +15669 +15298 +15177 +15198 +15869 +15299 +16269 +25187 +19816 +17542 +17432 +17923 +17162 +17222 +17322 +16872 +19407 +17132 +21229 +17362 +17872 +17002 +17312 +15639 +18184 +17673 +16800 +17111 +17062 +16680 +17512 +17041 +26088 +16681 +13605 +15188 +15107 +14126 +15097 +15108 +13746 +15348 +14978 +14857 +15107 +15088 +15870 +15979 +14758 +15079 +15508 +15309 +15058 +15129 +15319 +16691 +14327 +13765 +13686 +17031 +14066 +13795 +13997 +14267 +16050 +13745 +13795 +13876 +15058 +13915 +13545 +14287 +13976 +13796 +13806 +13985 +13755 +13826 +14146 +14086 +13886 +13756 +14527 +13516 +13666 +14226 +13916 +13805 +14115 +14748 +13674 +13855 +14006 +14207 +14467 +14448 +13986 +13816 +13626 +14407 +14055 +14367 +14076 +13935 +14015 +15297 +13955 +13886 +13885 +13496 +13916 +13846 +15409 +13916 +13766 +13906 +14035 +13524 +14125 +15850 +14857 +13454 +13886 +13915 +14387 +14526 +13936 +14026 +13695 +28923 +16541 +14126 +14747 +14046 +14167 +14427 +13636 +14197 +16229 +14176 +13796 +14125 +14197 +13935 +14417 +14187 +14376 +14426 +13746 +13796 +15930 +14086 +13865 +14115 +13936 +14818 +13947 +13855 +13935 +14237 +14186 +14286 +16811 +23574 +16491 +13895 +23263 +17963 +15179 +13354 +14156 +15419 +13796 +14197 +14095 +13836 +14027 +16280 +14376 +13775 +13976 +13816 +13764 +14075 +14227 +14287 +13946 +14126 +14767 +14236 +14357 +14508 +16331 +13504 +14185 +13534 +14717 +13745 +13947 +14087 +14076 +13796 +18074 +16692 +16511 +17632 +17703 +15378 +13476 +14556 +14636 +14175 +14115 +14426 +14446 +14075 +14367 +14967 +15279 +14306 +15818 +14478 +14798 +16110 +17652 +17092 +13564 +14757 +14167 +14958 +13985 +13935 +20818 +16431 +14548 +15870 +13925 +14165 +15238 +13285 +14726 +14266 +31498 +16149 +13967 +13686 +14427 +17332 +14947 +13886 +14096 +13556 +13696 +14667 +13785 +13355 +14175 +17462 +16089 +14848 +13735 +14417 +38292 +13765 +13594 +13415 +13185 +13534 +13466 +13736 +15629 +14516 +14087 +13786 +14336 +17361 +15589 +14546 +14287 +13544 +14046 +13466 +13947 +14307 +14416 +13785 +13766 +14887 +14097 +14257 +32410 +13536 +14217 +14487 +17222 +14396 +16120 +13265 +13065 +13264 +13414 +12213 +13946 +14577 +13485 +12814 +13385 +15168 +13986 +13235 +14567 +12834 +14055 +15710 +13434 +12403 +12563 +12573 +12513 +13255 +12494 +14377 +12233 +13334 +12322 +12353 +16721 +12904 +12803 +12734 +13015 +12633 +12373 +12483 +13004 +13055 +12182 +12203 +13305 +13234 +17993 +12273 +11361 +19427 +11009 +10890 +12163 +12363 +15327 +13105 +12874 +11500 +15178 +12964 +12452 +17452 +15538 +14256 +12292 +12554 +12364 +13073 +12682 +12633 +12633 +12593 +12704 +12854 +12534 +12753 +28432 +12823 +13576 +12482 +12413 +12444 +12713 +12403 +13235 +12263 +14598 +12253 +12764 +12483 +12723 +13555 +12352 +12784 +13574 +13003 +16921 +15177 +15358 +13395 +12704 +12753 +13295 +12984 +13845 +12452 +12773 +12614 +12824 +13335 +12453 +15117 +13135 +13314 +12413 +13144 +11812 +12554 +14787 +13245 +12895 +12794 +12814 +12484 +12654 +12634 +12763 +14977 +12784 +12683 +12664 +12894 +12252 +12773 +12754 +11441 +13555 +12834 +12965 +15028 +13285 +12935 +12443 +12903 +12493 +12283 +15630 +13736 +12973 +12503 +12473 +12654 +12593 +11902 +12484 +12434 +11821 +11812 +12303 +17082 +15077 +12293 +12512 +12324 +12652 +12312 +12854 +12734 +12414 +13054 +14417 +13165 +12924 +12333 +11612 +12633 +12503 +12142 +11090 +12464 +12444 +12683 +12342 +14236 +12894 +12453 +12824 +12503 +12894 +12342 +12744 +17692 +15038 +11562 +12574 +13103 +12524 +12373 +12774 +12654 +13064 +12814 +13234 +12714 +12413 +22571 +13344 +12564 +12855 +12414 +12413 +12383 +12684 +12514 +12263 +12783 +1114645 +16059 +13896 +14828 +14015 +11101 +11893 +11331 +12923 +11472 +11602 +15229 +14617 +12814 +12413 +12492 +11481 +12273 +11572 +11941 +11622 +11711 +12082 +11702 +15789 +12574 +11531 +11532 +12604 +11963 +14376 +12021 +11752 +11962 +17662 +11141 +10861 +13345 +11903 +10469 +13345 +10610 +11781 +12342 +11793 +11561 +12894 +12454 +12253 +13384 +11581 +11831 +14277 +12594 +12092 +12133 +12293 +11491 +12283 +12083 +12233 +12273 +442600 +12713 +11091 +13105 +10930 +11121 +11141 +20558 +12963 +13765 +17272 +13455 +13405 +12051 +10720 +14086 +11280 +10909 +11401 +11100 +10971 +14296 +14107 +13254 +11611 +11261 +12213 +11351 +11320 +19155 +12183 +11241 +11131 +10960 +11482 +11551 +11230 +11672 +11091 +12013 +11191 +11591 +11512 +13555 +11520 +10820 +12302 +9978 +11211 +12905 +11321 +11702 +15429 +11901 +11751 +14046 +11541 +9909 +11010 +11341 +11522 +11051 +11552 +11382 +11101 +10980 +11311 +9957 +11460 +10239 +10810 +9997 +10890 +10240 +10961 +9999 +11161 +10550 +11440 +10188 +11171 +12434 +13205 +12083 +13504 +11070 +10939 +11381 +10880 +10790 +10820 +10890 +12414 +14656 +11321 +12032 +11821 +10559 +11632 +11472 +11582 +11271 +20228 +12433 +11992 +11130 +11401 +11021 +11712 +11422 +12082 +10931 +11301 +11341 +11390 +11912 +12994 +10930 +11161 +11131 +11331 +11071 +11350 +14838 +11031 +11120 +11331 +11181 +10990 +11330 +11582 +11651 +27240 +11221 +11090 +12183 +11231 +11301 +11191 +11261 +11001 +11450 +11401 +11241 +11510 +11972 +11581 +11331 +11762 +11732 +16331 +11822 +11260 +11400 +11120 +10921 +14566 +11782 +10931 +10910 +13234 +15448 +11952 +14938 +10640 +11211 +11461 +11411 +11410 +11572 +11220 +13185 +11191 +10329 +11101 +11140 +11100 +11321 +11401 +11261 +11170 +13946 +12312 +20679 +16851 +11832 +12352 +11432 +11151 +11602 +11120 +12203 +11542 +11221 +11702 +11151 +24555 +22691 +18114 +15108 +12042 +12153 +10539 +10259 +9819 +9948 +10179 +10299 +10810 +10810 +10871 +10951 +11061 +10829 +11121 +10970 +10871 +11071 +10700 +20058 +11712 +10700 +12343 +10841 +10900 +10881 +11501 +11151 +10941 +10680 +10931 +10930 +11010 +10951 +11100 +10749 +11091 +10980 +11060 +10579 +10780 +10700 +10720 +11150 +10931 +11251 +11111 +11020 +10880 +10999 +11040 +11191 +10879 +11060 +11091 +12553 +10991 +11241 +10791 +10881 +10630 +11121 +11341 +10880 +10679 +11091 +10781 +10911 +10981 +11602 +10920 +11191 +11312 +12222 +11792 +10819 +10831 +10790 +10689 +10940 +10791 +10660 +10769 +10709 +10720 +10991 +10780 +11151 +10900 +10921 +11031 +11010 +10691 +10871 +10740 +10909 +12403 +10891 +10729 +10770 +10490 +10801 +12182 +11111 +10881 +10620 +11220 +11692 +10650 +10881 +10860 +10789 +10920 +11421 +10619 +10730 +10700 +10861 +11031 +10751 +10730 +10900 +10919 +10941 +10851 +10650 +11472 +10900 +10690 +11732 +10780 +10699 +10921 +11009 +10779 +10910 +11111 +10800 +10849 +10920 +10870 +11031 +10930 +11232 +10990 +11702 +11431 +11471 +10820 +10770 +13014 +12504 +11181 +10901 +11301 +10951 +11100 +10619 +10890 +11091 +12543 +10859 +11491 +11392 +11201 +10930 +11773 +11371 +10980 +11091 +10981 +10881 +10961 +11302 +11492 +11060 +10831 +11381 +11060 +10830 +11010 +12232 +10960 +11582 +11341 +10911 +10660 +12123 +13364 +11682 +11119 +11391 +11581 +11130 +11252 +11762 +11211 +11241 +10949 +11261 +11341 +11161 +10980 +11231 +11231 +10830 +11010 +11051 +10760 +10821 +10770 +10920 +10720 +11071 +10891 +11862 +16390 +11672 +12022 +10089 +10980 +10859 +12323 +10900 +10129 +10940 +10851 +11211 +11040 +10911 +11010 +10740 +10049 +11090 +20267 +12452 +10939 +10950 +11302 +10991 +11262 +9838 +10850 +10849 +10899 +11120 +10800 +9767 +10870 +11151 +10869 +11051 +11010 +10890 +10981 +10829 +10960 +10849 +10870 +11571 +11011 +11231 +11852 +11111 +11160 +11100 +10049 +11111 +11081 +10750 +10690 +11272 +10730 +11522 +11260 +11241 +11250 +10730 +10740 +10690 +10619 +11081 +11231 +11181 +10880 +10890 +11101 +12362 +10769 +10820 +11030 +11271 +10820 +11121 +10940 +11390 +12314 +11071 +10770 +11061 +10700 +10840 +10809 +11030 +11321 +10849 +10959 +10799 +11021 +11181 +11200 +10921 +11020 +11281 +11331 +11181 +10981 +12242 +11080 +10700 +11201 +11150 +10900 +10841 +12494 +11361 +11651 +10760 +11111 +10941 +11251 +10930 +11171 +10830 +10780 +11441 +11251 +10721 +10680 +10690 +11943 +13595 +11201 +10940 +10810 +11200 +11281 +10941 +11100 +11050 +10911 +11180 +14957 +11482 +11031 +10700 +11130 +11381 +10881 +10770 +11451 +12633 +10940 +10970 +10980 +10950 +10630 +10810 +11120 +10881 +27610 +10749 +10780 +10851 +11191 +10750 +11031 +12232 +11091 +10991 +10700 +11001 +10871 +11242 +11241 +11551 +11071 +11021 +10780 +48249 +11111 +10850 +10881 +10880 +10770 +11061 +11080 +11101 +11161 +10981 +10791 +11070 +12323 +10950 +10740 +10820 +10740 +10740 +10610 +10720 +10871 +10720 +10680 +10771 +10589 +12334 +11061 +11361 +10900 +11141 +11121 +11251 +10779 +10910 +11110 +19376 +11742 +10720 +10680 +10620 +11081 +10860 +11091 +11140 +10961 +11121 +10780 +11100 +10951 +11672 +11391 +11181 +11081 +11040 +10700 +11160 +11320 +11121 +12003 +10609 +10881 +10940 +11011 +10880 +10920 +10910 +10800 +11011 +11101 +11151 +11101 +11040 +11101 +17011 +12814 +12193 +10169 +11712 +11170 +11401 +11341 +11452 +11430 +12053 +11351 +11301 +10960 +11091 +11231 +11602 +11982 +11181 +11191 +10899 +11392 +10961 +10991 +11390 +10991 +10980 +10761 +11361 +11191 +10881 +10780 +11201 +10991 +10730 +11231 +11230 +11312 +11272 +10870 +11061 +11070 +10880 +10990 +10900 +10679 +10879 +11311 +10810 +11261 +11883 +12083 +10970 +10720 +10801 +10720 +10990 +12632 +11532 +10740 +11101 +12273 +10790 +10750 +10900 +12903 +12502 +12985 +9818 +9176 +10921 +11250 +11021 +10890 +11221 +11030 +10931 +11220 +11351 +10710 +12163 +11091 +11131 +10940 +11041 +13284 +12032 +11291 +11030 +10890 +11131 +12513 +10810 +10750 +11783 +9779 +11091 +10891 +11021 +11240 +11141 +11371 +11030 +11050 +11121 +11121 +13575 +12513 +10880 +10961 +10801 +11089 +11011 +10720 +10689 +10951 +17883 +11391 +20078 +11611 +11000 +10639 +11191 +11561 +9788 +9198 +11121 +11301 +11661 +10900 +10690 +10911 +11020 +11191 +10900 +10991 +11260 +11000 +10900 +11180 +10879 +10941 +10569 +12212 +10730 +11130 +11481 +10861 +11071 +12493 +10851 +10820 +11020 +12123 +11091 +10449 +10650 +10921 +11161 +10861 +10841 +10850 +11261 +10990 +12542 +10709 +12142 +10931 +10740 +12112 +10810 +10991 +11050 +10750 +11019 +14066 +11552 +10939 +10659 +10740 +10789 +11742 +10870 +10730 +12183 +10599 +10790 +10810 +10791 +10780 +11181 +11071 +11029 +10730 +11261 +10991 +12183 +12433 +10910 +11512 +10849 +10921 +26389 +10920 +11272 +12562 +11081 +11382 +28023 +11161 +11372 +11392 +11090 +11000 +11021 +10760 +11009 +11140 +10128 +9939 +10959 +11712 +10620 +11000 +11521 +10720 +10950 +10900 +11211 +10811 +20338 +11702 +11190 +10951 +10961 +11039 +10721 +11020 +11031 +11190 +11159 +11321 +10970 +11120 +11041 +10961 +11090 +11240 +10809 +10710 +12393 +10029 +10991 +10841 +11350 +11001 +14577 +11190 +10890 +11100 +11692 +10960 +11321 +10860 +10799 +10929 +10880 +11221 +10901 +10911 +10670 +10871 +10951 +11130 +12293 +11431 +11231 +10990 +11401 +11672 +11081 +11231 +10740 +10940 +11041 +10761 +11050 +11261 +10740 +10760 +10789 +11121 +11041 +10880 +10981 +11230 +11931 +11011 +10881 +11271 +11360 +10951 +11071 +11161 +11291 +11071 +11180 +10901 +11020 +11151 +10990 +11572 +11101 +11211 +11101 +11031 +11390 +10880 +11412 +11091 +10780 +11271 +10048 +10840 +10950 +10740 +11271 +20408 +11551 +10840 +11131 +10640 +10941 +11120 +11011 +11993 +10740 +11069 +33582 +11120 +11001 +26390 +10891 +11000 +11090 +11311 +10940 +10720 +11060 +11251 +10740 +12503 +11812 +10811 +10960 +11131 +11130 +10931 +11001 +11110 +10990 +11050 +15399 +11141 +10710 +11010 +10879 +11111 +11823 +10730 +10981 +11511 +11041 +10991 +10821 +11090 +11091 +10999 +11311 +10981 +12613 +11091 +10971 +10950 +10880 +10931 +10860 +11340 +10930 +11131 +11010 +10690 +10911 +11190 +11041 +11010 +10961 +11049 +10890 +11131 +11091 +10821 +11191 +11150 +11231 +11161 +11331 +11041 +11049 +12353 +11049 +11121 +10850 +10999 +11081 +11061 +11211 +12362 +11081 +10779 +10861 +10941 +11039 +11141 +10850 +11071 +10851 +11001 +11210 +11151 +11131 +10890 +10870 +10950 +11792 +11131 +10910 +10961 +10239 +10981 +11241 +11231 +10869 +10931 +11039 +11902 +11221 +11091 +11101 +10850 +11451 +10840 +10920 +11180 +10860 +11261 +12562 +11071 +10931 +10960 +11081 +10941 +11041 +11331 +11301 +20238 +11891 +10940 +11011 +11762 +11039 +11813 +11071 +10961 +11071 +11071 +11211 +11652 +10811 +10899 +10871 +10950 +10801 +10830 +11081 +10940 +10891 +10910 +11993 +10971 +10650 +10921 +10800 +10810 +11520 +11492 +11081 +11231 +11231 +10931 +10881 +18044 +10159 +11021 +11191 +10790 +11160 +11171 +11019 +11020 +11331 +10880 +11071 +11522 +12553 +11211 +10790 +10719 +11120 +10940 +11381 +11181 +11351 +28312 +10669 +10810 +11401 +11300 +11081 +11161 +10821 +11411 +11360 +10981 +11010 +10990 +11301 +10969 +10951 +10931 +12042 +10991 +11501 +11171 +11081 +10871 +10970 +11781 +11101 +11321 +11101 +12664 +11111 +11161 +11121 +11031 +11091 +11061 +11021 +10879 +11011 +10971 +11642 +11301 +10911 +11159 +12754 +10991 +11421 +10839 +11001 +10850 +11151 +10891 +11281 +10980 +10779 +10890 +11001 +11221 +10939 +10901 +28644 +11031 +10971 +11011 +10740 +10991 +12443 +12193 +11271 +11511 +10598 +10870 +11180 +11061 +10909 +11261 +11030 +11101 +11191 +10910 +10949 +11059 +11181 +10810 +10960 +10930 +13405 +15228 +14196 +13986 +13626 +14077 +15519 +15069 +14787 +12933 +14167 +16349 +17102 +12824 +11141 +387489 +10389 +10630 +10640 +18153 +12804 +11111 +10761 +10911 +10810 +10690 +13806 +14546 +10709 +10930 +10940 +10640 +10780 +10649 +10660 +10720 +10470 +10840 +21000 +11310 +10530 +10890 +10450 +15418 +12152 +10319 +11921 +10821 +10609 +10498 +10459 +10410 +10770 +10519 +10719 +10560 +10829 +10389 +11301 +10921 +10510 +10500 +10459 +11272 +10558 +10508 +10620 +10750 +10830 +10650 +10610 +10810 +10740 +10710 +10719 +10770 +10840 +11040 +10340 +12303 +10640 +10539 +10580 +9507 +10330 +10570 +467386 +10500 +10400 +11471 +10229 +10039 +10038 +9839 +9979 +10119 +9878 +9708 +9097 +9758 +10620 +9317 +9217 +10419 +12073 +9969 +10109 +10851 +10169 +10539 +8946 +10791 +10620 +10610 +9346 +10369 +10048 +9958 +11371 +10279 +20738 +16000 +15189 +14447 +20769 +14417 +14196 +15219 +14668 +15710 +22031 +15619 +25798 +25688 +14798 +9979 +10029 +9797 +10510 +10229 +10479 +10229 +9508 +10309 +9477 +10519 +10950 +10419 +10079 +9929 +10320 +9938 +10149 +10149 +10238 +11832 +9868 +10460 +10660 +8626 +9789 +11131 +9117 +8635 +8847 +8716 +10029 +10188 +10429 +10188 +10419 +10149 +9758 +10009 +9849 +10830 +10509 +10640 +10379 +10650 +10259 +9968 +13235 +10230 +9868 +10009 +10189 +10169 +10148 +9828 +10159 +9908 +10610 +10269 +9958 +9728 +10038 +9838 +9878 +9978 +9918 +10078 +10129 +9899 +10209 +10109 +12574 +9827 +9918 +9929 +9918 +11732 +9899 +15007 +15309 +9126 +8566 +8866 +9838 +10280 +9898 +10049 +9929 +10018 +9858 +10119 +10099 +10219 +10719 +10089 +10018 +9848 +10079 +10690 +11341 +10139 +10248 +10510 +10279 +10690 +10550 +10088 +11341 +10680 +10169 +10189 +9998 +10169 +14548 +14466 +18465 +21019 +21120 +21208 +19797 +17783 +15848 +14197 +15048 +15588 +11181 +10400 +10449 +9808 +9868 +9909 +10109 +9837 +9999 +9747 +9999 +10430 +9988 +10189 +10269 +9778 +10159 +10069 +9858 +9718 +9968 +10008 +10610 +9867 +10109 +10069 +10269 +9988 +11372 +10540 +20359 +9959 +9698 +9838 +10009 +9819 +10218 +9968 +9898 +9849 +9909 +10239 +10109 +10529 +9978 +9928 +10720 +10390 +10339 +10079 +11341 +10539 +10740 +10250 +9959 +9928 +10179 +10099 +9848 +9989 +10079 +9958 +9988 +9778 +9898 +9758 +9908 +10269 +9908 +9778 +10099 +10810 +10168 +9999 +9938 +11151 +10369 +10139 +10079 +9869 +10099 +10349 +10398 +10119 +10249 +10209 +9939 +10118 +10028 +9909 +10089 +10028 +10549 +10027 +9888 +14467 +10229 +9859 +10240 +20177 +11171 +10369 +10239 +10139 +10841 +9948 +20087 +10780 +10239 +10179 +10039 +9877 +9638 +9998 +9989 +11541 +9688 +9867 +10109 +10078 +9958 +9908 +9598 +10210 +10079 +9818 +10840 +10099 +9999 +10349 +9839 +10218 +9949 +10399 +10008 +9918 +10400 +9668 +10078 +10380 +9969 +9869 +10119 +9829 +11461 +10079 +9729 +9928 +9889 +9909 +9868 +9939 +10079 +9989 +9929 +9808 +9888 +9849 +9899 +8676 +10739 +10150 +9749 +10029 +9808 +11502 +9819 +10048 +9999 +10059 +10049 +9879 +10729 +9837 +10149 +9718 +10379 +10590 +10249 +9769 +10871 +9839 +9939 +9558 +10109 +9999 +10550 +9568 +10019 +10298 +11421 +10009 +9889 +9739 +9928 +9718 +9958 +10049 +9818 +10149 +9808 +9858 +10128 +9718 +9818 +10008 +13435 +10359 +9899 +10259 +10078 +9918 +10139 +10308 +10169 +9848 +10108 +9848 +10490 +10119 +10159 +9929 +10048 +11732 +9927 +10028 +10148 +10178 +10630 +10169 +10099 +10559 +10029 +10007 +10048 +10449 +9999 +9968 +9938 +10350 +10129 +10499 +10369 +10199 +9978 +10399 +10017 +9879 +10008 +9989 +9869 +9588 +10018 +9808 +9878 +9839 +10509 +9838 +9858 +10129 +9809 +10148 +10009 +9838 +9879 +10248 +10118 +10460 +10059 +9929 +10048 +10109 +10238 +9958 +9948 +10079 +8926 +9849 +10199 +13245 +10219 +9838 +9949 +10409 +10249 +10028 +10019 +10029 +10569 +10390 +11361 +10078 +10409 +9868 +10119 +9689 +9768 +10009 +10169 +9959 +9798 +10259 +9728 +10500 +9808 +8966 +9869 +9939 +9858 +9989 +10058 +9828 +10108 +9818 +10189 +10409 +10149 +10029 +9749 +10069 +10019 +10019 +9788 +10209 +9978 +10239 +9989 +9888 +10159 +10229 +10229 +9839 +9978 +9999 +10029 +10650 +9999 +9758 +9188 +10119 +10129 +9818 +10489 +10099 +9687 +10069 +9797 +9879 +10189 +9969 +9889 +10268 +10791 +10269 +9687 +10128 +9789 +9958 +9999 +9999 +9748 +10219 +9969 +9879 +10449 +10449 +10150 +10849 +9868 +9988 +9969 +10039 +9888 +10089 +9838 +9768 +10520 +10059 +10059 +10399 +11992 +9879 +9848 +17492 +15689 +9848 +17332 +10549 +10039 +9477 +9829 +9877 +9979 +9649 +10029 +9818 +10229 +9678 +9708 +9818 +9798 +9879 +9777 +9617 +10258 +11802 +10300 +9928 +9658 +9707 +9829 +10048 +9748 +9749 +9807 +9768 +9908 +10159 +9758 +9889 +9618 +9838 +9869 +9568 +9969 +9928 +9558 +9927 +9889 +9939 +10099 +9818 +10449 +9838 +9848 +9778 +9797 +10590 +9678 +9757 +9718 +9638 +10048 +9798 +9528 +9598 +9448 +9598 +10029 +9688 +9808 +9567 +9718 +9899 +9737 +9928 +9989 +9667 +9708 +9858 +9577 +9768 +10420 +9829 +8926 +9819 +9899 +9858 +9608 +9667 +9537 +9497 +10029 +9648 +9709 +9718 +9518 +9558 +9919 +9738 +9347 +9818 +9939 +8726 +10229 +9879 +10069 +9838 +9828 +9458 +9817 +9748 +10158 +10189 +10068 +10789 +9869 +9778 +10240 +9938 +9969 +11201 +10018 +9988 +11432 +9627 +10369 +9798 +9757 +9828 +10139 +9878 +9877 +9899 +9968 +10129 +9709 +11372 +9828 +9939 +9818 +18925 +25086 +10089 +9919 +9869 +12001 +8927 +10129 +10098 +9777 +9697 +10119 +9919 +9788 +9989 +10029 +9617 +9858 +9748 +10349 +10529 +9728 +10079 +11532 +9999 +9918 +10188 +17912 +14907 +9758 +9658 +9708 +9448 +9798 +9488 +9137 +9307 +9648 +9868 +9828 +9748 +9768 +9558 +10049 +10028 +9759 +9977 +9788 +9738 +9929 +9278 +10209 +9567 +9668 +9638 +9418 +9779 +9197 +9256 +9457 +9318 +9607 +9438 +9477 +9608 +9487 +9277 +9327 +9738 +9899 +9828 +8326 +9378 +9277 +9297 +9226 +9487 +9858 +9197 +9418 +9637 +10019 +10259 +9467 +8285 +9697 +9367 +9548 +9397 +9938 +11371 +9468 +9537 +9848 +9378 +9548 +9467 +9667 +9588 +9577 +9598 +8947 +9308 +9247 +9497 +9337 +9437 +9548 +9308 +9388 +9398 +9476 +9538 +9528 +9687 +9578 +9167 +9307 +9287 +9769 +8395 +10038 +8696 +9408 +9528 +9838 +8917 +9378 +9488 +9458 +9328 +11059 +9699 +9328 +9608 +9879 +9628 +9528 +9368 +9347 +9488 +9206 +9498 +9577 +9958 +9718 +9437 +9327 +9658 +9437 +9348 +9538 +9527 +9368 +9137 +9348 +8877 +8396 +9347 +11692 +8346 +9166 +9508 +9378 +9728 +9568 +9488 +9538 +9177 +9568 +9537 +19997 +9898 +11251 +9888 +9878 +10089 +9457 +9618 +9478 +9568 +9608 +9818 +10179 +8465 +9687 +10689 +11341 +9807 +10009 +9728 +9908 +9779 +9818 +9577 +19617 +9759 +10109 +9708 +9317 +9627 +9468 +9588 +9458 +9318 +9678 +9568 +9678 +9618 +9498 +9478 +9538 +9268 +10029 +9778 +9638 +9446 +9968 +10599 +9478 +9396 +9518 +9317 +10890 +9118 +9467 +9317 +9328 +9306 +9608 +9698 +9257 +9578 +9367 +9147 +10329 +9337 +9257 +9488 +9687 +9428 +9357 +9618 +9578 +9558 +9538 +9588 +9677 +9738 +9348 +9568 +9336 +9267 +9558 +9426 +9277 +9448 +10290 +9358 +9428 +9456 +9318 +9408 +9177 +9277 +9447 +9498 +9989 +9368 +9337 +10179 +9277 +9498 +11592 +8496 +10169 +9498 +9428 +9488 +9447 +9648 +9737 +9838 +9498 +9438 +9758 +9888 +9256 +9307 +9487 +9618 +9237 +9506 +9548 +9769 +8335 +9447 +9487 +9919 +9258 +9748 +12854 +9458 +9507 +8516 +9408 +9438 +9587 +9377 +9798 +9607 +11029 +8766 +9678 +9418 +11292 +9408 +9378 +9638 +9417 +9458 +9217 +9287 +9828 +9618 +9348 +8626 +9428 +8777 +9467 +9548 +9207 +9306 +9428 +9458 +9618 +9628 +9398 +9969 +10238 +9668 +9558 +9448 +9538 +9378 +9628 +9428 +9739 +9457 +10490 +9498 +9508 +9678 +10779 +9749 +9248 +9858 +9467 +8646 +9328 +9327 +9347 +9697 +9538 +9307 +9427 +9327 +9438 +9438 +9127 +9638 +9217 +10680 +10118 +10520 +14216 +10630 +33101 +9227 +9147 +9537 +9457 +10588 +9297 +9527 +17142 +10319 +8596 +9437 +10350 +8344 +10870 +9789 +9598 +9267 +9388 +9538 +11832 +9508 +9458 +9689 +9518 +9498 +9678 +9627 +9317 +9438 +9467 +9357 +14908 +9498 +9408 +9347 +9518 +9257 +9648 +9187 +9618 +9468 +9207 +9588 +9648 +9528 +9658 +10509 +9097 +9248 +9618 +9698 +9407 +9508 +9367 +9348 +9558 +9358 +9397 +9297 +9287 +9909 +9447 +9317 +9316 +9397 +9347 +9538 +9768 +9358 +9348 +9428 +9248 +9577 +9497 +9608 +9689 +9408 +9397 +9789 +9889 +9678 +9296 +9687 +9768 +9918 +9768 +15028 +9237 +9829 +9338 +9478 +9387 +10059 +9758 +9658 +10349 +9617 +9708 +10129 +10428 +9758 +11331 +10230 +9658 +9638 +10399 +9738 +10129 +11852 +8626 +9959 +10039 +10570 +9879 +9377 +10369 +9937 +9838 +10440 +9607 +9808 +10961 +9827 +10239 +9657 +9517 +9698 +9348 +10129 +9677 +9488 +9668 +9718 +9518 +9828 +9797 +9598 +10430 +10159 +10369 +10850 +9698 +10118 +11701 +9627 +10169 +9928 +10630 +9828 +8716 +9198 +9828 +9587 +10679 +9477 +10059 +10670 +9928 +9698 +11361 +10119 +9827 +9818 +15949 +10339 +11461 +10239 +9468 +9277 +9347 +9448 +9398 +9438 +9608 +9398 +9517 +10048 +10129 +9748 +9808 +9858 +10009 +9668 +9647 +9747 +9467 +9628 +11601 +10228 +9789 +9899 +9498 +9727 +10951 +9848 +9718 +12293 +19226 +12173 +9799 +9587 +9688 +10279 +9608 +9668 +9738 +9758 +10079 +11341 +9658 +10259 +9658 +9909 +9838 +9488 +9598 +9768 +9999 +10710 +9798 +10109 +9256 +9478 +9588 +9799 +9608 +9588 +9407 +9258 +15810 +9938 +9718 +9468 +9598 +9718 +9428 +9689 +9859 +9818 +9406 +9688 +9629 +9948 +418725 +11121 +10249 +10028 +10159 +9939 +9968 +10490 +9949 +9929 +10088 +10029 +10048 +10028 +10118 +10210 +10360 +9999 +10590 +10410 +10098 +10059 +10348 +10129 +9757 +10108 +10390 +10258 +10179 +9818 +9949 +10079 +11662 +9738 +10059 +10058 +10168 +9858 +10039 +9778 +10569 +10619 +10119 +9949 +9929 +9978 +10099 +9808 +10109 +10220 +9939 +10079 +10198 +10009 +9748 +10259 +19446 +10469 +10029 +8846 +10209 +10259 +9858 +10068 +10188 +10530 +9988 +10169 +10029 +9908 +9918 +9808 +9858 +10308 +10099 +9977 +9807 +10128 +9938 +9979 +9938 +10580 +9749 +9788 +9899 +10238 +10028 +9899 +10650 +9999 +10089 +9928 +10008 +10019 +10039 +9879 +9959 +10320 +10860 +9998 +10039 +10419 +10169 +10249 +9889 +9799 +10168 +9979 +10189 +9999 +9288 +10009 +9849 +8936 +11431 +9959 +10529 +10248 +9989 +10169 +9979 +9868 +9918 +12944 +9928 +10159 +10029 +10239 +9948 +10209 +10239 +10469 +10630 +10109 +21671 +16690 +10690 +9859 +9839 +9057 +8496 +8395 +8626 +10600 +18695 +15379 +10299 +10358 +11340 +9027 +10159 +9839 +9958 +9848 +9867 +8847 +10119 +9967 +8726 +10049 +9929 +10390 +9818 +10449 +11020 +9988 +9969 +10339 +10198 +10079 +9869 +9887 +10069 +10179 +9698 +10220 +9969 +9738 diff --git a/bdd/features/e2e/E2E-002-stop.feature b/bdd/features/e2e/E2E-002-stop.feature index 801d784cf..f8f3f2cbb 100644 --- a/bdd/features/e2e/E2E-002-stop.feature +++ b/bdd/features/e2e/E2E-002-stop.feature @@ -1,49 +1,51 @@ Feature: Stop e2e tests @ci - Scenario: E2E-002 TC-001 Stop instance process after 0s canKeepAlive true + Scenario: E2E-002 TC-001 Send stop, sequence sends keepAlive Given host is running - When sequence "../packages/samples/hello-alice-out.tar.gz" loaded - And wait for "4000" ms - And instance started with arguments "/package/data.json" + When sequence "../packages/reference-apps/can-keep-alive.tar.gz" loaded + And instance started with arguments "SEND_KEEPALIVE" And wait for "4000" ms And get instance health - And instance health is "true" - And send stop message to instance with arguments timeout 0 and canCallKeepAlive "true" - And wait for "4000" ms And get containerId + And instance health is "true" + And send stop message to instance with arguments timeout 5000 and canCallKeepAlive "true" + And wait for "3000" ms + And get instance health + And instance health is "true" + And send stop message to instance with arguments timeout 5000 and canCallKeepAlive "true" + And wait for "3000" ms + And get instance health + And instance health is "true" + And send stop message to instance with arguments timeout 0 and canCallKeepAlive "false" + And wait for "2000" ms And container is closed Then host is still running @ci - Scenario: E2E-002 TC-002 Stop instance process after 0s canKeepAlive false + Scenario: E2E-002 TC-002 Send stop, sequence doesn't send keepAlive Given host is running - When sequence "../packages/samples/hello-alice-out.tar.gz" loaded - And wait for "4000" ms - And instance started with arguments "/package/data.json" + When sequence "../packages/reference-apps/can-keep-alive.tar.gz" loaded + And instance started with arguments "" And wait for "4000" ms - When get instance health + And get instance health And get containerId And instance health is "true" - And send stop message to instance with arguments timeout 0 and canCallKeepAlive "false" - And wait for "4000" ms + And send stop message to instance with arguments timeout 2000 and canCallKeepAlive "true" + And wait for "5000" ms And container is closed Then host is still running @ci - Scenario: E2E-002 TC-003 Stop instance process after 4s canKeepAlive true + Scenario: E2E-002 TC-003 Send stop, sequence send keepAlive Given host is running - When sequence "../packages/samples/hello-alice-out.tar.gz" loaded - And wait for "4000" ms - And instance started with arguments "/package/data.json" + When sequence "../packages/reference-apps/can-keep-alive.tar.gz" loaded + And instance started with arguments "SEND_KEEPALIVE" And wait for "4000" ms And get instance health And get containerId And instance health is "true" - And send stop message to instance with arguments timeout 8000 and canCallKeepAlive "true" - And wait for "4000" ms - And get instance health - And instance health is "true" - And wait for "4000" ms + And send stop message to instance with arguments timeout 0 and canCallKeepAlive "false" + And wait for "2000" ms And container is closed Then host is still running diff --git a/bdd/features/e2e/E2E-003-kill.feature b/bdd/features/e2e/E2E-003-kill.feature index ef66be646..8c99b380e 100644 --- a/bdd/features/e2e/E2E-003-kill.feature +++ b/bdd/features/e2e/E2E-003-kill.feature @@ -15,6 +15,7 @@ Feature: Kill e2e tests And container is closed Then host is still running + # This is a potential edge case so it's currently ignored. @ignore Scenario: E2E-003 TC-002 Kill sequence - kill handler should emit event when executed Given host is running @@ -25,7 +26,7 @@ Feature: Kill e2e tests When get instance health And get containerId And instance health is "true" - Then get event from instance + Then get event "kill-handler-called" from instance When send kill message to instance Then instance response body is "{\"eventName\":\"kill-handler-called\",\"message\":\"\"}" When wait for "8000" ms diff --git a/bdd/features/e2e/E2E-004-event.feature b/bdd/features/e2e/E2E-004-event.feature index eb0578449..fd0faf78a 100644 --- a/bdd/features/e2e/E2E-004-event.feature +++ b/bdd/features/e2e/E2E-004-event.feature @@ -4,14 +4,13 @@ Feature: Event e2e tests Scenario: E2E-004 TC-001 Send test-event through API and get event emitted by sequence Given host is running When sequence "../packages/reference-apps/event-sequence.tar.gz" loaded - And instance started with arguments "20" + And instance started with arguments "10" And wait for "6000" ms And get instance health And get containerId And instance health is "true" And send event "test-event" to instance with message "test message" - And wait for "5000" ms - Then get event from instance + Then wait for event "test-event-response" from instance When wait for "1000" ms Then instance response body is "{\"eventName\":\"test-event-response\",\"message\":\"message from sequence\"}" When wait for "10000" ms @@ -27,7 +26,7 @@ Feature: Event e2e tests And get instance health And get containerId And instance health is "true" - Then get event from instance + Then get event "new-test-event" from instance Then instance response body is "{\"eventName\":\"new-test-event\",\"message\":\"event sent between functions in one sequence\"}" When wait for "10000" ms And container is closed diff --git a/bdd/features/e2e/E2E-010-cli.feature b/bdd/features/e2e/E2E-010-cli.feature new file mode 100644 index 000000000..bf7b0fec7 --- /dev/null +++ b/bdd/features/e2e/E2E-010-cli.feature @@ -0,0 +1,181 @@ +Feature: CLI tests + + @si + Scenario: E2E-010 TC-001 CLI displays help + Given host is running + When I execute CLI with "help" arguments + Then I get a help information + And the exit status is 0 + And host is still running + + @si + Scenario: E2E-010 TC-002 Shows Host load information + Given host is running + When I execute CLI with "host load" arguments + Then I get Host load information + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-003 Pack sequence + Given host is running + When I execute CLI with "pack ../packages/reference-apps/transform-function -o ../packages/reference-apps/transform-function.tar.gz" arguments + Then I get location "../packages/reference-apps/transform-function.tar.gz" of compressed directory + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-004 Send package + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-005 List sequences + Given host is running + When I execute CLI with "seq ls --format json" arguments + Then I get array of information about sequences + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-006 Start sequence + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + Then the exit status is 0 + And I get instance id + And host is still running + + @ci + Scenario: E2E-010 TC-007 Kill instance + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + And I get instance id + Then I kill instance + And host is still running + + @ci + Scenario: E2E-010 TC-008 Delete sequence + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I delete sequence + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-009 Get health from instance + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + And I get instance id + And wait for "6000" ms + Then I get instance health + And host is still running + + # Test E2E-010 TC-010 works but it is ignored, because changes need to be made in CLI to end the displayed stream. + @ignore + Scenario: E2E-010 TC-010 Get log from instance + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + And I get instance id + Then I get instance log + And host is still running + + + @ci + Scenario: E2E-010 TC-011 Send input data to Instance + Given host is running + When I execute CLI with "seq send ../packages/reference-apps/checksum-sequence.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + And I get instance id + Then I send input data "../dist/reference-apps/checksum-sequence/data.json" + And host is still running + + @ci + Scenario: E2E-010 TC-012 Stop instance + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + And I get instance id + Then I stop instance "3000" "false" + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-013 List instances + Given host is running + When I execute CLI with "seq send ../packages/reference-apps/event-sequence-2.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + Then I get list of instances + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-014 Get instance info + Given host is running + When I execute CLI with "seq send ../packages/samples/hello-alice-out.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + Then I get instance info + And the exit status is 0 + And host is still running + + @ci + Scenario: E2E-010 TC-015 Send event + Given host is running + When I execute CLI with "seq send ../packages/reference-apps/event-sequence.tar.gz --format json" arguments + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + Then I get instance info + Then I send an event named "test-event" with event message "test message" to Instance + And the exit status is 0 + And wait for "5000" ms + Then I get event "test-event-response" with event message "{\"eventName\":\"test-event-response\",\"message\":\"message from sequence\"}" from instance + And the exit status is 0 + + And host is still running + + @ci + Scenario: E2E-010 TC-016 Package and send with stdout + Given host is running + When I execute CLI with bash command "$SI pack ../packages/reference-apps/transform-function -c | $SI seq send --format json" + And the exit status is 0 + Then I get Sequence id + Then I start Sequence + And the exit status is 0 + Then I get list of instances + And the exit status is 0 + And host is still running diff --git a/bdd/features/performance-tests/PT-005-ports.feature b/bdd/features/performance-tests/PT-005-ports.feature index b07b25d5e..355973f44 100644 --- a/bdd/features/performance-tests/PT-005-ports.feature +++ b/bdd/features/performance-tests/PT-005-ports.feature @@ -5,10 +5,31 @@ Feature: Ports e2e tests When sequence "../packages/reference-apps/ports-sequence.tar.gz" loaded And instance started with arguments "tcp" And get instance info + And get instance health + And get containerId And start reading "log" stream - And connect to instance on port 17006 - And send data to instance tcp server + And connect to instance on port 17006 using "tcp" server + And send "testMessage" to "tcp" server And wait for "3000" ms And check stream for message sent - And send "null" to tcp server + And send "null" to "tcp" server + And wait for "5000" ms + And container is closed + Then host is still running + + Scenario: PT-005 TC-002 UDP Connection + Given host is running + When sequence "../packages/reference-apps/ports-sequence.tar.gz" loaded + And instance started with arguments "udp" + And get instance info + And get instance health + And get containerId + And start reading "log" stream + And connect to instance on port 17008 using "udp" server + And send "testMessage" to "udp" server + And wait for "3000" ms + And check stream for message sent + And send "null" to "udp" server + And wait for "5000" ms + And container is closed Then host is still running diff --git a/bdd/lib/host-utils.ts b/bdd/lib/host-utils.ts index 691f6925f..ce95121a6 100644 --- a/bdd/lib/host-utils.ts +++ b/bdd/lib/host-utils.ts @@ -4,7 +4,10 @@ import { ChildProcess, spawn } from "child_process"; import { SIGTERM } from "constants"; import { StringDecoder } from "string_decoder"; -const hostExecutableFilePath = "../dist/host/bin/start.js"; +const hostExecutableCommand = process.env.SCRAMJET_SPAWN_TS + ? ["npx", "ts-node", "../packages/sth/src/bin/hub.ts"] + : ["node", "../dist/sth/bin/hub.js"] +; export class HostUtils { hostProcessStopped = false; @@ -45,12 +48,17 @@ export class HostUtils { async spawnHost() { if (this.hostUrl) { console.error("Host is supposedly running at", this.hostUrl); + const hostClient = new HostClient(this.hostUrl); assert.equal( - (await new HostClient(this.hostUrl).getLoadCheck()).status, // TODO: change to version and log it + (await hostClient.getLoadCheck()).status, // TODO: change to version and log it 200, "Remote host doesn't respond" ); + // TODO: Consider this, but needs testing. + // if (process.env.SCRAMJET_TEST_LOG) { + // (await hostClient.getLogStream()).data?.pipe(process.stderr); + // } return Promise.resolve(); } @@ -58,7 +66,7 @@ export class HostUtils { return new Promise((resolve) => { console.error("Spawning host..."); - const command: string[] = ["node", hostExecutableFilePath]; + const command: string[] = hostExecutableCommand; this.host = spawn("/usr/bin/env", command); diff --git a/bdd/lib/utils.ts b/bdd/lib/utils.ts index 9729d36de..74e60aef1 100644 --- a/bdd/lib/utils.ts +++ b/bdd/lib/utils.ts @@ -2,7 +2,7 @@ import * as fs from "fs"; import { strict as assert } from "assert"; import { promisify } from "util"; -import { exec } from "child_process"; +import { exec, spawn } from "child_process"; import { PassThrough, Readable } from "stream"; const lineByLine = require("n-readlines"); @@ -114,3 +114,36 @@ export async function removeFile(filePath: any) { return 0; } } + +export async function getStreamsFromSpawn( + command: string, options: string[], env: NodeJS.ProcessEnv = process.env +): Promise<[string, string, any]> { + + if (process.env.SCRAMJET_TEST_LOG) console.error("Spawning command", command, ...options); + + const child = spawn(command, options, { + env + }); + const [stdout, stderr, statusCode] = await Promise.all([ + streamToString(child.stdout), + streamToString(child.stderr), + new Promise((res, rej) => { + child.on("error", rej); + child.on("exit", res); + }) + ]); + + return [stdout, stderr, statusCode]; + +} + +export async function getStreamsFromSpawnSuccess( + command: string, options: string[], env: NodeJS.ProcessEnv = process.env +): Promise<[string, string]> { + const [stdout, stderr, code] = await getStreamsFromSpawn(command, options, env); + + if (process.env.SCRAMJET_TEST_LOG) console.error("Results", { stdout, stderr }); + if (code) throw new Error(`Non zero exit code: ${code}`); + + return [stdout, stderr]; +} diff --git a/bdd/step-definitions/e2e/cli.ts b/bdd/step-definitions/e2e/cli.ts new file mode 100644 index 000000000..dd9e02290 --- /dev/null +++ b/bdd/step-definitions/e2e/cli.ts @@ -0,0 +1,139 @@ +import { Then, When } from "@cucumber/cucumber"; +import { strict as assert } from "assert"; +import * as fs from "fs"; +import { getStreamsFromSpawn } from "../../lib/utils"; + +const si = process.env.SCRAMJET_SPAWN_TS + ? ["npx", "ts-node", "../packages/cli/src/bin/index.ts"] + : ["node", "../dist/cli/bin"] +; +const formatFlags = ["-L", "--format", "json"]; + +let stdio: [stdout: string, stderr: string, statusCode: any]; +let sequenceId: string; +let instanceId: string; + +When("I execute CLI with bash command {string}", { timeout: 30000 }, async function(cmd: string) { + stdio = await getStreamsFromSpawn("/bin/bash", ["-c", cmd], { ...process.env, SI: si.join(" ") }); +}); +When("I execute CLI with {string} arguments", { timeout: 30000 }, async function(args: string) { + stdio = await getStreamsFromSpawn("/usr/bin/env", si.concat(args.split(" "))); +}); + + +Then("I get a help information", function() { + + assert.equal(stdio[0].includes("Usage:"), true); +}); + +Then("the exit status is {int}", function(status: number) { + if (stdio[2] !== status) { + console.error(stdio); + assert.equal(stdio[2], status); + } + assert.ok(true); +}); + +Then("I get Sequence id", function() { + const seq = JSON.parse(stdio[0].replace("\n", "")); + + sequenceId = seq._id; + assert.equal(typeof sequenceId !== "undefined", true); +}); + +Then("I get location {string} of compressed directory", function(filepath: string) { + + assert.equal(fs.existsSync(filepath), true); +}); + +Then("I get Host load information", function() { + assert.equal(stdio[0].includes("avgLoad:"), true); +}); + +Then("I get array of information about sequences", function() { + const arr = JSON.parse(stdio[0].replace("\n", "")); + + assert.equal(Array.isArray(arr), true); + +}); + +Then("I start Sequence", async function() { + try { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "seq", "start", sequenceId, "-C", "{}", "[]", ...formatFlags]); + if (process.env.SCRAMJET_TEST_LOG) console.error(stdio[0]); + const instance = JSON.parse(stdio[0].replace("\n", "")); + + console.log(instance); + + instanceId = instance._id; + } catch (e) { + console.error(e.stack, stdio); + assert.fail("Error occurred"); + } +}); + +Then("I get instance id", function() { + assert.equal(typeof instanceId !== "undefined", true); +}); + +Then("I kill instance", async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "kill", instanceId, ...formatFlags]); +}); + + +Then("I delete sequence", { timeout: 10000 }, async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "seq", "delete", sequenceId, ...formatFlags]); +}); + +Then("I get instance health", { timeout: 10000 }, async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "health", instanceId, ...formatFlags]); + const msg = JSON.parse(stdio[0].replace("\n", "")); + + assert.equal(typeof msg.healthy !== "undefined", true); +}); + +Then("I get instance log", { timeout: 30000 }, async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "log", instanceId]); +}); + +Then("I send input data {string}", async function(pathToFile: string) { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "input", instanceId, pathToFile, ...formatFlags]); +}); + +Then("I stop instance {string} {string}", async function(timeout: string, canCallKeepAlive: string) { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "stop", instanceId, timeout, canCallKeepAlive, ...formatFlags]); +}); + +Then("I get list of instances", async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "ls", ...formatFlags]); + const sequences = JSON.parse(stdio[0].replace("\n", "")); + + let instanceFound = false; + + for (let i = 0; i < sequences.length; i++) { + const instances = sequences[i].sequence.instances; + + if (instances.includes(instanceId)) + instanceFound = true; + } + assert.equal(instanceFound, true); + +}); + +Then("I get instance info", async function() { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "info", instanceId, ...formatFlags]); + const info = JSON.parse(stdio[0].replace("\n", "")); + const seqId = info.sequenceId; + + assert.equal(seqId, sequenceId); +}); + +When("I send an event named {string} with event message {string} to Instance", async function(eventName: string, eventMsg: string) { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "emit", instanceId, eventName, eventMsg, ...formatFlags]); +}); + +Then("I get event {string} with event message {string} from instance", async function(eventName: string, value: string) { + stdio = await getStreamsFromSpawn("/usr/bin/env", [...si, "inst", "on", instanceId, eventName, ...formatFlags]); + assert.equal(stdio[0].trim(), value); +}); + diff --git a/bdd/step-definitions/e2e/host-steps.ts b/bdd/step-definitions/e2e/host-steps.ts index fc2fa5e87..d46f9f218 100644 --- a/bdd/step-definitions/e2e/host-steps.ts +++ b/bdd/step-definitions/e2e/host-steps.ts @@ -31,7 +31,7 @@ let streams: { [key: string]: Promise } = {}; const actualResponse = () => actualStatusResponse || actualHealthResponse; -BeforeAll(async () => { +BeforeAll({ timeout: 10e3 }, async () => { await hostUtils.spawnHost(); }); @@ -43,6 +43,28 @@ AfterAll(async () => { } }); + +if (process.env.SCRAMJET_TEST_LOG) { + hostClient.client.addLogger({ + ok(result) { + const { + status, statusText, config: { url, method } + } = result; + + // eslint-disable-next-line no-console + console.error("Request ok:", method, url, `status: ${status} ${statusText}`); + }, + error(error) { + const { code, reason: result } = error; + const { status, statusText } = result?.response || {}; + const { url, method } = result?.config || {}; + + // eslint-disable-next-line no-console + console.error(`Request ${method} "${url}" failed with code "${code}" status: ${status} ${statusText}`); + } + }); +} + Before(() => { actualHealthResponse = ""; actualStatusResponse = ""; @@ -230,10 +252,17 @@ When("send event {string} to instance with message {string}", async (eventName, assert.equal(resp?.status, 202); }); -Then("get event from instance", { timeout: 10000 }, async () => { +Then("wait for event {string} from instance", { timeout: 10000 }, async (event: string) => { + const expectedHttpCode = 200; + + actualStatusResponse = await instance?.getNextEvent(event); + assert.equal(actualStatusResponse?.status, expectedHttpCode); +}); + +Then("get event {string} from instance", { timeout: 10000 }, async (event: string) => { const expectedHttpCode = 200; - actualStatusResponse = await instance?.getEvent(); + actualStatusResponse = await instance?.getEvent(event); assert.equal(actualStatusResponse?.status, expectedHttpCode); }); diff --git a/bdd/step-definitions/performance-tests/durability.ts b/bdd/step-definitions/performance-tests/durability.ts index 4e0ee2e07..ff5f7c166 100644 --- a/bdd/step-definitions/performance-tests/durability.ts +++ b/bdd/step-definitions/performance-tests/durability.ts @@ -98,7 +98,7 @@ Then("check every {float} seconds if instances respond for {float} hours", { tim await new Promise(res => setTimeout(res, 5000)); try { - const response = await instance.getEvent(); + const response = await instance.getNextEvent("ok"); if (response.data?.message.asked === hash) { resolve(); diff --git a/bdd/step-definitions/performance-tests/ports.ts b/bdd/step-definitions/performance-tests/ports.ts index 039aee320..d227858a7 100644 --- a/bdd/step-definitions/performance-tests/ports.ts +++ b/bdd/step-definitions/performance-tests/ports.ts @@ -3,8 +3,7 @@ import { InstanceClient, InstanceOutputStream } from "@scramjet/api-client"; import * as net from "net"; import { strict as assert } from "assert"; import { PassThrough, Stream } from "stream"; -import * as crypto from "crypto"; - +import * as dgram from "dgram"; import { CustomWorld } from "../world"; import { URL } from "url"; @@ -25,30 +24,58 @@ When("get instance info", async function(this: CustomWorld) { console.log(this.resources.instanceInfo); }); -When("connect to instance on port {int}", { timeout: 20000 }, async function(this: CustomWorld, internalPort: number) { +When("connect to instance on port {int} using {string} server", { timeout: 20000 }, async function(this: CustomWorld, internalPort: number, givenServer: string) { const instanceInfo = this.resources.instanceInfo; - const port = instanceInfo.ports[internalPort + "/tcp"]; + const host = process.env.SCRAMJET_HOST_URL ? new URL(process.env.SCRAMJET_HOST_URL).hostname : "localhost"; + + if (givenServer === "tcp") { + const port = instanceInfo.ports[internalPort + "/tcp"]; - console.log("Attempting to connect on port", port); + console.log(`Attempting to connect on ${givenServer} port:${port}, host: ${host}`); - this.resources.connection = await new Promise((resolve, reject) => { - const connection = net.createConnection({ - port: instanceInfo.ports[internalPort + "/tcp"], - host: process.env.SCRAMJET_HOST_URL ? new URL(process.env.SCRAMJET_HOST_URL).hostname : "localhost" - }) - .once("connect", () => { - resolve(connection); + this.resources.connection = await new Promise((resolve, reject) => { + const connection = net.createConnection({ + port: instanceInfo.ports[internalPort + "/tcp"], + host: host }) - .once("error", (e) => { - reject(e); + .once("connect", () => { + resolve(connection); + }) + .once("error", (e) => { + reject(e); + }); + }); + } else if (givenServer === "udp") { + this.resources.internalPort = internalPort; + const port = instanceInfo.ports[internalPort + "/udp"]; + + console.log(`Attempting to connect on ${givenServer} port:${port}, host: ${host}`); + + this.resources.client = dgram.createSocket("udp4") + .on("error", (err) => { + console.log(`server error:\n${err.stack}`); }); - }); + } else { + console.log(`Sequence argument not recognized: ${givenServer}`); + } }); +When("send {string} to {string} server", async function(this: CustomWorld, str: string, serverType: string) { + + if (serverType === "tcp") { + this.resources.connection.write(str); + } -When("send data to instance tcp server", async function(this: CustomWorld) { - this.resources.testMessage = crypto.randomBytes(128).toString("hex"); - this.resources.connection.write(this.resources.testMessage); + if (serverType === "udp") { + const client = this.resources.client as dgram.Socket; + const instanceInfo = this.resources.instanceInfo; + const port = instanceInfo.ports[this.resources.internalPort + "/udp"]; + const host = process.env.SCRAMJET_HOST_URL ? new URL(process.env.SCRAMJET_HOST_URL).hostname : "localhost"; + + client.send(str, 0, str.length, port, host, (err) => { + console.log(err?.stack); + }); + } }); When("start reading {string} stream", async function(this: CustomWorld, log: InstanceOutputStream) { @@ -67,7 +94,3 @@ When("check stream for message sent", async function(this: CustomWorld) { str.includes(this.resources.testMessage), null ); }); - -When("send {string} to tcp server", async function(this: CustomWorld, str) { - this.resources.connection.write(str); -}); diff --git a/bdd/tsconfig.json b/bdd/tsconfig.json index bb66b4777..8f2d94944 100644 --- a/bdd/tsconfig.json +++ b/bdd/tsconfig.json @@ -1,6 +1,7 @@ { "compilerOptions": { - "outDir": "./dist" + "outDir": "./dist", + "skipLibCheck": true }, "extends": "../tsconfig.base.json", "include": [ diff --git a/docs/development-guide/scramjet-interface-cli.md b/docs/development-guide/scramjet-interface-cli.md new file mode 100644 index 000000000..d2beb34f1 --- /dev/null +++ b/docs/development-guide/scramjet-interface-cli.md @@ -0,0 +1,199 @@ +Scramjet Interface +=== + +- [Scramjet tool set](#scramjet-tool-set) + - [Download CLI](#download-cli) + - [Download SDK](#download-sdk) +- [CLI base usage](#cli-base-usage) + - [Login](#login) + - [Set up config](#set-up-config) +- [Create a package](#create-a-package) +- [Sequence operations](#sequence-operations) + - [Post](#post) + - [List](#list) + - [Start](#start) + - [Delete](#delete) + - [Seq help](#seq-help) +- [Instance operations](#instance-operations) + - [List inst](#list-inst) + - [Show inst data](#show-inst-data) + - [Stop](#stop) + - [Kill](#kill) + - [Health status](#health-status) + - [Instance events](#instance-events) + - [Inst logs](#inst-logs) +- [Stdio](#stdio) + - [Inst help](#inst-help) +- [CLI troubleshooting](#cli-troubleshooting) + - [linux](#linux) + - [windows](#windows) + - [macos](#macos) + +## Scramjet tool set + +Scramjet platform can be managed by: + +- Web Interface, +- Application Protocol Interface (API), +- Command Line Interface (CLI) + +All of those are providing same level of control but usage differs. + +Scramjet CLI is a tool allowing developer to control sequences and TaaS process from command line of her/his computer. + +### Download CLI + +CLI is not yet published, but it will be obtainable via: + +- pre-compiled binary, +- node package manager - [NPM](npm_link). + +Download the binary: + +```bash +curl 'https://scramjet.sh/install-cli.sh' | bash -s #verify +``` + +Install via npm globally: + +```bash +npm install -g @scramjet/cli #verify +``` + + + +### Download SDK + +Download [Scramjet Software Developer Kit](https://github.com/scramjetorg/transform-hub) (SDK) via GitHub to develop your code comfortably. + +```bash +git clone git@github.com:scramjetorg/transform-hub +``` + +To build the source code you need: + +- [Node.js](https://nodejs.org/en/), +- [NPM](https://www.npmjs.com/get-npm) or other package manager. + +For Windows users we recommend using WSL as terminal emulator or GitBash. Powershell or default windows cmd are currently not supported. + +[Read more about SDK usage.](xxx) + +## CLI base usage + +Once installed CLI is available under command: + +```bash +si +``` + +To shows the list of available commands: + +```bash +si -h +``` + +__Result:__ a list of possible commands: + +```bash +Usage: si [options...] | si [command] [options...] + +General options + +* `-L, --log-level Specify log level (default: "trace")` +* `-a, --api-url Specify base API url (default: "http://127.0.0.1:8000/api/v1")` +* `-h, --help display help for command` + +Commands + +* `pack [options]` +* `host [command] something` +* `config, c [command] configuration file operations` +* `sequence, seq [command] operations on sequence` +* `instance, inst [command] operations on instance` +* `help [command] display help for command` + +Show sequence and instance help by providing --help option after each. + +``` + +### Set up config + +Set STH url: + +```bash +si config apiUrl "http://url.to.host:8000" +``` + + +## Create a package + +Usage: `si pack [options] ` + +Options: + +* `-c, --stdout output to stdout (ignores -o)` +* `-o, --output output path - defaults to dirname` +* `-h, --help display help for command` + +## Sequence operations + +```bash +si seq run [options] [package] [args...] # Uploads a package and immediatelly executes it with given arguments +si seq send [] # send packed and compressed sequence file +si seq list|ls # list the sequences +si seq start [options] # start the sequence +si seq get # get data about the sequence +si seq delete|rm # delete the sequence +si seq help [command] # display help for command +``` + +## Instance operations + +```bash +si inst list|ls # list the instances +si inst kill # kill instance without waiting for unfinished tasks +si inst stop # end instance gracefully waiting for unfinished tasks +si inst status # status data about the instance +si inst health # show the instance health status +si inst info # show info about the instance +si inst invokeEvent|emit [] # send event with eventName and a JSON formatted event payload +si inst event|on [options] # invoke the event by eventName and optionally with message +si inst input [] # send file to input, if file not given the data will be read from stdin +si inst output # show stream on output +si inst log # show instance log +si inst attach # connect to all stdio - stdin, stdout, stderr of a running instance +si inst stdin [] # send file to stdin, if file not given the data will be read from stdin +si inst stderr # show stream on stderr +si inst stdout # show stream on stdout +si inst help [command] # display help for command +``` + +## CLI troubleshooting + +### linux + +| Problem | Possible solution| +| When I run `si` nothing is happening | ... | +| `'si' is not recognized` | ... | + +### windows + +| Problem | Possible solution| +| When I run `si` nothing is happening | ... | +| `'si' is not recognized` | ... | + +### macos + +| Problem | Possible solution| +| When I run `si` nothing is happening | ... | +| `'si' is not recognized` | ... | + diff --git a/docs/development-guide/stream-and-api.md b/docs/development-guide/stream-and-api.md index b84291954..a4f3822fd 100644 --- a/docs/development-guide/stream-and-api.md +++ b/docs/development-guide/stream-and-api.md @@ -292,12 +292,19 @@ cat test.txt | curl -H "Content-Type: application/json" -d "[5001, {2000}]" http curl -H "Content-Type: application/json" -d "{}" http://localhost:8000/api/v1/sequence/status ``` + ### Event Event contains `` with optional ``. ```bash -curl -H "Content-Type: application/json" -d "{'event_name', message}" http://localhost:8000/api/v1/sequence/event +curl -H "Content-Type: application/json" http://localhost:8000/api/v1/sequence/events/ +``` + +Get the event only once: + +```bash +curl -H "Content-Type: application/json" http://localhost:8000/api/v1/sequence/once/ ``` --- diff --git a/packages/api-client/.eslintrc b/packages/api-client/.eslintrc index 513366fcd..a583bf4c9 100644 --- a/packages/api-client/.eslintrc +++ b/packages/api-client/.eslintrc @@ -1 +1,9 @@ -{"parserOptions":{"project": "./tsconfig.json", "tsconfigRootDir":"./packages/api-client"}} +{ + "parserOptions": { + "project": "./tsconfig.json", + "tsconfigRootDir": "./packages/api-client" + }, + "rules": { + "no-console": 2 + } +} diff --git a/packages/api-client/package.json b/packages/api-client/package.json index e674abcbc..296b2b5c9 100644 --- a/packages/api-client/package.json +++ b/packages/api-client/package.json @@ -14,7 +14,7 @@ "postbuild": "yarn prepack" }, "author": "Scramjet ", - "license": "AGPL-3.0", + "license": "MIT", "dependencies": { "@scramjet/logger": "^0.10.0-pre.10", "@scramjet/model": "^0.10.0-pre.10", diff --git a/packages/api-client/src/client-error.ts b/packages/api-client/src/client-error.ts new file mode 100644 index 000000000..dba39ca20 --- /dev/null +++ b/packages/api-client/src/client-error.ts @@ -0,0 +1,50 @@ +import { AxiosError } from "axios"; + +export type ClientErrorCode = + "GENERAL_ERROR" | + "BAD_PARAMETERS" | + "NEED_AUTHENTICATION" | + "NOT_AUTHORIZED" | + "NOT_FOUND" | + "GONE" | + "SERVER_ERROR" | + "REQUEST_ERROR" | + "UNKNOWN_ERROR" | + "CANNOT_CONNECT" | + "INVALID_RESPONSE" + ; + +export class ClientError extends Error { + reason?: AxiosError; + code: string; + + constructor(code: ClientErrorCode, reason?: Error | string, message?: string) { + super(message || (reason instanceof Error ? reason.message : reason)); + + this.code = code; + if (reason instanceof Error) { + this.reason = reason as AxiosError; + } + } + + // eslint-disable-next-line complexity + static from(error: Error | AxiosError, message?: string): ClientError { + if (error instanceof ClientError) { + return error; + } else if ("isAxiosError" in error && error.isAxiosError) { + if (error.code) { + if (error.code === "400") return new this("BAD_PARAMETERS", error, message); + if (error.code === "401") return new this("NEED_AUTHENTICATION", error, message); + if (error.code === "403") return new this("NOT_AUTHORIZED", error, message); + if (error.code === "404") return new this("NOT_FOUND", error, message); + if (error.code === "410") return new this("GONE", error, message); + if (error.code === "ECONNREFUSED") return new this("CANNOT_CONNECT", error, message); + if (+error.code >= 500) return new this("SERVER_ERROR", error, message); + if (+error.code >= 400) return new this("REQUEST_ERROR", error, message); + return new this("UNKNOWN_ERROR", error, `Response code is "${error.code}"`); + } + return new this("CANNOT_CONNECT", error); + } + return new this("GENERAL_ERROR", error); + } +} diff --git a/packages/api-client/src/client-utils.ts b/packages/api-client/src/client-utils.ts index 7042a17c9..2365d9c02 100644 --- a/packages/api-client/src/client-utils.ts +++ b/packages/api-client/src/client-utils.ts @@ -1,115 +1,86 @@ -import axios, { AxiosError, AxiosResponse } from "axios"; +import axios, { AxiosResponse } from "axios"; import { Stream } from "stream"; +import { ClientError } from "./client-error"; +import { Headers, HttpClient, RequestLogger, Response, ResponseStream, SendStreamOptions } from "./types"; -export type Response = { - data?: { [ key: string ]: any }; - status: number | undefined; -}; - -export type ResponseStream = { - data?: Stream; - status: number | undefined; -}; - -export type SendStreamOptions = Partial<{ - type: string; - end: boolean; -}>; - -export type Headers = { - [key: string]: string; -}; - -const logOk = (result: AxiosResponse) => { - if (process.env.SCRAMJET_TEST_LOG) { - const { - status, statusText, config: { url, method } - } = result; +export class ClientUtils implements HttpClient { + apiBase: string = ""; + private log?: RequestLogger; - console.error("Request ok:", method, url, `status: ${status} ${statusText}`); + constructor(apiBase: string) { + this.apiBase = apiBase; } - return result; -}; -const logError = (result: AxiosError) => { - if (process.env.SCRAMJET_TEST_LOG) { - const { status, statusText } = result.response || {}; - const { url, method } = result.config; - console.error("Request failed:", method, url, `status: ${status} ${statusText}`); + public addLogger(logger: RequestLogger) { + this.log = logger; } - return Promise.reject(result); -}; -class ClientUtils { - apiBase: string = ""; + private safeRequest(_resp: Promise): typeof _resp { + const resp = _resp.catch(e => Promise.reject(ClientError.from(e))); - init(apiBase: string) { - this.apiBase = apiBase; - } + if (this.log) { + const log = this.log; + + return resp.then( + res => { log.ok(res); return res; }, + err => { log.error(err); throw err; } + ); + } - private handleError(error: AxiosError) { - return Promise.reject({ - message: error.response?.statusText, - status: error.response?.status - }); + return resp; } async get(url: string): Promise { - return axios.get(`${this.apiBase}/${url}`, { + return this.safeRequest(axios.get(`${this.apiBase}/${url}`, { headers: { Accept: "*/*" } - }) - .then(logOk, logError) - .catch(this.handleError); + })); } async getStream(url: string): Promise { - return axios({ + return this.safeRequest(axios({ method: "GET", url: `${this.apiBase}/${url}`, headers: { Accept: "*/*" }, responseType: "stream" - }) - .then(logOk, logError) + })) .then((d) => { return { status: d.status, data: d.data }; }) - .catch(this.handleError); + ; } async post(url: string, data: any, headers: Headers = {}): Promise { - return axios({ + return this.safeRequest(axios({ method: "POST", url: `${this.apiBase}/${url}`, data, headers - }) - .then(logOk, logError) + })) .then(res => ({ status: res.status, data: res.data })) - .catch(this.handleError); + ; } async delete(url: string): Promise { - return axios.delete( + return this.safeRequest(axios.delete( `${this.apiBase}/${url}`, { headers: { "Content-Type": "application/json" } - }) - .then(logOk, logError) + })) .then((res) => ({ status: res.status })) - .catch(this.handleError); + ; } async sendStream( @@ -133,5 +104,3 @@ class ClientUtils { ); } } - -export const clientUtils = new ClientUtils(); diff --git a/packages/api-client/src/host-client.ts b/packages/api-client/src/host-client.ts index dd25fa386..40a2c377f 100644 --- a/packages/api-client/src/host-client.ts +++ b/packages/api-client/src/host-client.ts @@ -1,73 +1,63 @@ /* eslint-disable @typescript-eslint/no-unused-vars */ +import { Readable } from "stream"; +import { ClientUtils } from "./client-utils"; import { SequenceClient } from "./sequence-client"; -import { clientUtils } from "./client-utils"; -import { ReadStream } from "fs"; -import { AxiosError } from "axios"; +import { ClientProvider } from "./types"; -export class HostClient { +export class HostClient implements ClientProvider { apiBase: string; + client: ClientUtils; - constructor(apiBase: string) { + constructor(apiBase: string, utils = new ClientUtils(apiBase)) { this.apiBase = apiBase.replace(/\/$/, ""); - clientUtils.init(apiBase); + + this.client = utils; } - listSequences() { - return clientUtils.get("sequences"); + async listSequences() { + return await this.client.get("sequences"); } - listInstances() { - return clientUtils.get("instances"); + async listInstances() { + return await this.client.get("instances"); } // TODO: Dedicated log stream for host not yet implemented. - getLogStream() { - return clientUtils.getStream("log"); + async getLogStream() { + return await this.client.getStream("stream/log"); } - async sendSequence(sequencePackage: ReadStream): Promise { - const response = await clientUtils.post("sequence", sequencePackage, { + async sendSequence(sequencePackage: Readable): Promise { + const response = await this.client.post("sequence", sequencePackage, { "content-type": "application/octet-stream" - }).catch((error: AxiosError) => { - return { - ...error.response - }; }); - if (response.data?.error || !response.data?.id) { - console.error(response.data?.error); - throw new Error("Sequence upload failed"); - } - - return SequenceClient.from(response.data?.id); + return SequenceClient.from(response.data?.id, this); } - getSequence(sequenceId: string) { - return clientUtils.get(`sequence/${sequenceId}`); + async getSequence(sequenceId: string) { + return await this.client.get(`sequence/${sequenceId}`); } async deleteSequence(sequenceId: string) { - const response = await clientUtils.delete(`sequence/${sequenceId}`).catch((error: AxiosError) => { - return { - ...error.response - }; - }); + const response = await this.client.delete(`sequence/${sequenceId}`); - if (response.data?.error) { - console.error(response.data?.error); - throw new Error("Sequence delete failed"); - } + return { + data: response.data, + status: response.status + }; } - getInstance(instanceId: string) { - return clientUtils.get(`instance/${instanceId}`); + // REVIEW: move this to InstanceClient..getInfo()? + async getInstanceInfo(instanceId: string) { + return this.client.get(`instance/${instanceId}`); } - getLoadCheck() { - return clientUtils.get("load-check"); + async getLoadCheck() { + return this.client.get("load-check"); } - getVersion() { - return clientUtils.get("version"); + async getVersion() { + return this.client.get("version"); } } diff --git a/packages/api-client/src/index.ts b/packages/api-client/src/index.ts index e9dae6d4b..dd8a07a84 100644 --- a/packages/api-client/src/index.ts +++ b/packages/api-client/src/index.ts @@ -1,4 +1,7 @@ +export { ClientUtils } from "./client-utils"; +export { ClientError, ClientErrorCode } from "./client-error"; export { HostClient } from "./host-client"; -export { SequenceClient } from "./sequence-client"; export { InstanceClient, InstanceInputStream, InstanceOutputStream } from "./instance-client"; -export { Response } from "./client-utils"; +export { SequenceClient } from "./sequence-client"; +export { Response, ResponseStream, RequestLogger, ClientProvider, HttpClient } from "./types"; + diff --git a/packages/api-client/src/instance-client.ts b/packages/api-client/src/instance-client.ts index 17641c571..3b457cf42 100644 --- a/packages/api-client/src/instance-client.ts +++ b/packages/api-client/src/instance-client.ts @@ -1,36 +1,43 @@ -import { clientUtils, Response, ResponseStream, SendStreamOptions } from "./client-utils"; +import { Response, ResponseStream, SendStreamOptions, ClientProvider, HttpClient } from "./types"; import { RunnerMessageCode } from "@scramjet/symbols"; import { EncodedControlMessage } from "@scramjet/types"; import { Stream } from "stream"; import { IDProvider } from "@scramjet/model"; + export type InstanceInputStream = "stdin" | "input"; export type InstanceOutputStream = "stdout" | "stderr" | "output" | "log" export class InstanceClient { private _id: string; private instanceURL: string; + private host: ClientProvider; public get id(): string { return this._id; } - static from(id: string): InstanceClient { - return new this(id); + private get clientUtils(): HttpClient { + return this.host.client; + } + + static from(id: string, host: ClientProvider): InstanceClient { + return new this(id, host); } - private constructor(id: string) { + private constructor(id: string, host: ClientProvider) { + this.host = host; if (!IDProvider.isValid(id)) { - throw new Error("Invalid id."); + throw new Error(`Invalid id: ${id}`); } this._id = id; this.instanceURL = `instance/${this._id}`; - console.log("New instance:", this.id); + } async stop(timeout: number, canCallKeepalive: boolean): Promise { - return clientUtils.post(`${this.instanceURL}/_stop`, [ + return this.clientUtils.post(`${this.instanceURL}/_stop`, [ RunnerMessageCode.STOP, { timeout, canCallKeepalive @@ -38,7 +45,7 @@ export class InstanceClient { } async kill(): Promise { - return clientUtils.post(`${this.instanceURL}/_kill`, [ + return this.clientUtils.post(`${this.instanceURL}/_kill`, [ RunnerMessageCode.KILL, {} ] as EncodedControlMessage); @@ -51,34 +58,53 @@ export class InstanceClient { message }] as EncodedControlMessage; - return clientUtils.post(`${this.instanceURL}/_event`, data); + return this.clientUtils.post(`${this.instanceURL}/_event`, data); + } + + async getNextEvent(eventName: string) { + return this.clientUtils.get(`${this.instanceURL}/once/${eventName}`); + } + + async getEvent(eventName: string) { + return this.clientUtils.get(`${this.instanceURL}/event/${eventName}`); } - async getEvent() { - return clientUtils.get(`${this.instanceURL}/event`); + /** + * Fetches event + * + * @param eventName - event name + * @param previous - return old event if was ever fired + * @returns stream of events from instance + **/ + async getEventStream(eventName: string) { + return this.clientUtils.getStream(`${this.instanceURL}/events/${eventName}`); } async getHealth() { - return clientUtils.get(`${this.instanceURL}/health`); + return this.clientUtils.get(`${this.instanceURL}/health`); } async getStatus() { - return clientUtils.get(`${this.instanceURL}/status`); + return this.clientUtils.get(`${this.instanceURL}/status`); } async getInfo() { - return clientUtils.get(`${this.instanceURL}`); + return this.clientUtils.get(`${this.instanceURL}`); } async getStream(streamId: InstanceOutputStream): Promise { - return clientUtils.getStream(`${this.instanceURL}/${streamId}`); + return this.clientUtils.getStream(`${this.instanceURL}/${streamId}`); } async sendStream(streamId: InstanceInputStream, stream: Stream | string, options?: SendStreamOptions) { - return clientUtils.sendStream(`${this.instanceURL}/${streamId}`, stream, options); + return this.clientUtils.sendStream(`${this.instanceURL}/${streamId}`, stream, options); } async sendInput(stream: Stream | string) { return this.sendStream("input", stream); } + + async sendStdin(stream: Stream | string) { + return this.sendStream("stdin", stream); + } } diff --git a/packages/api-client/src/sequence-client.ts b/packages/api-client/src/sequence-client.ts index ba850b524..6c2eb25a6 100644 --- a/packages/api-client/src/sequence-client.ts +++ b/packages/api-client/src/sequence-client.ts @@ -1,55 +1,56 @@ import { IDProvider } from "@scramjet/model"; -import { clientUtils } from "./client-utils"; +import { ClientError } from "./client-error"; import { InstanceClient } from "./instance-client"; +import { ClientProvider, HttpClient } from "./types"; export class SequenceClient { private _id: string; private sequenceURL: string; + private host: ClientProvider; public get id(): string { return this._id; } - static from(id: string): SequenceClient { - return new this(id); + private get clientUtils(): HttpClient { + return this.host.client; } - private constructor(id: string) { + static from(id: string, host: ClientProvider): SequenceClient { + return new this(id, host); + } + + private constructor(id: string, host: ClientProvider) { if (!IDProvider.isValid(id)) { - throw new Error("Invalid id."); + throw new Error(`Invalid id: ${id}`); } this._id = id; + this.host = host; this.sequenceURL = `sequence/${id}`; - - console.log("New sequence:", this.id); } - async start(appConfig: any, args: any): Promise { - const response = await clientUtils.post( - `${this.sequenceURL}/start`, { - appConfig, - args - } + async start(appConfig: any, args: any): Promise { + const response = await this.clientUtils.post( + `${this.sequenceURL}/start`, { appConfig, args } ); if (response.data?.id) { - return InstanceClient.from(response.data.id); + return InstanceClient.from(response.data.id, this.host); } - - return undefined; + throw new ClientError("INVALID_RESPONSE", "Response did not include instance id."); } async listInstances() { - return clientUtils.get(`${this.sequenceURL}/instances`); + return this.clientUtils.get(`${this.sequenceURL}/instances`); } - async getInstance(id: string) { - return InstanceClient.from(id); + async getInstance(id: string, host: ClientProvider) { + return InstanceClient.from(id, host); } async getInfo() { - return clientUtils.get(this.sequenceURL); + return this.clientUtils.get(this.sequenceURL); } async overwrite() { diff --git a/packages/api-client/src/types/index.ts b/packages/api-client/src/types/index.ts new file mode 100644 index 000000000..e0ec4936d --- /dev/null +++ b/packages/api-client/src/types/index.ts @@ -0,0 +1,46 @@ +import { AxiosResponse } from "axios"; + +import { Stream } from "stream"; +import { ClientError } from "../client-error"; + +export type Response = { + data?: { [ key: string ]: any }; + status: number | undefined; +}; + +export type ResponseStream = { + data?: Stream; + status: number | undefined; +}; + +export type SendStreamOptions = Partial<{ + type: string; + end: boolean; +}>; + +export type Headers = { + [key: string]: string; +}; + +export type RequestLogger = { + ok: (res: AxiosResponse) => void; + error: (res: ClientError) => void; +}; + +export interface HttpClient { + addLogger(logger: RequestLogger): void; + get(url: string): Promise; + getStream(url: string): Promise; + post(url: string, data: any, headers?: Headers): Promise; + delete(url: string): Promise; + sendStream( + url: string, + stream: Stream | string, + options?: SendStreamOptions + ): Promise; +} + +export interface ClientProvider { + client: HttpClient; +} + diff --git a/packages/api-server/src/handlers/stream.ts b/packages/api-server/src/handlers/stream.ts index c8851ae80..6e4fb0c7b 100644 --- a/packages/api-server/src/handlers/stream.ts +++ b/packages/api-server/src/handlers/stream.ts @@ -69,7 +69,7 @@ export function createStreamHandlers(router: SequentialCeroRouter) { router.get(path, async (req, res, next) => { try { const type = checkAccepts(req.headers.accept, text, json); - const data = await getStream(req, stream); + const data = await getStream(req, res, stream); return decorator(data, type, encoding, res); } catch (e) { diff --git a/packages/api-server/src/lib/data-extractors.ts b/packages/api-server/src/lib/data-extractors.ts index d8eda18ce..dbeb7b417 100644 --- a/packages/api-server/src/lib/data-extractors.ts +++ b/packages/api-server/src/lib/data-extractors.ts @@ -1,4 +1,4 @@ -import { StreamInput } from "@scramjet/types"; +import { ParsedMessage, StreamInput } from "@scramjet/types"; import { ServerResponse, IncomingMessage } from "http"; import { Readable, Writable } from "stream"; import { CeroError } from "./definitions"; @@ -21,6 +21,7 @@ export async function getWritable(object: any, req: IncomingMessage, res: Server export async function getStream( req: IncomingMessage, + res: ServerResponse, stream: StreamInput ): Promise { if (!stream) @@ -29,9 +30,9 @@ export async function getStream( else if (typeof (stream as Readable).readable === "boolean") return stream as Readable; else if (stream instanceof Promise) - return getStream(req, await stream); + return getStream(req, res, await stream); else if (typeof stream === "function") - return getStream(req, await stream(req)); + return getStream(req, res, await stream(req as ParsedMessage, res)); throw new CeroError("ERR_FAILED_FETCH_DATA"); } diff --git a/packages/cli/package.json b/packages/cli/package.json index 9a988d26e..3a425d6f8 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -13,7 +13,7 @@ "watch": "tsc -b --watch", "test": "echo no tests yet -- # npm run test:ava", "test:ava": "ava", - "preinstall": "tar --version", + "preinstall": "tar --version 2>&1 >/dev/null", "prepack": "node ../../scripts/publish.js", "postbuild": "yarn prepack" }, @@ -21,7 +21,10 @@ "license": "AGPL-3.0", "dependencies": { "@scramjet/api-client": "0.10.0-pre.10", - "commander": "^8.0.0" + "commander": "^8.0.0", + "minimatch": "^3.0.4", + "scramjet": "^4.35.20", + "tar": "^6.1.0" }, "devDependencies": { "@types/node": "15.12.5", @@ -29,7 +32,9 @@ "ts-node": "^10.0.0", "typedoc": "0.21.2", "typedoc-plugin-markdown": "3.10.2", - "typescript": "^4.3.4" + "typescript": "^4.3.4", + "@types/minimatch": "^3.0.4", + "@types/tar": "^4.0.4" }, "ava": { "extensions": [ diff --git a/packages/cli/src/bin/index.ts b/packages/cli/src/bin/index.ts index 4cb42c980..effef2991 100755 --- a/packages/cli/src/bin/index.ts +++ b/packages/cli/src/bin/index.ts @@ -1,11 +1,34 @@ #!/usr/bin/env ts-node import { Command } from "commander"; +import { ClientError } from "@scramjet/api-client"; // import { version } from "../../package.json"; import { commands } from "../lib/commands/index"; import { getConfig } from "../lib/config"; +const getExitCode = (_err: ClientError) => 1; const program: Command = new Command() as Command; +const errorHandler = (err: ClientError) => { + process.exitCode = getExitCode(err); + const opts = program.opts(); + + if (opts.format === "json") { + console.log(JSON.stringify({ + error: true, + code: err?.code, + stack: opts.log ? err?.stack : undefined, + message: err?.message, + reason: err?.reason?.message + })); + } else { + console.error(err.stack); + if (err.reason) { + console.error("Caused by:"); + console.error(err.reason.stack); + } + } + process.exit(); +}; (async () => { const conf = getConfig(); @@ -16,12 +39,16 @@ const program: Command = new Command() as Command; program // .version(version) .description("https://github.com/scramjetorg/scramjet-sequence-template#dictionary") - .option("-L, --log-level ", "Specify log level", conf.logLevel) + .option("-L, --log", "Logs all API requests in detail", conf.log) .option("-a, --api-url ", "Specify base API url", conf.apiUrl) + .option("-f, --format ", "Specify display formatting: json or pretty", conf.format) .parse(process.argv) .opts() ; -})().catch(e => { - console.error(e.stack); - process.exitCode = e.code || 1; -}); + + await new Promise(res => program.hook("postAction", res)); +})() + .catch(errorHandler); + +process.on("uncaughtException", errorHandler); +process.on("unhandledRejection", errorHandler); diff --git a/packages/cli/src/lib/commands/config.ts b/packages/cli/src/lib/commands/config.ts index 819857594..50a643fd6 100644 --- a/packages/cli/src/lib/commands/config.ts +++ b/packages/cli/src/lib/commands/config.ts @@ -20,7 +20,7 @@ export const config: CommandDefinition = (program) => { configCmd.command("logLevel ") .description("set the hub API url") - .action((value) => setConfigValue("logLevel", value)); + .action((value) => setConfigValue("log", value)); // apiUrl: 'http://127.0.0.1:8000/api/v1', // logLevel: 'trace', diff --git a/packages/cli/src/lib/commands/host.ts b/packages/cli/src/lib/commands/host.ts index 644a694ff..8fd2ce83f 100644 --- a/packages/cli/src/lib/commands/host.ts +++ b/packages/cli/src/lib/commands/host.ts @@ -1,15 +1,10 @@ -import { HostClient } from "@scramjet/api-client"; import { CommandDefinition } from "../../types"; +import { getHostClient } from "../common"; import { displayEntity } from "../output"; export const host: CommandDefinition = (program) => { - let hostClient: HostClient; - - const getHostClient = () => { - return hostClient || (hostClient = new HostClient(program.opts().apiUrl)); - }; const hostCmd = program .command("host [command]") .description("something"); @@ -17,15 +12,16 @@ export const host: CommandDefinition = (program) => { hostCmd .command("version") .description("get version") - .action(async () => displayEntity(program, getHostClient().getVersion())); + .action(async () => displayEntity(program, getHostClient(program).getVersion())); + // response status 500 hostCmd .command("logs") .description("show all logs") - .action(async () => displayEntity(program, getHostClient().getLogStream())); + .action(async () => displayEntity(program, getHostClient(program).getLogStream())); hostCmd .command("load") .description("show load") - .action(async () => displayEntity(program, getHostClient().getLoadCheck())); + .action(async () => displayEntity(program, getHostClient(program).getLoadCheck())); }; diff --git a/packages/cli/src/lib/commands/instance.ts b/packages/cli/src/lib/commands/instance.ts index b3afc7933..f3dd664da 100644 --- a/packages/cli/src/lib/commands/instance.ts +++ b/packages/cli/src/lib/commands/instance.ts @@ -1,4 +1,7 @@ +import { createReadStream } from "fs"; import { CommandDefinition } from "../../types"; +import { attachStdio, getHostClient, getInstance } from "../common"; +import { displayEntity, displayStream } from "../output"; export const instance: CommandDefinition = (program) => { @@ -7,75 +10,118 @@ export const instance: CommandDefinition = (program) => { .alias("inst") .description("operations on instance"); + instanceCmd.command("list") + .alias("ls") + .description("list the instances") + .action(async () => displayEntity(program, getHostClient(program).listInstances())); + instanceCmd.command("kill ") .description("kill instance without waiting for unfinished tasks") - .action((id) => { - console.log("Instance id ", id); + .action(async (id) => { + return displayEntity(program, getInstance(program, id).kill()); }); - instanceCmd.command("stop ") + /** + * @canCallKeepAlive + * if true instance can prolong its lifetime + * if false instance will end after timeout + */ + instanceCmd.command("stop ") .description("end instance gracefully waiting for unfinished tasks") - .action((id) => { - console.log("Instance id ", id); + .action(async (id, timeout) => { + return displayEntity(program, getInstance(program, id).stop(+timeout, true)); }); instanceCmd.command("status ") .description("status data about the instance") - .action((id) => { - console.log("Instance id ", id); - }); + .action(() => console.log("Not implemented")); instanceCmd.command("health ") .description("show the instance health status") .action((id) => { - console.log("Instance id ", id); + return displayEntity(program, getInstance(program, id).getHealth()); }); instanceCmd.command("info ") .description("show info about the instance") - .action((id) => { - console.log("Instance id ", id); - }); + .action(async (id) => displayEntity(program, getHostClient(program).getInstanceInfo(id))); - instanceCmd.command("sendEvent ") - .description("send event with eventName and object|array|function|number") - .action((id) => { - console.log("Instance id ", id); + instanceCmd.command("invokeEvent []") + .alias("emit") + .description("send event with eventName and a JSON formatted event payload") + .action(async (id, eventName, message) => { + const instanceClient = getInstance(program, id); + + return displayEntity(program, instanceClient.sendEvent(eventName, message)); }); - instanceCmd.command("event ") - .description("invoke the event by eventName and optionally with message") - .action((id) => { - console.log("Instance id ", id); + /** + * No eventName. + * Currently there is no event filtering. + * Only the last event instance is returned + */ + instanceCmd.command("event ") + .alias("on") + .option("-s, --stream", "stream the events (the stream will start with last event)") + .option("-n, --next", "wait for the next event occurrence") + .description("get the last event occurence (will wait for the first one if not yet retrieved)") + .action(async (id, event, { next, stream }) => { + if (stream) + return displayStream(program, getInstance(program, id).getEventStream(event)); + if (next) + return displayEntity(program, getInstance(program, id).getNextEvent(event)); + + return displayEntity(program, getInstance(program, id).getEvent(event)); }); - instanceCmd.command("input ") - .description("send stream to input") - .action((id) => { - console.log("Instance id ", id); + instanceCmd.command("input []") + .description("send file to input, if file not given the data will be read from stdin") + .action((id, stream) => { + const instanceClient = getInstance(program, id); + + return displayEntity(program, + instanceClient.sendStdin(stream ? createReadStream(stream) : process.stdin)); }); instanceCmd.command("output ") .description("show stream on output") .action((id) => { - console.log("Instance id ", id); + return displayStream(program, getInstance(program, id).getStream("output")); }); - instanceCmd.command("stdout ") - .description("show stream on stdout") + instanceCmd.command("log ") + .description("show instance log") .action((id) => { - console.log("Instance id ", id); + return displayStream(program, getInstance(program, id).getStream("log")); }); - instanceCmd.command("stdin ") - .description("send stream to stdin") - .action((id) => { - console.log("Instance id ", id); + instanceCmd.command("attach ") + .description("connect to all stdio - stdin, stdout, stderr of a running instance") + .action(async (id) => { + const inst = getInstance(program, id); + + await attachStdio(program, inst); + }); + + instanceCmd.command("stdin []") + .description("send file to stdin, if file not given the data will be read from stdin") + .action((id, stream) => { + const instanceClient = getInstance(program, id); + + return displayEntity(program, + instanceClient.sendStdin(stream ? createReadStream(stream) : process.stdin)); }); instanceCmd.command("stderr ") .description("show stream on stderr") .action((id) => { - console.log("Instance id ", id); + return displayStream(program, getInstance(program, id).getStream("stderr")); }); + + instanceCmd.command("stdout ") + .description("show stream on stdout") + .action((id) => { + return displayStream(program, getInstance(program, id).getStream("stdout")); + }); + }; diff --git a/packages/cli/src/lib/commands/pack.ts b/packages/cli/src/lib/commands/pack.ts index 716b387be..43e0fdf3f 100644 --- a/packages/cli/src/lib/commands/pack.ts +++ b/packages/cli/src/lib/commands/pack.ts @@ -1,7 +1,11 @@ import { CommandDefinition } from "../../types"; +import { packAction } from "../common"; export const pack: CommandDefinition = (program) => { - program.command("pack") - .option("-d, --dry-run", "Do not execute operations") - ; + const packProgram = program + .command("pack ") + .option("-c, --stdout", "output to stdout (ignores -o)") + .option("-o, --output ", "output path - defaults to dirname"); + + packProgram.action(packAction); }; diff --git a/packages/cli/src/lib/commands/sequence.ts b/packages/cli/src/lib/commands/sequence.ts index 49f2a4f4a..b925fc11f 100644 --- a/packages/cli/src/lib/commands/sequence.ts +++ b/packages/cli/src/lib/commands/sequence.ts @@ -1,35 +1,67 @@ +import { SequenceClient } from "@scramjet/api-client"; +import { createReadStream } from "fs"; +import { readFile } from "fs/promises"; import { CommandDefinition } from "../../types"; +import { attachStdio, getHostClient } from "../common"; +import { displayEntity, displayObject } from "../output"; export const sequence: CommandDefinition = (program) => { - const sequenceCmd = program .command("sequence [command]") .alias("seq") .description("operations on sequence"); + + sequenceCmd + .command("run [package] [args...]") + .description("Uploads a package and immediatelly executes it with given arguments") + .option("-d, --detached", "Don't attach to stdio") + .option("-c, --config ", "Appconfig path location") + .action(async (sequencePackage, args) => { + const { config: configPath, detached } = sequenceCmd.opts(); + const config = configPath ? JSON.parse(await readFile(configPath, "utf-8")) : {}; + const seq = await getHostClient(program) + .sendSequence(sequencePackage ? createReadStream(sequencePackage) : process.stdin); + const instance = await seq.start(config, args); + + if (!detached) { + await attachStdio(program, instance); + } + }) + ; + + sequenceCmd.command("send []") + .description("send packed and compressed sequence file") + .action(async (sequencePackage) => + displayObject(program, await getHostClient(program).sendSequence( + sequencePackage ? createReadStream(sequencePackage) : process.stdin + )) + ); + sequenceCmd.command("list") .alias("ls") .description("list the sequences") - .action(() => { - console.log("Not implemented"); - }); + .action(async () => displayEntity(program, getHostClient(program).listSequences())); - sequenceCmd.command("start ") + // args for example '[10000, 2000]' | '["tcp"]' + sequenceCmd.command("start ") .description("start the sequence") - .action(() => { - console.log("Not implemented"); + .option("-c, --config ") + .option("-C, --config-json ") + .action(async (id, args) => { + const { config, configJson } = sequenceCmd.opts(); + const sequenceClient = SequenceClient.from(id, getHostClient(program)); + + return displayObject(program, + await sequenceClient.start(configJson || config ? JSON.parse(configJson || await readFile(config, "utf-8")) : {}, JSON.parse(args))); }); sequenceCmd.command("get ") .description("get data about the sequence") - .action(() => { - console.log("Not implemented"); - }); + .action(async (id) => displayEntity(program, getHostClient(program).getSequence(id))); sequenceCmd.command("delete ") .alias("rm") .description("delete the sequence") - .action(() => { - console.log("Not implemented"); - }); + .action(async (id) => displayEntity(program, getHostClient(program).deleteSequence(id))); }; diff --git a/packages/cli/src/lib/common.ts b/packages/cli/src/lib/common.ts new file mode 100644 index 000000000..b58107ded --- /dev/null +++ b/packages/cli/src/lib/common.ts @@ -0,0 +1,98 @@ +import { InstanceClient, HostClient } from "@scramjet/api-client"; +import { Command } from "commander"; +import { access, readdir } from "fs/promises"; +import { createReadStream, createWriteStream, constants, PathLike } from "fs"; +import { resolve } from "path"; +import { displayEntity } from "./output"; +import { c } from "tar"; +import { StringStream } from "scramjet"; +import { filter as mmfilter } from "minimatch"; + +const { F_OK, R_OK } = constants; + +let hostClient: HostClient; + +export const getHostClient = (program: Command) => { + if (hostClient) return hostClient; + + hostClient = new HostClient(program.opts().apiUrl); + + if (program.opts().log) + hostClient.client.addLogger({ + ok(result) { + const { + status, statusText, config: { url, method } + } = result; + + // eslint-disable-next-line no-console + console.error("Request ok:", method, url, `status: ${status} ${statusText}`); + }, + error(error) { + const { code, reason: result } = error; + const { status, statusText } = result?.response || {}; + const { url, method } = result?.config || {}; + + // eslint-disable-next-line no-console + console.error(`Request ${method} "${url}" failed with code "${code}" status: ${status} ${statusText}`); + } + }); + return hostClient; +}; +export const getInstance = (program: Command, id: string) => InstanceClient.from(id, getHostClient(program)); +export const attachStdio = (program: Command, instance: InstanceClient) => { + return displayEntity( + program, + Promise.all([ + instance.sendStdin(process.stdin), + instance.getStream("stdout").then(out => out.data?.pipe(process.stdout)), + instance.getStream("stderr").then(err => err.data?.pipe(process.stderr)) + ]).then(() => undefined) + ); +}; +export const getIgnoreFunction = async (file: PathLike) => { + try { + await access(file, R_OK); + } catch { + return () => true; + } + + const rules: ReturnType[] = + await StringStream.from(createReadStream(file)) + .lines() + .filter((line:string) => line.substr(0, line.indexOf("#")).trim() === "") + .parse((line: string) => mmfilter(line)) + .catch(() => undefined) + .toArray() + ; + const fakeArr: string[] = []; + + return (f: string) => !rules.find(x => x(f, 0, fakeArr)); +}; +export const packAction = async (directory: string, { stdout, output }: { stdout: boolean, output: string }) => { + const cwd = resolve(process.cwd(), directory); + const target = stdout ? process.stdout : createWriteStream(output + ? resolve(process.cwd(), output) + : `${cwd}.tar.gz`); + const packageLocation = resolve(cwd, "package.json"); + const ignoreLocation = resolve(cwd, ".siignore"); + + await access(packageLocation, F_OK | R_OK); + // TODO: error handling? + // TODO: check package contents? + + const filter = await getIgnoreFunction(ignoreLocation); + const out = c( + { + gzip: true, + cwd, + filter + }, + await readdir(directory) + ) + .pipe(target); + + await new Promise((res, rej) => { + out.on("finish", res); + out.on("error", rej); + }); +}; diff --git a/packages/cli/src/lib/config.ts b/packages/cli/src/lib/config.ts index 33c53e44b..5f31e5c5c 100644 --- a/packages/cli/src/lib/config.ts +++ b/packages/cli/src/lib/config.ts @@ -5,7 +5,8 @@ import { resolve } from "path"; const defaultConfig = { configVersion: 1, apiUrl: "http://127.0.0.1:8000/api/v1", - logLevel: "trace" + log: true, + format: "pretty" }; type Config = typeof defaultConfig; diff --git a/packages/cli/src/lib/output.ts b/packages/cli/src/lib/output.ts index a1ddd31d3..38649e386 100644 --- a/packages/cli/src/lib/output.ts +++ b/packages/cli/src/lib/output.ts @@ -1,4 +1,4 @@ -import { Response } from "@scramjet/api-client"; +import { Response, ResponseStream } from "@scramjet/api-client"; import { Command } from "commander"; function display(type: "json" | "pretty" = "pretty", object: any) { @@ -16,15 +16,28 @@ export async function displayObject(_program: Command, object: any) { display(_program.opts().format, object); } -export async function displayEntity(_program: Command, request: Promise): Promise { - // todo: different displays depending on _program.opts().format +export async function displayStream( + _program: Command, + request: Promise, + output = process.stdout +): Promise { try { const req = await request; - display(_program.opts().format, req.data); + req.data?.pipe(output); + return new Promise((res, rej) => req.data?.on("finish", res).on("error", rej)); } catch (e) { console.error(e && e.stack || e); process.exitCode = e.exitCode || 1; + return Promise.reject(); } } + +export async function displayEntity(_program: Command, request: Promise): Promise { + // todo: different displays depending on _program.opts().format + const req = await request; + + if (!req) return; + display(_program.opts().format, req.data); +} diff --git a/packages/host/src/lib/csi-controller.ts b/packages/host/src/lib/csi-controller.ts index 0088e4274..8891d6793 100644 --- a/packages/host/src/lib/csi-controller.ts +++ b/packages/host/src/lib/csi-controller.ts @@ -2,12 +2,14 @@ import { getRouter } from "@scramjet/api-server"; import { AppError, CommunicationHandler, CSIControllerError, + HostError, MessageUtilities } from "@scramjet/model"; import { CommunicationChannel as CC, CommunicationChannel, RunnerMessageCode, SupervisorMessageCode } from "@scramjet/symbols"; import { - APIRoute, AppConfig, DownstreamStreamsConfig, ExitCode, FunctionDefinition, HandshakeAcknowledgeMessage, - InstanceConfigMessage, Logger, PassThroughStreamsConfig + APIRoute, AppConfig, DownstreamStreamsConfig, EventMessageData, ExitCode, + FunctionDefinition, HandshakeAcknowledgeMessage, ICommunicationHandler, + InstanceConfigMessage, Logger, ParsedMessage, PassThroughStreamsConfig } from "@scramjet/types"; import { ChildProcess, spawn } from "child_process"; import { EventEmitter } from "events"; @@ -16,6 +18,8 @@ import { DataStream } from "scramjet"; import { PassThrough } from "stream"; import { configService, development } from "@scramjet/sth-config"; import { Sequence } from "./sequence"; +import { ServerResponse } from "http"; +import { getLogger } from "@scramjet/logger"; export class CSIController extends EventEmitter { id: string; @@ -41,7 +45,7 @@ export class CSIController extends EventEmitter { private downStreams?: DownstreamStreamsConfig; private upStreams?: PassThroughStreamsConfig; - communicationHandler: CommunicationHandler; + communicationHandler: ICommunicationHandler; logger: Logger; private socketServerPath: string; @@ -50,8 +54,7 @@ export class CSIController extends EventEmitter { sequence: Sequence, appConfig: AppConfig, sequenceArgs: any[] | undefined, - communicationHandler: CommunicationHandler, - logger: Logger + communicationHandler: CommunicationHandler ) { super(); @@ -59,7 +62,7 @@ export class CSIController extends EventEmitter { this.sequence = sequence; this.appConfig = appConfig; this.sequenceArgs = sequenceArgs; - this.logger = logger; + this.logger = getLogger(this); this.communicationHandler = communicationHandler; this.socketServerPath = configService.getConfig().host.socketPath; @@ -254,8 +257,56 @@ export class CSIController extends EventEmitter { // monitoring data router.get("/health", RunnerMessageCode.MONITORING, this.communicationHandler); - router.get("/status", RunnerMessageCode.STATUS, this.communicationHandler); - router.get("/event", RunnerMessageCode.EVENT, this.communicationHandler); + + // We are not able to obtain all necessary information for this endpoint yet, disabling it for now + // router.get("/status", RunnerMessageCode.STATUS, this.communicationHandler); + + const localEmitter = Object.assign( + new EventEmitter(), + { lastEvents: {} } as {lastEvents: {[evname: string]: any}} + ); + + this.communicationHandler.addMonitoringHandler(RunnerMessageCode.EVENT, (data) => { + const event = data[1] as unknown as EventMessageData; + + if (!event.eventName) return; + localEmitter.lastEvents[event.eventName] = event; + localEmitter.emit(event.eventName, event); + }); + + router.upstream("/events/:name", async (req: ParsedMessage, res: ServerResponse) => { + const name = req.params?.name; + + if (!name) throw new HostError("EVENT_NAME_MISSING"); + + const out = new DataStream(); + const handler = (data: any) => res.write(data); + const clean = () => { + this.logger.debug(`Event stream "${name}" disconnected`); + localEmitter.off(name, handler); + }; + + this.logger.debug(`Event stream "${name}" connected`); + localEmitter.on(name, handler); + res.on("error", clean); + res.on("end", clean); + + return out.JSONStringify(); + }); + const awaitEvent = async (req: ParsedMessage): Promise => new Promise(res => { + const name = req.params?.name; + + if (!name) + throw new HostError("EVENT_NAME_MISSING"); + localEmitter.once(name, res); + }); + + router.get("/event/:name", async (req) => { + if (req.params?.name && localEmitter.lastEvents[req.params?.name]) + return localEmitter.lastEvents[req.params?.name]; + return awaitEvent(req); + }); + router.get("/once/:name", awaitEvent); // operations router.op("post", "/_monitoring_rate", RunnerMessageCode.MONITORING_RATE, this.communicationHandler); diff --git a/packages/host/src/lib/host.ts b/packages/host/src/lib/host.ts index b410bd359..58f6af04f 100644 --- a/packages/host/src/lib/host.ts +++ b/packages/host/src/lib/host.ts @@ -161,9 +161,7 @@ export class Host implements IComponent { this.logger.log("Deleting sequence: ", id); - return { - opStatus: (await this.sequencesStore.delete(id)).opStatus - }; + return await this.sequencesStore.delete(id); } async identifyExistingSequences() { @@ -260,7 +258,7 @@ export class Host implements IComponent { async startCSIController(sequence: Sequence, appConfig: AppConfig, sequenceArgs?: any[]): Promise { const communicationHandler = new CommunicationHandler(); const id = IDProvider.generate(); - const csic = new CSIController(id, sequence, appConfig, sequenceArgs, communicationHandler, this.logger); + const csic = new CSIController(id, sequence, appConfig, sequenceArgs, communicationHandler); this.logger.log("New CSIController created: ", id); @@ -299,6 +297,8 @@ export class Host implements IComponent { } getSequence(id: string) { + if (!this.sequencesStore.getById(id)) + throw new HostError("SEQUENCE_IDENTIFICATION_FAILED", "Sequence not found"); return this.sequencesStore.getById(id); } diff --git a/packages/host/src/lib/sequence-store.ts b/packages/host/src/lib/sequence-store.ts index af4956dc0..6dd96e09c 100644 --- a/packages/host/src/lib/sequence-store.ts +++ b/packages/host/src/lib/sequence-store.ts @@ -40,7 +40,7 @@ export class SequenceStore implements ISequenceStore { this.logger.log("New sequence added:", sequence.id); } - async delete(id: string): Promise<{ opStatus: ReasonPhrases, error?: string }> { + async delete(id: string): Promise<{ opStatus: ReasonPhrases, error?: string, id?:string }> { if (!id) { return { opStatus: ReasonPhrases.BAD_REQUEST @@ -77,7 +77,8 @@ export class SequenceStore implements ISequenceStore { this.logger.log("Volume removed:", volumeId); return { - opStatus: ReasonPhrases.OK + opStatus: ReasonPhrases.OK, + id }; } catch (error) { this.logger.error("Error removing sequence!", error); diff --git a/packages/logger/src/lib/get-name.ts b/packages/logger/src/lib/get-name.ts index d530beeda..21338e9c5 100644 --- a/packages/logger/src/lib/get-name.ts +++ b/packages/logger/src/lib/get-name.ts @@ -29,5 +29,7 @@ function getActualName(nameSource: any): string { * @returns the resulting string */ export function getName(item: any) { - return getActualName(item).replace(/\r?\n[\s\S]*$/g, "").replace(/[\s]+/g, ":"); + const id = item?.id ? `:${item.id}` : ""; + + return `${getActualName(item)}${id}`.replace(/\r?\n[\s\S]*$/g, "").replace(/[\s]+/g, ":"); } diff --git a/packages/model/src/stream-handler.ts b/packages/model/src/stream-handler.ts index 29605b4f0..c51e28eed 100644 --- a/packages/model/src/stream-handler.ts +++ b/packages/model/src/stream-handler.ts @@ -11,6 +11,7 @@ import { MessageDataType, MonitoringMessageCode, MonitoringMessageHandler, + MutatingMonitoringMessageHandler, PassThoughStream, UpstreamStreamsConfig, WritableStream @@ -20,7 +21,7 @@ import { DataStream, StringStream } from "scramjet"; import { PassThrough, Readable, Writable } from "stream"; export type ConfiguredMessageHandler = { - handler: MonitoringMessageHandler + handler: MutatingMonitoringMessageHandler blocking: boolean } | { handler: ControlMessageHandler @@ -231,7 +232,7 @@ export class CommunicationHandler implements ICommunicationHandler { addMonitoringHandler( _code: T, - handler: MonitoringMessageHandler, + handler: MonitoringMessageHandler | MutatingMonitoringMessageHandler, blocking: boolean = false ): this { this.monitoringHandlerHash[_code].push({ diff --git a/packages/pre-runner/package.json b/packages/pre-runner/package.json index 837f29bb3..71a2c0e83 100644 --- a/packages/pre-runner/package.json +++ b/packages/pre-runner/package.json @@ -11,7 +11,7 @@ "postbuild": "yarn prepack", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "AGPL-3.0", "repository": { "type": "git", diff --git a/packages/reference-apps/big-file-sequence/package.json b/packages/reference-apps/big-file-sequence/package.json index 2a58bcce5..a6dd0123b 100644 --- a/packages/reference-apps/big-file-sequence/package.json +++ b/packages/reference-apps/big-file-sequence/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/can-keep-alive/.eslintrc b/packages/reference-apps/can-keep-alive/.eslintrc new file mode 100644 index 000000000..8565a0c50 --- /dev/null +++ b/packages/reference-apps/can-keep-alive/.eslintrc @@ -0,0 +1 @@ +{"parserOptions":{"project": "./tsconfig.json", "tsconfigRootDir":"./packages/reference-apps/can-keep-alive"}} diff --git a/packages/reference-apps/can-keep-alive/data.json b/packages/reference-apps/can-keep-alive/data.json new file mode 100644 index 000000000..f146936b7 --- /dev/null +++ b/packages/reference-apps/can-keep-alive/data.json @@ -0,0 +1,72 @@ +[ + { + "name":"Alice", + "age":31, + "city":"New York" + }, + { + "name":"Ada", + "age":31, + "city":"New York" + }, + { + "name":"Aga", + "age":31, + "city":"New York" + }, + { + "name":"Michał", + "age":31, + "city":"New York" + }, + { + "name":"Maciek", + "age":31, + "city":"New York" + }, + { + "name":"Marcin", + "age":31, + "city":"New York" + }, + { + "name":"Patryk", + "age":31, + "city":"New York" + }, + { + "name":"Rafał", + "age":31, + "city":"New York" + }, + { + "name":"Aida", + "age":31, + "city":"New York" + }, + { + "name":"Basia", + "age":31, + "city":"New York" + }, + { + "name":"Natalia", + "age":31, + "city":"New York" + }, + { + "name":"Monika", + "age":31, + "city":"New York" + }, + { + "name":"Wojtek", + "age":31, + "city":"New York" + }, + { + "name":"Arek", + "age":31, + "city":"New York" + } + ] diff --git a/packages/reference-apps/can-keep-alive/package.json b/packages/reference-apps/can-keep-alive/package.json new file mode 100644 index 000000000..7952b6174 --- /dev/null +++ b/packages/reference-apps/can-keep-alive/package.json @@ -0,0 +1,26 @@ +{ + "name": "@scramjet/can-keep-alive", + "version": "0.10.0-pre.10", + "description": "", + "main": "index.js", + "scripts": { + "build:refapps": "tsc -p tsconfig.json", + "postbuild:refapps": "yarn prepack", + "prepack": "node ../../../scripts/publish.js", + "packseq": "node ../../../scripts/packsequence.js", + "clean": "rm -rf ./dist .bic_cache" + }, + "author": "Scramjet ", + "license": "ISC", + "devDependencies": { + "@scramjet/types": "0.10.0-pre.10", + "@types/node": "15.12.5" + }, + "dependencies": { + "scramjet": "4.35.20" + }, + "repository": { + "type": "git", + "url": "https://github.com/scramjetorg/transform-hub.git" + } +} diff --git a/packages/reference-apps/can-keep-alive/src/bin/run-local b/packages/reference-apps/can-keep-alive/src/bin/run-local new file mode 100644 index 000000000..6ac4667e6 --- /dev/null +++ b/packages/reference-apps/can-keep-alive/src/bin/run-local @@ -0,0 +1,9 @@ +#!/usr/bin/env ts-node + +import mod from "../index"; +const { PassThrough } = require("stream"); + + +// here I call the function from index.ts out + +mod(new PassThrough(), "./data.json", "./dataOut.txt"); diff --git a/packages/reference-apps/can-keep-alive/src/index.ts b/packages/reference-apps/can-keep-alive/src/index.ts new file mode 100644 index 000000000..82de8065f --- /dev/null +++ b/packages/reference-apps/can-keep-alive/src/index.ts @@ -0,0 +1,13 @@ +import { InertApp } from "@scramjet/types"; + +const mod = async function(_stream, ...args) { + if (args[0] === "SEND_KEEPALIVE") { + this.addStopHandler(() => { + this.keepAlive(parseInt(args[1], 10)); + }); + } + + await new Promise(res => setTimeout(res, 60000)); +} as InertApp; + +export default mod; diff --git a/packages/reference-apps/can-keep-alive/tsconfig.build.json b/packages/reference-apps/can-keep-alive/tsconfig.build.json new file mode 100644 index 000000000..e0848624e --- /dev/null +++ b/packages/reference-apps/can-keep-alive/tsconfig.build.json @@ -0,0 +1,9 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "outDir": "./dist" + }, + "include": [ + "src/**/*" + ] +} diff --git a/packages/reference-apps/can-keep-alive/tsconfig.json b/packages/reference-apps/can-keep-alive/tsconfig.json new file mode 100644 index 000000000..aa87dd9ab --- /dev/null +++ b/packages/reference-apps/can-keep-alive/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "outDir": "./dist" + }, + "extends": "../../../tsconfig.base.json", + "include": [ + "./src", + "./test" + ], + "exclude": [ + "node_modules" + ], + "typedocOptions": { + "entryPoints": ["index.ts"], + "out": "../../docs/model" + } +} diff --git a/packages/reference-apps/checksum-sequence/package.json b/packages/reference-apps/checksum-sequence/package.json index 550b045ca..664288999 100644 --- a/packages/reference-apps/checksum-sequence/package.json +++ b/packages/reference-apps/checksum-sequence/package.json @@ -13,7 +13,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/durability-preservation/package.json b/packages/reference-apps/durability-preservation/package.json index 0fb4953eb..d18240778 100644 --- a/packages/reference-apps/durability-preservation/package.json +++ b/packages/reference-apps/durability-preservation/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/event-sequence-2/package.json b/packages/reference-apps/event-sequence-2/package.json index 2d374ec45..85a3ff4f3 100644 --- a/packages/reference-apps/event-sequence-2/package.json +++ b/packages/reference-apps/event-sequence-2/package.json @@ -10,7 +10,7 @@ "build:refapps": "tsc -p tsconfig.json", "postbuild:refapps": "yarn prepack" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/event-sequence/package.json b/packages/reference-apps/event-sequence/package.json index fceb511bc..dc3e4e42c 100644 --- a/packages/reference-apps/event-sequence/package.json +++ b/packages/reference-apps/event-sequence/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/healthy-sequence/package.json b/packages/reference-apps/healthy-sequence/package.json index a91c54bd5..43ef4eb78 100644 --- a/packages/reference-apps/healthy-sequence/package.json +++ b/packages/reference-apps/healthy-sequence/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/inert-function-js/package.json b/packages/reference-apps/inert-function-js/package.json index 2900bbdd6..4c9264a8b 100644 --- a/packages/reference-apps/inert-function-js/package.json +++ b/packages/reference-apps/inert-function-js/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/inert-function/package.json b/packages/reference-apps/inert-function/package.json index 08f16ee6f..b16e2ea7b 100644 --- a/packages/reference-apps/inert-function/package.json +++ b/packages/reference-apps/inert-function/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/inert-sequence-1/package.json b/packages/reference-apps/inert-sequence-1/package.json index 50f890f5d..62072efaf 100644 --- a/packages/reference-apps/inert-sequence-1/package.json +++ b/packages/reference-apps/inert-sequence-1/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/reference-inert-function": "^0.10.0-pre.10", diff --git a/packages/reference-apps/inert-sequence-2-with-delay/package.json b/packages/reference-apps/inert-sequence-2-with-delay/package.json index 0ecae0fd1..6e7051c28 100644 --- a/packages/reference-apps/inert-sequence-2-with-delay/package.json +++ b/packages/reference-apps/inert-sequence-2-with-delay/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/inert-sequence-2/package.json b/packages/reference-apps/inert-sequence-2/package.json index d74a5a9e1..93ae13a77 100644 --- a/packages/reference-apps/inert-sequence-2/package.json +++ b/packages/reference-apps/inert-sequence-2/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/inert-sequence-3/package.json b/packages/reference-apps/inert-sequence-3/package.json index 69e61ac51..6a465310f 100644 --- a/packages/reference-apps/inert-sequence-3/package.json +++ b/packages/reference-apps/inert-sequence-3/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/multi-outputs/.eslintrc b/packages/reference-apps/multi-outputs/.eslintrc new file mode 100644 index 000000000..d05f9653f --- /dev/null +++ b/packages/reference-apps/multi-outputs/.eslintrc @@ -0,0 +1 @@ +{"parserOptions":{"project": "./tsconfig.json", "tsconfigRootDir":"./packages/reference-apps/multi-outputs"}} diff --git a/packages/reference-apps/multi-outputs/index.ts b/packages/reference-apps/multi-outputs/index.ts new file mode 100644 index 000000000..4f799b17d --- /dev/null +++ b/packages/reference-apps/multi-outputs/index.ts @@ -0,0 +1,39 @@ +import { ReadableApp } from "@scramjet/types"; +import { PassThrough } from "stream"; + +/** + * Mutli output application. + * + * @param _stream - dummy input stream + */ +export = async function(_stream) { + const ps = new PassThrough(); + + let cnt = 0; + let cnt2 = 0; + + setInterval(async () => { + // stdout + console.log(cnt); + + // log + this.logger.log(cnt); + + if (cnt === 0) { + cnt = 60; + + // stderr + console.error("Error stream test", cnt); + } + + cnt--; + }, 1000); + + setInterval(async () => { + // output + ps.write(cnt2 + "\n"); + cnt2++; + }, 500); + + return ps; +} as ReadableApp; diff --git a/packages/reference-apps/multi-outputs/package.json b/packages/reference-apps/multi-outputs/package.json new file mode 100644 index 000000000..4a6f22220 --- /dev/null +++ b/packages/reference-apps/multi-outputs/package.json @@ -0,0 +1,23 @@ +{ + "name": "@scramjet/multi-outputs", + "version": "0.10.0-pre.10", + "description": "", + "main": "index.js", + "scripts": { + "build:refapps": "tsc -p tsconfig.json", + "postbuild:refapps": "yarn prepack", + "prepack": "node ../../../scripts/publish.js", + "packseq": "node ../../../scripts/packsequence.js", + "clean": "rm -rf ./dist .bic_cache" + }, + "author": "Scramjet ", + "license": "ISC", + "devDependencies": { + "@scramjet/types": "0.10.0-pre.10", + "@types/node": "15.12.5" + }, + "repository": { + "type": "git", + "url": "https://github.com/scramjetorg/transform-hub.git" + } +} diff --git a/packages/reference-apps/multi-outputs/tsconfig.json b/packages/reference-apps/multi-outputs/tsconfig.json new file mode 100644 index 000000000..8434ab2f2 --- /dev/null +++ b/packages/reference-apps/multi-outputs/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "outDir": "./dist" + }, + "extends": "../../../tsconfig.base.json", + "include": [ + "./index.ts" + ], + "exclude": [ + "node_modules" + ] +} diff --git a/packages/reference-apps/output-streams/package.json b/packages/reference-apps/output-streams/package.json index 3fafaaf03..ee3eb1f64 100644 --- a/packages/reference-apps/output-streams/package.json +++ b/packages/reference-apps/output-streams/package.json @@ -10,7 +10,7 @@ "build:refapps": "tsc -p tsconfig.json", "postbuild:refapps": "yarn prepack" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/ports-sequence/index.ts b/packages/reference-apps/ports-sequence/index.ts index 03df1e4b3..c1cc73abf 100644 --- a/packages/reference-apps/ports-sequence/index.ts +++ b/packages/reference-apps/ports-sequence/index.ts @@ -4,7 +4,8 @@ import { Logger, ReadableApp } from "@scramjet/types"; import { Server } from "net"; import { Socket } from "dgram"; -const ports = [17006, 17007, 17008, 17009]; +const portsTCP = [17006, 17007]; +const portsUDP = [17008, 17009]; const servers: (Server | Socket)[] = []; const output = new PassThrough(); const dgram = require("dgram"); @@ -16,7 +17,7 @@ const createTCPServers = (logger: Logger): (Server | Socket)[] => { let server; - ports.forEach(function(port) { + portsTCP.forEach(function(port) { server = net.createServer(); server.on("close", function() { @@ -63,7 +64,7 @@ const createTCPServers = (logger: Logger): (Server | Socket)[] => { }; const createUDPServers = (logger: Logger): (Server | Socket)[] => { - ports.forEach(function(port) { + portsUDP.forEach(function(port) { const server = dgram.createSocket("udp4"); diff --git a/packages/reference-apps/ports-sequence/package.json b/packages/reference-apps/ports-sequence/package.json index 7b27dcfca..60e576135 100644 --- a/packages/reference-apps/ports-sequence/package.json +++ b/packages/reference-apps/ports-sequence/package.json @@ -9,8 +9,8 @@ "ports": [ "17006/tcp", "17007/tcp", - "17008/tcp", - "17009/tcp" + "17008/udp", + "17009/udp" ] } }, @@ -21,7 +21,7 @@ "build:refapps": "tsc -p tsconfig.json", "postbuild:refapps": "yarn prepack" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/read-function/package.json b/packages/reference-apps/read-function/package.json index 83159be63..0d65feb06 100644 --- a/packages/reference-apps/read-function/package.json +++ b/packages/reference-apps/read-function/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/read-sequence-1/package.json b/packages/reference-apps/read-sequence-1/package.json index a24d4517a..56ac8344a 100644 --- a/packages/reference-apps/read-sequence-1/package.json +++ b/packages/reference-apps/read-sequence-1/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/read-sequence-2/package.json b/packages/reference-apps/read-sequence-2/package.json index af91ad15e..08835932b 100644 --- a/packages/reference-apps/read-sequence-2/package.json +++ b/packages/reference-apps/read-sequence-2/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/read-sequence-3/package.json b/packages/reference-apps/read-sequence-3/package.json index 5aa7fd4e8..9fe9646f6 100644 --- a/packages/reference-apps/read-sequence-3/package.json +++ b/packages/reference-apps/read-sequence-3/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/sequence-20s-kill-handler/package.json b/packages/reference-apps/sequence-20s-kill-handler/package.json index 8ed2870cf..5f43ef6e7 100644 --- a/packages/reference-apps/sequence-20s-kill-handler/package.json +++ b/packages/reference-apps/sequence-20s-kill-handler/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/sequence-20s/package.json b/packages/reference-apps/sequence-20s/package.json index 31ef0bad9..0e890e18e 100644 --- a/packages/reference-apps/sequence-20s/package.json +++ b/packages/reference-apps/sequence-20s/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/sequence-with-error/package.json b/packages/reference-apps/sequence-with-error/package.json index 50851ff01..436269d48 100644 --- a/packages/reference-apps/sequence-with-error/package.json +++ b/packages/reference-apps/sequence-with-error/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/stdio-sequence/index.ts b/packages/reference-apps/stdio-sequence/index.ts index bda5ffc4c..266572569 100644 --- a/packages/reference-apps/stdio-sequence/index.ts +++ b/packages/reference-apps/stdio-sequence/index.ts @@ -24,7 +24,7 @@ module.exports = async function(_stream: any) { this.logger.info("wrote", num, stream, wrote); if (!wrote) - await new Promise(res => process.stdout.once("drain", res)); + await new Promise(res => process[stream].once("drain", res)); } ) .run() diff --git a/packages/reference-apps/stdio-sequence/package.json b/packages/reference-apps/stdio-sequence/package.json index 71292aca8..615ff5c24 100644 --- a/packages/reference-apps/stdio-sequence/package.json +++ b/packages/reference-apps/stdio-sequence/package.json @@ -10,7 +10,7 @@ "build:refapps": "tsc -p tsconfig.json", "postbuild:refapps": "yarn prepack" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/transform-function/package.json b/packages/reference-apps/transform-function/package.json index f13c019a2..0e24b3a87 100644 --- a/packages/reference-apps/transform-function/package.json +++ b/packages/reference-apps/transform-function/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/transform-sequence-1/package.json b/packages/reference-apps/transform-sequence-1/package.json index 968b1bf9c..9ce391dff 100644 --- a/packages/reference-apps/transform-sequence-1/package.json +++ b/packages/reference-apps/transform-sequence-1/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/transform-sequence-2/package.json b/packages/reference-apps/transform-sequence-2/package.json index 1ed9881e5..c7dd7d3fe 100644 --- a/packages/reference-apps/transform-sequence-2/package.json +++ b/packages/reference-apps/transform-sequence-2/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/transform-sequence-3/package.json b/packages/reference-apps/transform-sequence-3/package.json index 9a1b38e0b..6e0b43077 100644 --- a/packages/reference-apps/transform-sequence-3/package.json +++ b/packages/reference-apps/transform-sequence-3/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/unhealthy-sequence/package.json b/packages/reference-apps/unhealthy-sequence/package.json index d60391320..68a200acc 100644 --- a/packages/reference-apps/unhealthy-sequence/package.json +++ b/packages/reference-apps/unhealthy-sequence/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/write-function/package.json b/packages/reference-apps/write-function/package.json index 1abc8458f..0e20e1f78 100644 --- a/packages/reference-apps/write-function/package.json +++ b/packages/reference-apps/write-function/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/write-sequence-1/package.json b/packages/reference-apps/write-sequence-1/package.json index 138d8ed9a..d45501cf2 100644 --- a/packages/reference-apps/write-sequence-1/package.json +++ b/packages/reference-apps/write-sequence-1/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/write-sequence-2/package.json b/packages/reference-apps/write-sequence-2/package.json index abe43a594..984aa54fe 100644 --- a/packages/reference-apps/write-sequence-2/package.json +++ b/packages/reference-apps/write-sequence-2/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/reference-apps/write-sequence-3/package.json b/packages/reference-apps/write-sequence-3/package.json index d0692f16e..d031412e1 100644 --- a/packages/reference-apps/write-sequence-3/package.json +++ b/packages/reference-apps/write-sequence-3/package.json @@ -10,7 +10,7 @@ "packseq": "node ../../../scripts/packsequence.js", "clean": "rm -rf ./dist .bic_cache" }, - "author": "", + "author": "Scramjet ", "license": "ISC", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/runner/package.json b/packages/runner/package.json index 0b2ca9aac..952cfa932 100644 --- a/packages/runner/package.json +++ b/packages/runner/package.json @@ -17,8 +17,8 @@ "prepack": "node ../../scripts/publish.js", "postbuild": "yarn prepack" }, - "author": "", - "license": "AGPL-3.0", + "author": "Scramjet ", + "license": "MIT", "dependencies": { "@scramjet/logger": "^0.10.0-pre.10", "@scramjet/model": "^0.10.0-pre.10", diff --git a/packages/samples/currency-js/index.js b/packages/samples/currency-js/index.js new file mode 100644 index 000000000..e90fac4e4 --- /dev/null +++ b/packages/samples/currency-js/index.js @@ -0,0 +1,18 @@ +const { DataStream } = require("scramjet"); +const fetch = require("node-fetch"); + +module.exports = function(_stream, apikey, fr, to) { + const idx = `${fr}_${to}`; + const get = () => fetch(`https://free.currconv.com/api/v7/convert?q=${idx}&compact=ultra&apiKey=${apikey}`).then(r => r.json()); + const defer = (t = 10000) => new Promise((res) => setTimeout(res, t)); + + return DataStream + .from(async function*() { + while (true) + yield await Promise.all([ + get(), defer() + ]).then(([data]) => data); + }) + .do(async x => { console.log(x[idx]); }) // add some logic here + .run(); +}; diff --git a/packages/samples/currency-js/package.json b/packages/samples/currency-js/package.json new file mode 100644 index 000000000..7181555a9 --- /dev/null +++ b/packages/samples/currency-js/package.json @@ -0,0 +1,15 @@ +{ + "name": "@scramjet/currency-js", + "version": "0.10.0-pre.10", + "main": "./index", + "author": "Scramjet ", + "license": "GPL-3.0", + "dependencies": { + "node-fetch": "^2.6.1", + "scramjet": "4.35.20" + }, + "repository": { + "type": "git", + "url": "https://github.com/scramjetorg/transform-hub.git" + } +} diff --git a/packages/samples/example/package.json b/packages/samples/example/package.json index b1fca235b..61748714f 100644 --- a/packages/samples/example/package.json +++ b/packages/samples/example/package.json @@ -25,7 +25,7 @@ "--test": false } }, - "author": "Signicode ", + "author": "Scramjet ", "license": "GPL-3.0", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/samples/example2/package.json b/packages/samples/example2/package.json index 1cc5f5448..bef7db774 100644 --- a/packages/samples/example2/package.json +++ b/packages/samples/example2/package.json @@ -19,7 +19,7 @@ "scramjet": { "image": "node" }, - "author": "Signicode ", + "author": "Scramjet ", "license": "GPL-3.0", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/samples/hello-alice-out/package.json b/packages/samples/hello-alice-out/package.json index a1b7562a7..adad3a73d 100644 --- a/packages/samples/hello-alice-out/package.json +++ b/packages/samples/hello-alice-out/package.json @@ -21,7 +21,7 @@ "scramjet": { "image": "node" }, - "author": "Signicode ", + "author": "Scramjet ", "license": "GPL-3.0", "devDependencies": { "@scramjet/types": "0.10.0-pre.10", diff --git a/packages/sth-config/package.json b/packages/sth-config/package.json index 0a1e671bf..123b29cad 100644 --- a/packages/sth-config/package.json +++ b/packages/sth-config/package.json @@ -2,7 +2,7 @@ "name": "@scramjet/sth-config", "version": "0.10.0-pre.10", "description": "> TODO: description", - "author": "Budleigh Salterton ", + "author": "Scramjet ", "homepage": "https://github.com/scramjetorg/transform-hub#readme", "license": "ISC", "main": "src/index.ts", diff --git a/packages/sth/package.json b/packages/sth/package.json index ae105bac4..a5577bba0 100644 --- a/packages/sth/package.json +++ b/packages/sth/package.json @@ -4,7 +4,8 @@ "description": "Scramjet STH Excecutable", "main": "src/lib/index.ts", "bin": { - "scramjet-transform-hub": "src/bin/hub.ts" + "scramjet-transform-hub": "src/bin/hub.ts", + "sth": "src/bin/hub.ts" }, "scripts": { "start": "ts-node ./src/index", diff --git a/packages/supervisor/src/bin/supervisor.ts b/packages/supervisor/src/bin/supervisor.ts index 339bff663..ebe62031e 100644 --- a/packages/supervisor/src/bin/supervisor.ts +++ b/packages/supervisor/src/bin/supervisor.ts @@ -30,7 +30,8 @@ lcc.main() exitCode = e.data.exitCode; } - console.log(e.stack); + // eslint-disable-next-line no-console + console.error(e.stack); process.exitCode = exitCode; }, 100); }) diff --git a/packages/test/package.json b/packages/test/package.json index 4c6625bbf..7dc865204 100644 --- a/packages/test/package.json +++ b/packages/test/package.json @@ -23,5 +23,6 @@ "repository": { "type": "git", "url": "https://github.com/scramjetorg/transform-hub.git" - } + }, + "author": "Scramjet " } diff --git a/packages/types/src/api-expose.ts b/packages/types/src/api-expose.ts index 039edf7f1..d1df333b8 100644 --- a/packages/types/src/api-expose.ts +++ b/packages/types/src/api-expose.ts @@ -8,8 +8,10 @@ import { MaybePromise } from "./utils"; export type ParsedMessage = IncomingMessage & { body?: any, params: { [key: string]: any} | undefined}; export type HttpMethod = "get" | "head" | "post" | "put" | "delete" | "connect" | "trace" | "patch"; -export type StreamInput = ((req: IncomingMessage) => MaybePromise) | MaybePromise; -export type StreamOutput = ((req: IncomingMessage, res: ServerResponse) => MaybePromise) | MaybePromise; +export type StreamInput = + ((req: ParsedMessage, res: ServerResponse) => MaybePromise) | MaybePromise; +export type StreamOutput = + ((req: IncomingMessage, res: ServerResponse) => MaybePromise) | MaybePromise; export type GetResolver = (req: ParsedMessage) => MaybePromise; export type OpResolver = (req: ParsedMessage, res?: ServerResponse) => MaybePromise; @@ -75,7 +77,14 @@ export interface APIBase { * @param conn the communication handler to use */ get( - path: string | RegExp, msg: GetResolver | T, conn?: ICommunicationHandler): void; + path: string | RegExp, msg: T, conn: ICommunicationHandler): void; + /** + * Alternative GET request hook with dynamic resolution + * + * @param path the request path as string or regex + * @param op which operation + */ + get(path: string | RegExp, msg: GetResolver): void; /** * A method that allows to pass a stream to the specified path on the API server * diff --git a/packages/types/src/communication-handler.ts b/packages/types/src/communication-handler.ts index d626cfc58..3ff6d1842 100644 --- a/packages/types/src/communication-handler.ts +++ b/packages/types/src/communication-handler.ts @@ -8,6 +8,8 @@ import { import { MaybePromise } from "./utils"; export type MonitoringMessageHandler = + (msg: EncodedMessage) => void; +export type MutatingMonitoringMessageHandler = (msg: EncodedMessage) => MaybePromise | null>; export type ControlMessageHandler = (msg: EncodedMessage) => MaybePromise | null>; @@ -17,8 +19,12 @@ export interface ICommunicationHandler { hookUpstreamStreams(str: UpstreamStreamsConfig): this; hookDownstreamStreams(str: DownstreamStreamsConfig): this; + addMonitoringHandler(code: T, handler: MutatingMonitoringMessageHandler, + blocking: true): this; addMonitoringHandler(code: T, handler: MonitoringMessageHandler, - blocking?: boolean): this; + blocking: false): this; + addMonitoringHandler(code: T, handler: MonitoringMessageHandler): this; + // TODO: we need non-mutating handlers (addControlListener) addControlHandler(code: T, handler: ControlMessageHandler): this; diff --git a/packages/types/src/error-codes/host-error.ts b/packages/types/src/error-codes/host-error.ts index c3451ff6d..2ed015b25 100644 --- a/packages/types/src/error-codes/host-error.ts +++ b/packages/types/src/error-codes/host-error.ts @@ -6,6 +6,7 @@ export type HostErrorCode = "SEQUENCE_IDENTIFICATION_FAILED" | "UNKNOWN_SEQUENCE" | "UNKNOWN_INSTANCE" | + "EVENT_NAME_MISSING" | "CONTROLLER_ERROR" | "SOCKET_TAKEN" | "API_CONFIGURATION_ERROR" diff --git a/scripts/.eslintrc b/scripts/.eslintrc index e65fbd72c..e5ed8b782 100644 --- a/scripts/.eslintrc +++ b/scripts/.eslintrc @@ -1,5 +1,6 @@ { "rules": { + "no-console": 0, "import/no-extraneous-dependencies": 0 } } diff --git a/scripts/dev/start-seq.ts b/scripts/dev/start-seq.ts index ecb546da3..de25883db 100644 --- a/scripts/dev/start-seq.ts +++ b/scripts/dev/start-seq.ts @@ -7,11 +7,9 @@ const host = new HostClient("http://localhost:8000/api/v1"); (async () => { const pkg = createReadStream(resolve(__dirname, "../../packages/samples/hello-alice-out.tar.gz")); const sequence = await host.sendSequence(pkg); - - console.log((await sequence.getInfo()).data); - const instance = await sequence.start({}, ["/package/data.json"]); const instanceInfo = (await instance.getInfo()).data; - console.log(instanceInfo); + // eslint-disable-next-line no-console + console.error(instanceInfo); })(); diff --git a/yarn.lock b/yarn.lock index afc57eab3..8aedeb15a 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1317,7 +1317,7 @@ resolved "https://registry.yarnpkg.com/@types/json5/-/json5-0.0.29.tgz#ee28707ae94e11d2b827bcbe5270bcea7f3e71ee" integrity sha1-7ihweulOEdK4J7y+UnC86n8+ce4= -"@types/minimatch@^3.0.3": +"@types/minimatch@^3.0.3", "@types/minimatch@^3.0.4": version "3.0.4" resolved "https://registry.yarnpkg.com/@types/minimatch/-/minimatch-3.0.4.tgz#f0ec25dbf2f0e4b18647313ac031134ca5b24b21" integrity sha512-1z8k4wzFnNjVK/tlxvrWuK5WMt6mydWWP7+zvH5eFep4oj+UkrfiJTRtjCeBXNpwaA/FYqqtb4/QS4ianFpIRA== @@ -1327,6 +1327,13 @@ resolved "https://registry.yarnpkg.com/@types/minimist/-/minimist-1.2.1.tgz#283f669ff76d7b8260df8ab7a4262cc83d988256" integrity sha512-fZQQafSREFyuZcdWFAExYjBiCL7AUCdgsk80iO0q4yihYYdcIiH28CcuPTGFgLOCC8RlW49GSQxdHwZP+I7CNg== +"@types/minipass@*": + version "2.2.0" + resolved "https://registry.yarnpkg.com/@types/minipass/-/minipass-2.2.0.tgz#51ad404e8eb1fa961f75ec61205796807b6f9651" + integrity sha512-wuzZksN4w4kyfoOv/dlpov4NOunwutLA/q7uc00xU02ZyUY+aoM5PWIXEKBMnm0NHd4a+N71BMjq+x7+2Af1fg== + dependencies: + "@types/node" "*" + "@types/node@*", "@types/node@15.12.5": version "15.12.5" resolved "https://registry.yarnpkg.com/@types/node/-/node-15.12.5.tgz#9a78318a45d75c9523d2396131bd3cca54b2d185" @@ -1359,6 +1366,14 @@ dependencies: "@sinonjs/fake-timers" "^7.1.0" +"@types/tar@^4.0.4": + version "4.0.4" + resolved "https://registry.yarnpkg.com/@types/tar/-/tar-4.0.4.tgz#d680de60855e7778a51c672b755869a3b8d2889f" + integrity sha512-0Xv+xcmkTsOZdIF4yCnd7RkOOyfyqPaqJ7RZFKnwdxfDbkN3eAAE9sHl8zJFqBz4VhxolW9EErbjR1oyH7jK2A== + dependencies: + "@types/minipass" "*" + "@types/node" "*" + "@types/trouter@^3.1.0": version "3.1.0" resolved "https://registry.yarnpkg.com/@types/trouter/-/trouter-3.1.0.tgz#8a3c7b34352b1cb454217a194e8bf03e5cdf7cc4" @@ -6406,7 +6421,7 @@ scramjet-core@^4.31.19: resolved "https://registry.yarnpkg.com/scramjet-core/-/scramjet-core-4.31.19.tgz#ac69687c792abae40ab1a4bedc2cdca7a1f814ec" integrity sha512-/rmtiNw7CY2xLKCm7qBc7RwToltdZ+0MpLaS7XCdPAjMwheKT3+Jbu/r708DryySQypy2IKJQkUx5kx7b2qkeg== -scramjet@4.35.20: +scramjet@4.35.20, scramjet@^4.35.20: version "4.35.20" resolved "https://registry.yarnpkg.com/scramjet/-/scramjet-4.35.20.tgz#acce89a4d4419af3b8dd0d5dde95eaef2d839665" integrity sha512-Lnj3Js/y07VsaQM2vhMkNHl51jRHucnpAOPE55Ug3B6mo3CtPdO+DEpCsaV97u0ZBXcxWIbR9DN4SPftcAprDw==