Move some parts of the README to the colossus sub-package for better

context.

.gitignore

@@ -20,5 +20,8 @@ yarn-error.log*
 .vscode
 .*.sw*
 
+# Node modules
+node_modules/
+
 # Ignore nvm config file
 .nvmrc

README.md

@@ -1,5 +1,14 @@
 ![Storage Nodes for Joystream](./banner.svg)
 
+This repository contains several Node packages, located under the `packages/`
+subdirectory. See each individual package for details:
+
+* [colossus](./packages/colossus/README.md') - the main colossus app.
+* [storage](./packages/storage/README.md) - abstraction over the storage backend.
+* [runtime-api](./packages/runtime-api/README.md) - convenience wrappers for the runtime API.
+* [crypto](./packages/crypto/README.md) - cryptographic utility functions.
+* [util](./packages/util/README.md) - general utility functions.
+
 Installation
 ------------
 
@@ -24,95 +33,38 @@ $ brew install libtool automake autoconf
 *Building*
 
 ```bash
-$ yarn run build
+$ yarn install
 ```
 
-The command will run `yarn install`, perform post-install fixes and build
-TypeScript files.
-
-To make the `js_storage` script available globally, run:
+The command will install dependencies, and make a `colossus` executable available:
 
 ```bash
-$ npm install -g
+$ yarn run colossus --help
 ```
 
-This might be best if used within e.g. an [nvm](https://github.com/creationix/nvm)
-environment. *Note* that it's generally not advisable to mix `yarn` and `npm`, but
-this seems safe enough.
+*Testing*
 
-Development
------------
+Running tests from the repository root will run tests from all packages:
 
-Run a development server:
-
-```bash
-$ yarn run dev --config myconfig.json
 ```
-
-Run tests:
-```bash
 $ yarn run test
 ```
 
-Run linter:
-```bash
-$ yarn run lint
-```
-
-TypeScript files are used for type declarations for interaction with the chain.
-They are taken from [the app](https://github.com/Joystream/apps), and renamed
-to suit our use:
-
-`apps/packages/joy-members/src/types.ts` becomes `lib/joystream/types/members.ts`,
-etc. The only thing adjusted in the files are the imports of other joystream
-types.
-
-`lib/joystream/types/index.js` is manually maintained, but other JavaScript
-files in that folder are generated. The `update_types.sh` script can help with
-keeping these in sync.
-
-You can also switch on development mode with a configuration variable, see
-below.
-
-*Development Mode*
-
-Development mode modifies the storage node's behaviour a little bit, but only
-a little:
-
-1. In development mode, each host discovered with the DHT will *also* be
-   reported as localhost. This allows you to use the DHT resolution mechanism
-   between two locally running storage nodes. Port probing will find out more
-   about on what port they're running which service.
-
-Command-Line
-------------
-
-Running a storage server is (almost) as easy as running the bundled `js_storage`
-executable:
-
-```bash
-$ js_storage --storage=/path/to/storage/directory
-```
-
-Run with `--help` to see a list of available CLI options.
-
-You need to stake as a storage provider to run a storage node.
-
 Storage Provider Staking
 ------------------------
 
 Staking for the storage provider role happens in a few simple steps:
 
 1. Using [the app](https://github.com/Joystream/apps), create an account and make
    it a member. Make sure to save the JSON file. Not only is this account your
-   identity, the file is also needed for the signup process. Make sure thea ccount
-   has some currency. 
+   identity, the file is also needed for the signup process. Make sure the account
+   has some currency.
    - You need some currency to become a member.
    - You need to stake some currency to become a storage provider.
    - There's a transaction fee for applying as a storage provider.
-1. Using the `js_storage` cli, run the signup process:
+1. Using the `colossus` cli, run the signup process:
    ```bash
-   $ js_storage signup MEMBER_ADDRESS.json
+   $ colossus signup MEMBER_ADDRESS.json
    Enter passphrase for MEMBER_ADDRESS: asdf
    Account is working for staking, proceeding.
    Generated  ROLE_ADDRESS - this is going to be exported to a JSON file.
@@ -132,81 +84,10 @@ Staking for the storage provider role happens in a few simple steps:
 1. Stake for the role.
 1. Back with the CLI, run the server:
    ```bash
-   $ js_storage --key-file ROLE_ADDRESS.json
+   $ colossus --key-file ROLE_ADDRESS.json
    ```
 
 Note that the JSON files contain the full key pair of either account. It's best
-to protect them with a passphrase. If you want to run the `js_storage` server
+to protect them with a passphrase. If you want to run the `colossus` server
 more easily, you can also provide an empty passphrase when the JSON file is
 created - but be aware that you must then take extra care to secure this file!
-
-Configuration
--------------
-
-Most common configuration options are available as command-line options
-for the CLI.
-
-However, some advanced configuration options are only possible to set
-via the configuration file.
-
-* `filter` is a hash of upload filtering options.
-  * `max_size` sets the maximum permissible file upload size. If unset,
-    this defaults to 100 MiB.
-  * `mime` is a hash of...
-    * `accept` is an Array of mime types that are acceptable for uploads,
-      such as `text/plain`, etc. Mime types can also be specified for
-      wildcard matching, such as `video/*`.
-    * `reject` is an Array of mime types that are unacceptable for uploads.
-* `development` is a boolean flag. See Development Mode above for details.
-
-Upload Filtering
-----------------
-
-The upload filtering logic first tests whether any of the `accept` mime types
-are matched. If none are matched, the upload is rejected. If any is matched,
-then the upload is still rejected if any of the `reject` mime types are
-matched.
-
-This allows inclusive and exclusive filtering.
-
-* `{ accept: ['text/plain', 'text/html'] }` accepts *only* the two given mime types.
-* `{ accept: ['text/*'], reject: ['text/plain'] }` accepts any `text/*` that is not
-  `text/plain`.
-
-More advanced filtering is currently not available.
-
-API Packages
-------------
-
-Since it's not entirely clear yet how APIs will develop in future, the approach
-taken here is to package individual APIs up individually. That is, instead of
-providing an overall API version in `api-base.yml`, it should be part of each
-API package's path.
-
-For example, for a `foo` API in its version `v1`, its definitions should live
-in `./paths/foo/v1.js` and `./paths/foo/v1/*.js` respectively.
-
-*Note:* until a reasonably stable API is reached, this project uses a `v0`
-version prefix.
-
-Interface/implementation
-------------------------
-
-For reusability across API versions, it's best to keep files in the `paths`
-subfolder very thin, and instead inject implementations via the `dependencies`
-configuration value of `express-openapi`.
-
-These implementations line to the `./lib` subfolder. Adjust `server.js` as
-needed to make them available to API packages.
-
-Streaming Notes
----------------
-
-For streaming content, it is required that stream metadata is located at the
-start of the stream. Most software writes metadata at the end of the stream,
-because it is when the stream is committed to disk that the entirety of the
-metadata is known.
-
-To move metadata to the start of the stream, a CLI tool such as
-[qtfaststart](https://github.com/danielgtaylor/qtfaststart) for MP4 files might
-be used.

package.json

@@ -1,8 +1,7 @@
 {
-  "version": "0.1.0",
   "private": true,
   "engines": {
-    "node": ">=v10.15.3",
+    "node": "^10.15.3",
     "yarn": "^1.15.2"
   },
   "homepage": "https://github.com/Joystream/storage-node-joystream",
@@ -32,5 +31,12 @@
   ],
   "workspaces": [
     "packages/*"
-  ]
+  ],
+  "scripts": {
+    "test": "wsrun --serial test",
+    "lint": "wsrun --serial lint"
+  },
+  "devDependencies": {
+    "wsrun": "^3.6.5"
+  }
 }

packages/colossus/.eslintrc.js

@@ -0,0 +1 @@
+../../.eslintrc.js

packages/colossus/5CpGKFTCdub9TWVD8bpR1kZbwvBrWXJ53idqYNUSuk8hS5hJ.json

@@ -0,0 +1 @@
+{"address":"5CpGKFTCdub9TWVD8bpR1kZbwvBrWXJ53idqYNUSuk8hS5hJ","encoded":"0x3053020101300506032b65700422042058cc8db354ab43e4c318d641c3a8d78b451ff01bf15cf6beea53fa88ec0b40fd2138d116e4cb9e41b9c2012b3d0db4158c06e182cec68f4c5245c8b0f36b5558a1230321002138d116e4cb9e41b9c2012b3d0db4158c06e182cec68f4c5245c8b0f36b5558","encoding":{"content":["pkcs8","ed25519"],"type":"none","version":"2"},"meta":{"name":"storage role account for 5HfS9kwKMUxVpW5JoKnqvHE7tTz32wDN59p3BWRmYykygMK7"}}

packages/colossus/5HfS9kwKMUxVpW5JoKnqvHE7tTz32wDN59p3BWRmYykygMK7.json

@@ -0,0 +1 @@
+{"address":"5HfS9kwKMUxVpW5JoKnqvHE7tTz32wDN59p3BWRmYykygMK7","encoded":"0xbe1f340fe41a972670979ca26c1819558d70454bd3b3ebd19b0a06d30d6d90e0d8dd51e25cd8ee5c8dbd1fa551c3832adb37394e4a3850e3ce2b8a3e6e2c76c84086f86b34349c2e093db334057398b05d1b14c81e2c5e51fb0da7ec0e304211070f8c6ac748e5a287cf06e4ad948469bff075133cf62fc816b7df6dbcf4090e01f72b6bb7cb63781fb8e004b9bead9de46e084abeb38b60b9d97c9ed4","encoding":{"content":["pkcs8","ed25519"],"type":"xsalsa20-poly1305","version":"2"},"meta":{"name":"test","whenCreated":1557912803459}}

packages/colossus/README.md

@@ -0,0 +1,94 @@
+![Storage Nodes for Joystream](../../banner.svg)
+
+Development
+-----------
+
+Run a development server:
+
+```bash
+$ yarn run dev --config myconfig.json
+```
+
+Command-Line
+------------
+
+Running a storage server is (almost) as easy as running the bundled `colossus`
+executable:
+
+```bash
+$ colossus --storage=/path/to/storage/directory
+```
+
+Run with `--help` to see a list of available CLI options.
+
+You need to stake as a storage provider to run a storage node.
+
+Configuration
+-------------
+
+Most common configuration options are available as command-line options
+for the CLI.
+
+However, some advanced configuration options are only possible to set
+via the configuration file.
+
+* `filter` is a hash of upload filtering options.
+  * `max_size` sets the maximum permissible file upload size. If unset,
+    this defaults to 100 MiB.
+  * `mime` is a hash of...
+    * `accept` is an Array of mime types that are acceptable for uploads,
+      such as `text/plain`, etc. Mime types can also be specified for
+      wildcard matching, such as `video/*`.
+    * `reject` is an Array of mime types that are unacceptable for uploads.
+
+Upload Filtering
+----------------
+
+The upload filtering logic first tests whether any of the `accept` mime types
+are matched. If none are matched, the upload is rejected. If any is matched,
+then the upload is still rejected if any of the `reject` mime types are
+matched.
+
+This allows inclusive and exclusive filtering.
+
+* `{ accept: ['text/plain', 'text/html'] }` accepts *only* the two given mime types.
+* `{ accept: ['text/*'], reject: ['text/plain'] }` accepts any `text/*` that is not
+  `text/plain`.
+
+More advanced filtering is currently not available.
+
+API Packages
+------------
+
+Since it's not entirely clear yet how APIs will develop in future, the approach
+taken here is to package individual APIs up individually. That is, instead of
+providing an overall API version in `api-base.yml`, it should be part of each
+API package's path.
+
+For example, for a `foo` API in its version `v1`, its definitions should live
+in `./paths/foo/v1.js` and `./paths/foo/v1/*.js` respectively.
+
+*Note:* until a reasonably stable API is reached, this project uses a `v0`
+version prefix.
+
+Interface/implementation
+------------------------
+
+For reusability across API versions, it's best to keep files in the `paths`
+subfolder very thin, and instead inject implementations via the `dependencies`
+configuration value of `express-openapi`.
+
+These implementations line to the `./lib` subfolder. Adjust `server.js` as
+needed to make them available to API packages.
+
+Streaming Notes
+---------------
+
+For streaming content, it is required that stream metadata is located at the
+start of the stream. Most software writes metadata at the end of the stream,
+because it is when the stream is committed to disk that the entirety of the
+metadata is known.
+
+To move metadata to the start of the stream, a CLI tool such as
+[qtfaststart](https://github.com/danielgtaylor/qtfaststart) for MP4 files might
+be used.