Move Grid Node to a different repository (#531)

* MOVE Grid Node

Move Grid Node source code to another repository : https://github.com/OpenMined/GridNode

* Update CI tests

Comment  Grid Nodes tests

* FIX code style

* Fix code style

* Delete unused tests

* Remove unused conftest settings

* Move PyGrid app source files

* Update requirements.txt
- Add CI dependencies at requirements.txt file

* Update requirements.txt

* [REFACTORY] RENAME

Rename app gateway -> grid

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

README.md

@@ -7,6 +7,7 @@ PyGrid is a peer-to-peer network of data owners and data scientists who can coll
 
 ## Overview
 - [Overview](#overview)
+- [Architecture](#architecture)
 - [Getting started](#getting-started)
     - [Build Grid Platform Locally](#start-grid-platform-locally)
     - [Build images](#build-images)
@@ -17,22 +18,28 @@ PyGrid is a peer-to-peer network of data owners and data scientists who can coll
 - [Disclaimer](#disclaimer)
 - [License](#license)
 
+## Architecture
+PyGrid platform is composed by three different components.
+
+**PyGrid App** - A Flask based application used to manage/monitor/control and route grid Nodes/Workers remotely.  
+**Grid Nodes** - Server based apps used to store and manage data access in a secure and private way.  
+**Grid Workers** - Client based apps that uses different Syft based libraries to perform federated learning (ex: syft.js, KotlinSyft, SwiftSyft).
 
 ## Getting started
 To boot the entire PyGrid platform locally, we will use docker containers.
 To install docker the dependencies, just follow [docker documentation](https://docs.docker.com/install/).
 
 ### Start Grid platform locally
 
-#### Using Docker
+#### 1 - Using Docker
 
 The latest PyGrid Gateway and Node images are available on the Docker Hub.
-- PyGrid Gateway - `openmined/grid-node`
-- PyGrid Node - `openmined/grid-node`
+- PyGrid - `openmined/grid-gateway`
+- Grid Node - `openmined/grid-node`
 
-###### Setting the Domain Names
+##### 1.1 - Setting the Domain Names
 
-Before start the grid platform locally using docker, we need to setup the domain names used by the bridge network. In order to access these nodes from outside of containers context, you need to work-around by adding the following domain names on your `/etc/hosts`
+Before start the grid platform locally using docker, we need to set up the domain names used by the bridge network. In order to use these nodes from outside of containers context, you should add the following domain names on your `/etc/hosts`
 ```
 127.0.0.1 gateway
 127.0.0.1 bob
@@ -41,34 +48,63 @@ Before start the grid platform locally using docker, we need to setup the domain
 127.0.0.1 james
 ```
 
-
-It will download the latest openmined's docker images and start a grid platform with 1 gateway and 4 grid nodes.
-**PS:** Feel free to increase/decrease the number of initial PyGrid nodes ***(you can do this by changing the docker-compose.yml file)***.
+#### 1.2 - Run Docker Images
+To setup and start the PyGrid platform you just need start the docker-compose process.
 ```
 $ docker-compose up
 ```
-If you want to rebuild and run the images, you just need to add the `--build` param when running `docker-compose up`
+
+It will download the latest openmined's docker images and start a grid platform with 1 gateway and 4 grid nodes.  
+**PS:** Feel free to increase/decrease the number of initial PyGrid nodes ***(you can do this by changing the docker-compose.yml file)***.
+
+### 1.3 - Build your own images (Optional)
 ```
-$ docker-compose up --build
+$ docker build -t openmined/grid-node ./app/websocket/  # Build PyGrid node image
+$ docker build -t openmined/grid-gateway ./gateway/  # Build gateway image
 ```
 
 
-#### Starting manually
-Start the grid platform manually with 1 gateway and how many grid nodes you want.  
+#### 2 - Starting manually
+To start the PyGrid app manually, run:
 
-- **PyGrid Gateway** - Check out the instructions under [`/gateway`](./gateway)
+```
+python grid.py 
+```
+You can pass the arguments or use environment variables to set the gateway configs.  
 
-- **PyGrid Node** - Check out the instructions under [`/app/websocket`](./app/websocket)
+**Arguments**
+```
+  -h, --help                shows the help message and exit
+  -p [PORT], --port [PORT]  port to run server on (default: 5000)
+  --host [HOST]             the grid gateway host
+  --num_replicas            the number of replicas to provide fault tolerance to model hosting
+  --start_local_db          if this flag is used a SQLAlchemy DB URI is generated to use a local db
+```
 
-### Kubernetes deployment.
-You can now deploy the grid-gateway and grid-node docker containers on kubernetes. This can be either to a local (minikube) cluster or a remote cluster (GKE, EKS, AKS etc). The steps to setup the cluster can be found in [./k8s/Readme.md](https://github.com/OpenMined/PyGrid/tree/dev/k8s)
+**Environment Variables**
+- `GRID_GATEWAY_PORT` -  Port to run server on.
+- `GRID_GATEWAY_HOST` - The grid gateway host
+- `NUM_REPLICAS` - Number of replicas to provide fault tolerance to model hosting
+- `DATABASE_URL` - The gateway database URL
+- `SECRET_KEY` - The secret key
 
-### Build your own images
+#### For development purposes
+You can also start the PyGrid app by running the `dev_server.sh` script.
 ```
-$ docker build -t openmined/grid-node ./app/websocket/  # Build PyGrid node image
-$ docker build -t openmined/grid-gateway ./gateway/  # Build gateway image
+$ ./dev_server.sh
+```
+This script uses the `dev_server.conf.py` as configuration file, including some gunicorn preferences and environment variables. The file is pre-populated with the default environment variables. You can set them by editing the following property:
+```python
+raw_env = [
+    'PORT=5000',
+    'SECRET_KEY=ineedtoputasecrethere',
+    'DATABASE_URL=sqlite:///databasegateway.db',
+]
 ```
 
+### Kubernetes deployment.
+You can now deploy the PyGrid app and Grid Node docker containers on kubernetes. This can be either to a local (minikube) cluster or a remote cluster (GKE, EKS, AKS etc). The steps to setup the cluster can be found in [./k8s/Readme.md](https://github.com/OpenMined/PyGrid/tree/dev/k8s)
+
 ## Try out the Tutorials
 A comprehensive list of tutorials can be found [here](https://github.com/OpenMined/PySyft/tree/master/examples/tutorials/grid).
 

app/__init__.py

@@ -0,0 +1,91 @@
+import logging
+import os
+
+from flask import Flask
+from flask_cors import CORS
+from flask_sockets import Sockets
+from flask_sqlalchemy import SQLAlchemy
+from flask_migrate import Migrate
+from flask_executor import Executor
+
+# Default secret key used only for testing / development
+DEFAULT_SECRET_KEY = "justasecretkeythatishouldputhere"
+
+db = SQLAlchemy()
+executor = Executor()
+logging.getLogger().setLevel(logging.INFO)
+
+
+def set_database_config(app, test_config=None, verbose=False):
+    """ Set configs to use SQL Alchemy library.
+
+        Args:
+            app: Flask application.
+            test_config : Dictionary containing SQLAlchemy configs for test purposes.
+            verbose : Level of flask application verbosity.
+        Returns:
+            app: Flask application.
+        Raises:
+            RuntimeError : If DATABASE_URL or test_config didn't initialized, RuntimeError exception will be raised.
+    """
+    db_url = os.environ.get("DATABASE_URL")
+    migrate = Migrate(app, db)
+    if test_config is None:
+        if db_url:
+            app.config.from_mapping(
+                SQLALCHEMY_DATABASE_URI=db_url, SQLALCHEMY_TRACK_MODIFICATIONS=False
+            )
+        else:
+            raise RuntimeError(
+                "Invalid database address : Set DATABASE_URL environment var or add test_config parameter at create_app method."
+            )
+    else:
+        app.config["SQLALCHEMY_DATABASE_URI"] = test_config["SQLALCHEMY_DATABASE_URI"]
+        app.config["TESTING"] = (
+            test_config["TESTING"] if test_config.get("TESTING") else True
+        )
+        app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = (
+            test_config["SQLALCHEMY_TRACK_MODIFICATIONS"]
+            if test_config.get("SQLALCHEMY_TRACK_MODIFICATIONS")
+            else False
+        )
+    app.config["VERBOSE"] = verbose
+    db.init_app(app)
+
+
+def create_app(debug=False, n_replica=None, test_config=None):
+    """Create flask application."""
+    app = Flask(__name__)
+    app.debug = debug
+
+    app.config["SECRET_KEY"] = os.environ.get("SECRET_KEY", None)
+
+    if app.config["SECRET_KEY"] is None:
+        app.config["SECRET_KEY"] = DEFAULT_SECRET_KEY
+        logging.warn(
+            "Using default secrect key, this is not safe and should be used only for testing and development. To define a secrete key please define the environment variable SECRET_KEY."
+        )
+
+    app.config["N_REPLICA"] = n_replica
+    sockets = Sockets(app)
+
+    # Register app blueprints
+    from .main import main as main_blueprint, ws
+
+    app.register_blueprint(main_blueprint)
+    sockets.register_blueprint(ws, url_prefix=r"/")
+
+    # Set SQLAlchemy configs
+    set_database_config(app, test_config=test_config)
+    s = app.app_context().push()
+    db.create_all()
+    db.session.commit()
+
+    CORS(app)
+
+    # Threads
+    executor.init_app(app)
+    app.config["EXECUTOR_PROPAGATE_EXCEPTIONS"] = True
+    app.config["EXECUTOR_TYPE"] = "thread"
+
+    return app