Rework documentation

.gitignore

@@ -205,4 +205,5 @@ public/
 
 # Custom
 output-server/
-src-dapr/output
+src-dapr/output
+src/output

README.md

@@ -2,25 +2,42 @@
 
 > **Note:** This project is currently under heavy development and may change. It should not be used for production deployments yet.
 
-Roadwork RL is a Reinforcement Learning that act as an abstraction layer between different simulation environments and languages, allowing for simulator & code agnostic reinforcement learning research to be done.
+Roadwork-RL is a Reinforcement Learning Platform that aims to act as an abstraction layer between the actual simulator integration and the algorithm acting on it. Its goal is to reduce the complexity required to spin up experiments that require a horizontally and isolated scaled simulator. It allows researchers to focus on the Reinforcement Learning algorithms.
 
-As a simplified overview, the following diagram explains in more detail how this looks like from a **conceptual level**:
+## Architecture
 
-![/assets/architecture-high-level.png](./assets/architecture-high-level.png)
+A high-level architecture of the platform can be found below. The focus points of Roadwork-RL are:
+* Code agnostic platform, allowing clients in different languages
+* Easy simulator integrations
+* The OpenAI Gym language
+
+![/assets/roadwork-rl-abstraction.png](./assets/roadwork-rl-abstraction.png)
 
 ## Getting Started
 
-Getting started is independent of the Operating System and only requires a **Kubernetes** cluster to be available capable of running **Linux Containers**. However to help with this the following documentation is available:
+You can get started yourself in 2 different ways, for researchers we recommend "**Kubernetes**" while as for development purposes, we recommend running the "**Standalone**" installation. See the links below for the different Operating Systems to get started.
 
-* [Windows](./getting-started/windows.md)
-* [Linux](./getting-started/linux.md) - WIP
+> **Note:** We recommend Linux as the target Operating System
 
-## Dependencies
+* [Windows](./docs/getting-started/windows.md) - WIP
+* [Linux](./docs/getting-started/linux.md)
 
-* [grpc.io](https://grpc.io)
-* [Protobuf](https://github.com/protocolbuffers/protobuf)
-* [Dapr](https://github.com/dapr/dapr)
-* [Kubernetes](https://github.com/kubernetes/kubernetes)
+Once the installation is done, you can run your first experiment (Cartpole) through the following commands:
+
+```bash
+# 0a. Start X Server for rendering
+sudo Xvfb -screen 0 1024x768x24 &
+export DISPLAY=:0
+
+# 0b. Navigate to Roadwork
+cd ~/roadwork-rl
+
+# 1. Start Server
+sudo dapr run --app-id rw-server --app-port 3000 python3 ./src/Server/main.py
+
+# 2. Start Experiment (different window)
+sudo dapr run --app-id demo-client python3 ./src/Experiments/baselines/cartpole/train.py
+```
 
 ### Language SDKs Available
 
@@ -37,14 +54,19 @@ Getting started is independent of the Operating System and only requires a **Kub
 * [Protobuf Serialization](./docs/protobuf.md)
 * [Spaces](./docs/spaces.md)
 * [Simulators](./docs/simulators.md)
+* [grpc.io](https://grpc.io)
+* [Protobuf](https://github.com/protocolbuffers/protobuf)
+* [Dapr](https://github.com/dapr/dapr)
+* [Kubernetes](https://github.com/kubernetes/kubernetes)
 
 ## TODO
 
-* Integrate [Facebook ReAgent](https://github.com/facebookresearch/ReAgent) on top of this
+- [ ] Integrate [Facebook ReAgent](https://github.com/facebookresearch/ReAgent) on top of this
     * Simulation Observation Downloader
     * Trainer On-Policy & Off-Policy
-* Add [Project Malmo](https://www.microsoft.com/en-us/research/project/project-malmo/)
-* Add [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents)
+- Add more simulators
+  - [ ] [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents)
+  - [ ] [Project Malmo](https://www.microsoft.com/en-us/research/project/project-malmo/)
+- [ ] Performance Benchmarks (what is the impact of this library compared to a vanilla implemented)
 * Create a custom language for state describing
-    * Currently we can describe a state as shown before: `Tuple([ Box(0, 255, shape=(64, 64, 3)), Box(-50, 50, shape=(3, )) ])`. This might be too abstract or language dependent and could be done easier + more efficient. E.g. think of a Robotic arm, where we should be able to describe each join independently.
-* Performance Benchmarks (what is the impact of this library compared to a vanilla implemented)
+    * Currently we can describe a state as shown before: `Tuple([ Box(0, 255, shape=(64, 64, 3)), Box(-50, 50, shape=(3, )) ])`. This might be too abstract or language dependent and could be done easier + more efficient. E.g. think of a Robotic arm, where we should be able to describe each join independently.

docs/getting-started/linux.md

@@ -1,45 +1,135 @@
-# Getting Started - Linux Terminal
+# Installation - Linux
 
-It is possible to run this under a terminal environment. For this we have to install a virtual screen through `sudo apt install xvfb` and starting it up with `xvfb-run -s "-screen 0 1400x900x24" bash`.
+## Standalone Installation
 
-## Installation
+### Installing Python 3.7
 
-The following script can be utilized to install all the packages on an Ubuntu Distribution. Execute this from the root directory.
+```bash
+# Install Python 3.7
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt update
+sudo apt install python3.7 -y
+
+# Set Python 3.7 as our default
+sudo update-alternatives --config python3
+sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1
+
+# Relink
+sudo rm /usr/bin/python3
+sudo ln -s /usr/bin/python3.7 /usr/bin/python3
+
+# Install Pip (also for sudo)
+curl -s https://bootstrap.pypa.io/get-pip.py | python3
+sudo curl -s https://bootstrap.pypa.io/get-pip.py | sudo python3
+```
+
+> **Note:** Check with `sudo python3 --version` to see if the correct version is installed
+
+### Platform Installation
+
+```bash
+# Install dependencies
+sudo apt install -y --no-install-recommends apt-utils build-essential curl xvfb ffmpeg xorg-dev libsdl2-dev swig cmake python-opengl dos2unix
+
+# Install Docker
+sudo apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
+# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+# sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
+# sudo apt install docker-ce docker-ce-cli containerd.io
+sudo apt install docker.io -y
+
+# Install Dapr (https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md)
+wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
+
+# Navigate to home dir
+cd ~
+
+# Install Dapr/Dapr-Flask
+git clone https://github.com/dapr/python-sdk.git dapr-python-sdk
+cd dapr-python-sdk; sudo pip3 install -e .; cd ~;
+cd dapr-python-sdk/ext/flask_dapr; sudo pip3 install -e .; cd ~;
+
+# Clone Roadwork
+cd ~
+git clone https://github.com/roadwork/roadwork-rl
+
+# Install Roadwork Python SDK
+cd ~/roadwork-rl/src/Lib/python/roadwork
+sudo pip3 install -e .
+
+# Install requirements Server
+cd ~/roadwork-rl/src/Server
+sudo pip3 install -r requirements.txt
+
+# Install requirements Cartpole Experiment
+cd ~/roadwork-rl/src/Experiments/baselines/cartpole
+sudo pip3 install -r requirements.txt
+
+# Init Dapr
+sudo dapr init
+```
+
+### Temporary: Patching Daprd to edge version
+
+We need to have daprd running on the edge version (see `sudo dapr --version`), since it has a fix we require for the platform. Sadly enough this is a manual patch, seeing that GitHub doesn't allow artifact downloading by guests (e.g. https://api.github.com/repos/dapr/dapr/actions/artifacts/12021957/zip).
+
+1. Download the latest artifact from https://github.com/dapr/dapr/actions/runs/180628900
+2. Unzip until you get daprd as a binary
+3. Copy daprd to /usr/local/bin and replace the old one
+4. Run `sudo dapr --version` and confirm that you see: `Runtime version: edge`
+
+## Kubernetes Installation
+
+### Installing Redis
 
 ```bash
-# 1. Installing Dependencies
-sudo apt install ffmpeg xvfb python-opengl 
-sudo apt install swig # Required for OpenAI Box2D Environment
-sudo apt install xorg-dev libglu1-mesa libgl1-mesa-dev xvfb libxinerama1 libxcursor1 # Virtual Screens
-sudo apt install nvidia-384 nvidia-modprobe # Add NVIDIA Support
-
-# 2. Installing Anaconda
-wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/miniconda.sh
-bash /tmp/miniconda.sh -b -p $HOME/miniconda
-
-# 3. Install Python Library
-# Note: currently not in the Pip repo yet
-pip install -e ./src/Lib/python/roadwork
+# Install Helm
+# https://helm.sh/docs/intro/install/
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+chmod 700 get_helm.sh
+./get_helm.sh
+
+# Install redis into cluster
+helm repo add bitnami https://charts.bitnami.com/bitnami
+helm install redis bitnami/redis
+
+echo "Redis is now running, credentials:"
+echo "Host: redis-master:6379"
+echo "Password: $(kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 --decode)"
+
+# Apply
+kubectl apply -f Server/redis.yaml
 ```
 
-## Running Experiments
+### Installing Server
 
-### CartPole
 
 ```bash
+# Build Server
+./Scripts/linux/build-server.sh Server/ roadwork.io/rw-server
+
+# Remove old Server
+kubectl delete deployment rw-server
+
 # Start Server
-./Scripts/linux/run-server.sh openai $(pwd)/../output-server/cartpole
+kubectl apply -f Server/kubernetes.yaml
 
-# Start Experiment
-./Scripts/linux/run-experiment.sh python cartpole
+# Get Logs
+kubectl logs -f deployment/rw-server -c server -f
 ```
 
-### Lunar Lander
+### Installing Client
 
 ```bash
-# Start Server
-./Scripts/linux/run-server.sh openai $(pwd)/../output-server/lunar-lander
+# Build Client
+./Scripts/linux/build-client.sh Experiments/baselines/cartpole roadwork.io/rw-exp-baselines-cartpole
+
+# Remove old Client
+kubectl delete pod p-rw-exp-cartpole
+
+# Start Client
+kubectl apply -f Experiments/baselines/cartpole/kubernetes.yaml
 
-# Start Experiment
-./Scripts/linux/run-experiment.sh python lunar-lander
+# Get Logs
+kubectl logs pod/p-rw-exp-cartpole -c experiment -f
 ```

src/Server/requirements.txt

@@ -10,5 +10,5 @@ grpcio-tools==1.26.0
 protobuf==3.11.3
 six==1.14.0
 tox == 3.15.0
-roadwork==0.0.1
+roadwork
 gym==0.17.2