Add more extensive documentation

README.md

@@ -15,55 +15,9 @@ automatically by various testing-methods like unit tests or code smell detection
 Code FREAK is currently under heavy development. This means things can break without any further notice.
 Please keep this in mind when testing the application.
 
-## Development environment setup
-Create a file `src/main/resources/application-dev.yml`. For documentation on how to configure the
-server see [application.yml](https://github.com/code-freak/code-freak/blob/master/src/main/resources/application.yml)
-in the same directory. Minimum configuration that uses the in-memory database:
-```yaml
-spring:
-  jpa:
-    database: HSQL
-    hibernate:
-      ddl-auto: create
-```
-
-### Database
-You can either use the embedded HSQL storage or a PostgreSQL database. Data from the HSQL database will get lost when
-the application shuts down. For Postgres create at least a dedicated database and adjust the configuration accordingly:
-```yaml
-spring:
-  datasource:
-    url: "jdbc:postgresql://[host]:[port]/database"
-    username: user
-    pasword: supersecure
-    driver-class-name: org.postgresql.Driver
-  jpa:
-    database-platform: org.hibernate.dialect.PostgreSQLDialect
-    hibernate:
-      ddl-auto: update
-```
-
-### Docker
-For many parts of the application we need connection to a (dedicated) Docker daemon. By default we use the default
-socket for your platform. If you are on Linux please follow the installation guidelines for your distribution.
-For Windows, MacOS and other OS that cannot run Docker natively you will either need the official Docker for Windows/MacOS
-software stacks or use the Vagrant machine that is included in this repository. The Vagrant machine will make the Docker
-daemon available at `127.0.0.1:2375` (the official Docker port). If you setup Docker for Windows/MacOS the Daemon should
-be reachable on the same address. So if you are on Windows or MacOS adjust your `application-dev.properties` file so it points
-at a Docker daemon.
-```yaml
-code-freak:
-  docker:
-    host: "tcp://127.0.0.1:2375"
-```
-
-### Run
-Run the command `./gradlew bootRun`. The application is started at `http://localhost:8080`.
-
-### Fix linting issues
-```console
-$ ./gradlew spotlessApply
-```
+## Documentation
+For instructions how to run, configure and use the application please find the documentation in the
+[`/docs`](./docs/index.adoc) directory.
 
 ## License
     Code FREAK | Code Feedback Review & Evaluation Kit

docs/architecture.adoc

@@ -0,0 +1,44 @@
+= System Architecture
+:sectnums:
+:toc: left
+:toclevels: 3
+
+== High level overview
+Currently there are three major components of the system:
+
+1. The backend application based on Spring Boot
+2. Many Development Environments (IDE) for each user
+3. A Docker daemon controlled by the backend application
+
+image::img/Code FREAK Overview.png[Code FREAK Overview, 600]
+
+== Backend Application
+The Backend Application (app) is the main component of Code FREAK and is written in Java based on Spring Boot. It is
+responsible for storing all kind of data including users, tasks and results. The app keeps a connection to a dedicated
+Docker daemon to spin up new containers and pushing/pulling files from that containers. Currently there is no direct
+communication between the app and containers. All messages are send over the Docker daemon.
+
+It is not required but highly recommended to run the application using the official Docker image.
+
+== IDE
+The Integrated Development Environments (IDE) are used by the users to work on the programming tasks. Because each IDE
+runs in its own container they are isolated environments that offer both flexibility and security. Currently we use
+Visual Studio Code as IDE based on https://github.com/cdr/code-server[code-server]. The IDE runs directly in the browser
+of the user and do not require any local setup. Beside the IDE itself the containers contain all required runtimes
+and toolchains (Java, Gradle, nodejs, npm, …) to execute and debug the code directly on the server. Because containers
+can be limited in RAM and CPU-usage users cannot steal resources from other users.
+
+The files are synced between the browser and the container in real time. The backend application will monitor the IDE
+container and pull the files from it periodically to store it in a database.
+
+== Docker Daemon
+The Docker daemon is currently used for starting IDE instances (and testing code in the near future). Docker containers
+allow a good isolation between individual environments by using standard Linux kernel features. The backend app connects
+via the standard socket or HTTP API the Docker daemon.
+
+If you run the Code FREAK application itself on Docker you should use different daemons for the app and for IDE. Even
+if the IDE containers are resource-limited, too many IDE containers could consume all available resources on the host
+and crash the backend application.
+
+Because a single Docker host does not scale very well we plan to support other container orchestration systems in the
+future.

docs/development.adoc

@@ -0,0 +1,62 @@
+= Development environment setup
+:sectnums:
+:toc: left
+:toclevels: 3
+
+== Configuration
+Create a file `src/main/resources/application-dev.yml`. For documentation on how to configure the
+server see https://github.com/code-freak/code-freak/blob/master/src/main/resources/application.yml[`application.yml`]
+in the same directory. Minimum configuration that uses the in-memory database:
+```yaml
+spring:
+  jpa:
+    database: HSQL
+    hibernate:
+      ddl-auto: create
+```
+
+=== Database
+You can either use the embedded HSQL storage or a PostgreSQL database. Data from the HSQL database will get lost when
+the application shuts down. For Postgres create at least a dedicated database and adjust the configuration accordingly:
+```yaml
+spring:
+  datasource:
+    url: "jdbc:postgresql://[host]:[port]/database"
+    username: user
+    pasword: supersecure
+    driver-class-name: org.postgresql.Driver
+  jpa:
+    database-platform: org.hibernate.dialect.PostgreSQLDialect
+    hibernate:
+      ddl-auto: update
+```
+
+=== Docker
+For many parts of the application we need connection to a (dedicated) Docker daemon. By default we use the default
+socket for your platform. If you are on Linux please follow the installation guidelines for your distribution.
+For Windows, MacOS and other OS that cannot run Docker natively you will either need the official Docker for Windows/MacOS
+software stacks or use the Vagrant machine that is included in this repository. The Vagrant machine will make the Docker
+daemon available at `127.0.0.1:2375` (the official Docker port). If you setup Docker for Windows/MacOS the Daemon should
+be reachable on the same address. So if you are on Windows or MacOS adjust your `application-dev.properties` file so it points
+at a Docker daemon.
+```yaml
+code-freak:
+  docker:
+    host: "tcp://127.0.0.1:2375"
+```
+
+== Run the application
+If you use IntelliJ simply start the `CodeFreakApplication` Spring Boot application.
+For Gradle run the command `./gradlew bootRun`. The application is started at `http://localhost:8080`.
+
+== Running tests
+To run the tests you need a working Docker connection. It is also possible to use the Vagrant machine for Testing:
+```shell
+$ ./gradlew test          # for testing with local Docker daemon
+$ ./gradlew vagrantTest   # for testing with Vagrant Docker daemon
+```
+
+== Fix linting issues
+```console
+$ ./gradlew spotlessApply
+```

docs/img/Code FREAK Overview.drawio

@@ -0,0 +1 @@
+<mxfile modified="2019-06-27T08:35:18.835Z" host="www.draw.io" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" etag="W-0cZtVVL5drmmakpdPM" version="10.8.1" type="device"><diagram id="Pbe8_UnveNCYmR3DGHl-" name="Page-1">7VvZkuI2FP0aqpKHpmzJNvDI1jOpzFQl6elM5mlKYAGeFhaxxQD5+ki2vMiywTQY6MX9gHUleTn3+NyrpVtwuNx+CNBq8Zm6mLSA4W5bcNQCwIQ9i/8Iyy629GwnNswDz5WNMsOD9x+WRkNa156LQ6Uho5Qwb6Uap9T38ZQpNhQEdKM2m1Gi3nWF5lgzPEwR0a1fPZctpNV0elnFR+zNF/LWXdCJK5YoaSzfJFwgl25yJjhuwWFAKYvPltshJgK8BJe4331FbfpgAfZZnQ5Pf8Kvj4/fJ+NPYLb++wdH8TO5k1f5ichavvCQu7AFHMKvOZgE/Gwuzu7/Gvd/183yzdgugSuga9/F4o4Gr94sPIYfVmgqajecINy2YEvCSyY/nXmEDCmhQdQX9kBv1He4PWQBfcK5GgAty3ZFD+qzB3k3U5Zz7e6jI32uPDbJi+KA4W3OJLH6gOkSs2DHmyS1jvSbJC6wZHmT0aArTYscAdJ+SDJvnl46cw4/kf45wldA89WITp8wd4bxC12x9q+necNF4SJqWwJsLzqaAdYy6gFrNYUrrMS1Ubob7qQjuup0NwxzNBjqXphFh/YZnO6SFFvpEujoLknbXMQnluaT30bjhh3i2rjrWuUOEcd56F/EukxXyqA2QVNY268X6+6tYe28WqwhuDWsO68X6xq5yWWxNvXkpN1ua9Dy92MqfipOPvVxAVRpQsSb+7w45ajx0AwHAi2P5+l9WbH0XFfcptRhqkv3ZTrxUZFqyuEJcM7jRqv4yXTrubE5L+phV3Mh9t2+GFcJXxAUht408iIKmG7OfycF+Kpg176btKYScOwqQzgd7hycdgmciS3ABDHvpzrwK8NY3uEP6vEnyYJNwZuw02vb6kVCug6mWPbLD9WKl3IOXooDPsdMu1Tk9vTVT2CCHqgS5RSeUyjh/LumScVdGPm4zxuYzmrLfyLPGcTz8V0CuqjlgtgCdtY5U2Opz4nBTCz8PSbFVtwWP06FlPM39VZhlSrk6InCVTx7MPO25Un5xHJsw9IVf4bcDpo2IxCmZasCYesCAUsYDRsTCD2mXokW4A3TAnZVWqSp19Vo0b0VWsA3TAt+S5UWCU2uRQvQ0yHmsTrJqmjAFnROfUTGmTWXqgmkszafKF1J4w/M2E6Ch9aMqs7BW4/9IzK9dseWxW+5qtFWZoFRYScLFQlMZL73SHm2ms/qc/7kufAzZibj3GAfmkbNdOfEPAZWTVAcSD04emiXa7YSDcL6+RIwCvPXh55Lbc9P4ic4ax6UPOMpygaEshWV6zHMJhgP6dJRA9I9FFWlaBD9HUHdc886FieCzZIwdtGJ4GTp5lp6BerrFUc92OVVTpS/5SuzflHpJQldss4X6cy+tKMRQTxWx7RRvLlfxzTdMy+hY90XE4pz1H6zzLZug9nFacbiCvOBaYtC+4aYrS9hv0foM0XodGn2ZiK0vQf6K8XfI0XqVPXBZBIpXDL/LUwHlbymNp6gbOBFxWxYJHbvgLLtb9+QsjnvMfsGYnZdZtu3wWzrOGYf+BIaYjZ4j9lNxezinofrx2x9f5Wua89bVKxeLKwdTTPhAopymQeUqyJrSJWxVCafr0J1Bw61JwkvsyZq2uVj3eNXRIsyVdxl2fB6aDJHfv0VDut9hSNjgXHlFQ6oDz6/iC0uRdz1fS8B5sxAk6iB4ISM67y1PWjZI+ECnpqFUsJa2QYYgmfiUsXtL0ykd4OQO9Tz51+iXO/OOkeCdMSypG0p3rE6unfOtJ2FF7ON8/E3nv37ARz/Dw==</diagram></mxfile>

docs/index.adoc

@@ -0,0 +1,83 @@
+= Code FREAK Documentation
+:sectnums:
+:toc: left
+:toclevels: 3
+
+:toc!:
+
+Welcome to the Code FREAK documentation.
+Code FREAK is an online IDE for educational purposes that allows teachers to create and enroll programming tasks
+allows students to programm and execute their code without leaving the browser. Submissions can be evaluated
+automatically by various testing-methods like unit tests or code smell detection.
+
+WARNING: Code FREAK is under heavy development. Things might break between updates even if we try to be backwards-compatible.
+
+== Getting started
+If you are new to Code FREAK here are some links that may help you to understand the basic concepts of the system:
+
+* link:architecture.adoc[Architecture of Code FREAK]
+* link:development.adoc[Development Setup]
+* More resources coming soon...
+
+== Security
+We utilize Linux containers based on Docker which allow a good logical isolation (separate process-set, resource-limits, …).
+It should be very difficult to escape from the containers and do harm to the underlying host-system but we do not guarantee
+that it is impossible. This is why we recommend to run a dedicated Docker daemon only for Code FREAK that does nothing
+else except hosting the IDE instances and running untrusted code.
+Additionally you should not run the Code FREAK application and its sidekick Docker daemon on the same host for similar
+reasons: If someone manages to escape the IDE containers they could hijack the backend application and steal sensitive
+information. Even if someone does not accomplishes to break out of the container they could spin up so many containers
+that all host system resources are exhausted.
+
+== Installation
+There are various methods to run the application in production. We only support the Docker/docker-compose setup.
+If you plan to run the application directly on a host-system you should know what you are doing. If you are looking
+for a development setup please take a look at the <<Development / Contribution>> section.
+
+=== Prerequisites
+For the backend application:
+
+* A Docker host/daemon
+* About 1-2GB RAM
+
+For the dedicated docker daemon:
+
+* Separated Docker daemon that does not run on the same host as the backend application
+* Lots of memory. You should plan 3-4GB RAM per concurrent user (depending on the
+programming language). We are working on reducing the RAM usage.
+* About CPU 1 Core / concurrent 10 users
+
+=== Preparing the Docker daemon
+Docker allows to connect via HTTP/TLS to the integrated API. This requires to set up a PKI for certificate-management.
+There is an extensive documentation on how to do this in the official Docker docs.
+https://docs.docker.com/engine/security/https/[Please follow this article] carefully. As the result you should have
+a `ca.pem`, `cert.pem` and `key.pem` file that can be used on your backend application host.
+
+=== Set up using Docker Compose
+We have included a sample `docker-compose.yml` in the main repository. This should work fine out-of-the-box but uses
+a Docker-in-Docker container which is not suited for development. Please remove the `docker` service and adjust the
+environment-variables of the `app` service that it points to your dedicated Docker host. Place the `ca.pem`, `cert.pem`
+and `key.pem` files in a directoy `certificates` next to your `docker-compose.yml` and adjust the configuration
+accordingly:
+
+```yaml
+version: "3"
+services:
+  app:
+    # ... SNIP ...
+    volumes:
+      - "./certificates:/var/code-freak/certs"
+    environment:
+      DOCKER_HOST: "http://docker-host.at-your-company.domain.com:2376"
+      DOCKER_CERT_PATH: "/var/code-freak/certs"
+      # ... SNIP ...
+```
+
+== Configuration
+`TODO`
+
+== Development / Contribution
+`TODO`
+
+== License
+Code FREAK is licensed under the AGPLv3. For more information see the LICENSE file that comes with the source code.