Collaborative server for Katago
- License
MIT
Installation and Setup --------After setting up KataGo you should do the following steps on Ubuntu:
sudo apt install docker docker-compose
sudo systemctl enable docker
sudo usermod -aG docker INSERT_YOUR_USERNAME
then reboot
Starting/Stopping Docker (for a Local Test Server) --------
docker-compose -f local.yml build
docker-compose -f local.yml up # start up and output logs to current shel
docker-compose -f local.yml up -d # daemonize rather than attaching to the current shell
To stop it, ctrl-c. Or if it was started daemonized or you want to stop it from another shell:
docker-compose -f local.yml down
You should be able to see your server in a browser from something like http://localhost:3000
Running the tests (for a Local Test Server) --------
Using the local server you set up, this command will run some tests:
docker-compose -f local.yml run --rm django pytest -vv
Or you can do this if you also want to print stdout as well in each test, which can be useful for debugging a test itself:
docker-compose -f local.yml run --rm django pytest -vvs
Create an initial admin user for the website --------
docker-compose -f local.yml run --rm django python manage.py createsuperuser
Once you have an initial admin user, you should be able to visit http://localhost:3000/admin and create a Run and create an initial random network for the run (or a non-random network, if you want to start with a network trained from elsewhere).
You will also want to immediately create periodic jobs for updating the bayesian Elo, for refreshing the materialized views that store stats about uploaded games and data, setting them to run every few minutes.
Connecting KataGo to the local server --------
Once you've put in the necessary configs into the Run and set up the first network through the admin panel, you should be able to connect the distributed KataGo client to it (katago contribute) specifying http://localhost:3000 as the url of the server and have it work.
The distributed KataGo client can be built from the "distributed" branch of https://github.com/lightvector/KataGo/ or once it is released you can also download a prebuilt binary.
Getting a shell inside a container --------If you want to get "inside" a container and actually have a running interactive shell there to be able to inspect things, run commands, etc, try this, depending on which container (django, nginx, etc) you want to get a shell inside:
docker-compose -f local.yml run --rm django bash docker-compose -f local.yml run --rm postgres bash docker-compose -f local.yml run --rm nginx bash
Accessing the raw database --------If you want to get direct raw access to the database to run raw postgres queries an inspect the tables that django has set up, then you can do something like this:
docker exec -it NAME_OF_POSTGRES_CONTAINER psql -U DATABASE_USER_NAME -d katago_server_db
You can run "docker container list" to see the containers that you have running, and fill in the appropriate name or id in this command line, and also unless you changed it, the user name for a local server database is hardcoded to "debug", so you might have something like:
docker exec -it server_postgres_1 psql -U debug -d katago_server_db
Migrations --------If you change any of the model definitions (Run, TrainingGame, Network, StartPos, etc) or add a new one, you will want to run:
docker-compose -f local.yml run --rm django python manage.py makemigrations
This will tell django to make a file that will perform the necessary database alterations. Depending on how you set up docker, this migrations file might be owned by root or something like that because it was created from within docker. You may need to sudo chown the file to be owned by you. Then, add the file to be tracked under github and commit it along with your changes.
On a local server, migrations should be actually applied the next time you start up the server. But to explicitly and manually do this if you want:
docker-compose -f local.yml run --rm django python manage.py migrate
Removing Docker Images and Volumes ("I messed up and want to start over") --------
Check for images and volumes:
docker image list
docker volume list
Then you can prune both lists for unused stuff:
docker image prune
docker volume prune
You can directly remove images you don't want:
docker image rm INSERT_NAME_OF_IMAGE
If you stop all containers and remove everything then this should put you back in a clean state to before you built the server or did anything.
Setting up a production server --------To set up a production server, you'll need to also:
- Copy ./envs/production_example to envs/production and edit each of those files where it indicates you should fill in an domain name, email, api key, or other parameter for your actual production site. These are the various environment variables that docker compose will expose within all the individual containers for django, postgres, etc, which those containers' main process (django process, postgres database process, etc) will use to configure themselves.
- Copy ./env_example to ./.env and edit the few environment variables within similarly. It must be named ".env" - this is the name of the file that docker-compose attempts to read upon startup to grab extra environment variables out of, which are used in the docker-compose file itself.
- In the process, you'll need to own an actual domain name with nameservers pointed appropriately to the box that this server will run on, sign up for mailgun and sentry and a few other recommended monitoring services for the website, and such.
On a production server to pick up code changes, you will need to rerun:
docker-compose -f production.yml build
And also if there are migrations, you will need to run this to actually apply the migrations to the production database:
docker-compose -f production.yml run --rm django python manage.py migrate
Be a little careful about whether the site should stay up or be taken down while the database is migrated. For a local server, these steps happen automatically simply when you "up" the server, but for prod, they must be done explicitly.
Helpful links --------
- Docker: https://docs.docker.com/get-started/overview/
- Docker compose: https://docs.docker.com/compose/compose-file/
- Django server settings: http://cookiecutter-django.readthedocs.io/en/latest/settings.html
- Django model fields for defining models and the backend: https://docs.djangoproject.com/en/3.1/ref/models/fields/
- Django queryset API for accessing db within django: https://docs.djangoproject.com/en/3.1/ref/models/querysets/
- Django template language for frontend: https://docs.djangoproject.com/en/3.1/ref/templates/
- Postgres SQL language reference: https://www.postgresql.org/docs/12/queries.html
- Traefik: https://doc.traefik.io/traefik/getting-started/configuration-overview/
- Traefik with docker: https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-18-04
- Django cookiecutter docker: http://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html
- CSS reloading and SASS: http://cookiecutter-django.readthedocs.io/en/latest/live-reloading-and-sass-compilation.html
- Bulma Summary: https://devhints.io/bulma
- WGO.js SGF API: http://wgo.waltheri.net/documentation
- d3 API: https://github.com/d3/d3/blob/master/API.md
- SVG trick: https://stackoverflow.com/questions/16265123/resize-svg-when-window-is-resized-in-d3-js
lightvector: This bit is leftover from the django cookiecutter readme, I'm not sure how relevant it is with all the docker containers in the way, but it sounds maybe like a useful thing to be able to do when testing, modulo the fact that there's potentially a docker container layer in between that you have to work through?.
In development, it is often nice to be able to see emails that are being sent from your application. For that reason local SMTP server MailHog with a web interface is available as docker container.
Container mailhog will start automatically when you will run all docker containers. Please check cookiecutter-django Docker documentation for more details how to start all containers.
With MailHog running, to view messages that are sent by your application, open your browser and go to
http://127.0.0.1:8025