Development Guide
First of all, thanks for considering contributing to SpaceDock!
Spacedock uses the following branching system for development:
-
alpha: This is where development happens. All changes (features/enhancements and bug fixes) go here first. If you want to do code changes, you should branch off of thealphabranch.
You can access a server running this code at https://alpha.spacedock.info/ -
beta: Once a change has been cooked long enough, and it is confirmed that it works as intended without unwanted side effects, it will be merged into thebetabranch, together with potential bug fixes for it that have been merged to alpha before too. Now to confirm it is really doing what it should do.
You can access a server running this code at https://beta.spacedock.info/ -
master: this branch always matches production, i.e. that's the code that runs https://spacedock.info/. For that reason, everything that gets merged there should be rock stable!
Current problems and enhancement requests are tracked in the GitHub Issues section. If you need something to work on, go through this list.
The team is communicating via a Matrix/Riot server. You can access it at https://im.52k.de/. It is also bridged to a Discord server (https://discord.gg/fMkwgKS) and IRC channel (https://webchat.esper.net/#spacedock).
That means, if somebody writes a message on one communication platform, you can also read it on the two others.
Be aware, though, that bridges work with some lag. So for continuous discussion the use of Matrix/Riot server is strongly encouraged.
# make a root directory for you projects, if you need
cd ~
mkdir git-repos
cd git-repos
# clone the source code
git clone -b alpha https://github.com/KSP-SpaceDock/SpaceDock.git
cd SpaceDockHowever, you might want to fork it first.
We use python-3.8, so you need this interpreter version installed. If your distribution doesn't have it, on linux you can use pyenv to install one locally.
It is strongly recommended to use virtual environment for development to isolate your system from all the requirements:
python3.8 -m venv venv
Then enter the venv:
source venv/bin/activate
And install all dependencies:
pip install -r requirements.txt
Always work within the venv. This saves from many issues with local environment.
There are several files that could be used to configure server for specific environment:
-
config.ini- holds many backend-specific options. Extensively documented. -
logging.ini- holds logging configuration. You can change it (formatting, log handlers and such) to your liking. See Python Logging for details. -
alembic.ini- holds configuration of the database migration system. Do not edit it unless you know what you're doing.
When you first clone the repository both config.ini and alembic.ini come with additional .example extension so as not to override your local settings when you update the code. You may copy them right away:
cp config.ini.example config.ini
cp alembic.ini.example alembic.ini
cp logging.ini.example logging.iniOr wait until the ./start-server.sh script (see below) will do this for you.
The easiest way to run the server locally is using Docker. Docker is a containerization tool, that allows you to run a local instance of SpaceDock without needing to install and fiddle around with webservers, databases, networking, dependencies and all the other complicated stuff.
It's available for Windows 10 Pro, macOS and Linux distributions, take a look at the System Requirements on their installation page.
You will also need the Docker Compose to be installed:
pip install docker-compose (from within venv)
If you need help setting up Docker, contact the SpaceDock team. We will try our best to get you started. Also your preferred search engine has a lot of answers on that topic.
If you did the code changes you want to try out, you can start a local server with ./start-server.sh.
This will start the Docker daemon if it's not running yet, create default configuration files if they don't yet exist, and start the docker containers that constitute the server:
-
spacedock_db_1- PostgreSQL 11.4 database server -
spacedock_adminer_1- Adminer web interface todb -
spacedock_redis_1- Redis storage and message bus to pass tasks frombackendtocelery -
spacedock_backend_1- Flask application that is run in development mode -
spacedock_celery_1- Celery workers for asynchronous tasks -
spacedock_frontend_1- Nginx 1.17 with static files and proxy pass to thebackend
Once there's a lot of green text saying done, go ahead and check out
(or http://192.168.99.100:5080/ on older versions of Windows)
You should see a familiar looking website.
To see the logs from the running server it is recommended to start:
docker-compose logs -f
immediately after you started the server.
You can also access the database via an Adminer interface on
localhost:8080.
Credentials for DB access, including database name, are in the .env file.
To open a shell in a running container, run
docker exec -it <container_name> bash.
Uploaded mods will be saved under ./storage in your SpaceDock directory, and the database data can be found under ./spacedock-db/pgdata. Both folders are mounted to the respective containers, so the data is persisted between container builds and runs.
Any code changes in the backend are automatically applied on the fly. But if you have changed something in celery code, the service has to be rebuild and restarted with the same ./start-server.sh script.
Likewise, if you change anything in the site configuration in config.ini, you have to restart the service.
Important note:
Postgres, SpaceDock's DBMS, wants to chown and chmod the ./spacedock-db/pgdata folder. It must be able to do that, else the startup will fail. So make sure your local git clone is not on a mounted NTFS drive or whatsoever.
If you don't want postgres to do that, change the docker-compose.yml to not mount a folder from your host system, but instead use a named docker volume for persistence, like this:
[...]
services:
db:
image: postgres:11.4
[...]
volumes:
- spacedock-dev-db:/var/lib/postgresql/data # <-- change this to not mount a host folder, but a named volume
[...]
networks:
spacedock-net:
driver: bridge
volumes:
- spacedock-dev-db # <-- Add it to the list of volumes at the end of docker-compose.yml.docker-compose down
This will gracefully shut down all the containers.
sudo apt install python3 python3-dev python3-pip python3-virtualenv nodejs npm
sudo apt install postgresql-client redis apache2 libapache2-mod-xsendfile git
To enable the required Apache2 modules:
sudo a2enmod expires rewrite proxy xsendfileproxy_http proxy_balancer lbmethod_byrequests
[TODO] By someone who knows how to do that and what to be aware of.
Whenever we need to change the table layout (called schema) of one of the database tables, we need to write a migration to tell Postgres what to do. We use the alembic tool to help with the heavy lifting.
Alembic can automatically write migration scripts for us by analyzing the currently active schema of the database and comparing it with what the ORM code in objects.py describes.
If you've the Docker environment set up, you can use the ./generate_revision.sh script in the repository. It outputs an auto-generated revision to the alembic/versions/ directory.
Please check this script carefully (both upgrade() and downgrade(), to make sure it does what you want it to do. Also give any new indices or constraints names, so they can be deleted in the downgrade() function.
We never know whether a deployment may fail or not, so always make sure the downgrade script works 100%, so we have a way out in emergencies!
The backend container automatically migrates the db to the latest revision on startup.
If you want to checkout a different branch that doesn't have all the migrations of the current branch, you need to downgrade the revision before changing branches.
- Make sure you are on the branch that has the revisions you want to get rid of in the alembic directory, and start the server:
./start-server.sh - Start a shell in the backend container:
docker-compose exec backend bash - Find the identifier of the revision you want to downgrade to (the last common revision across the two branches):
alembic history - Downgrade the revision:
alembic downgrade <revision_id> - Exit the shell, checkout the new branch, restart the server:
./start-server.sh - Done!
Contact us via Riot, Discord or IRC, open up a new issue or write an email at support@spacedock.info.
