SETUP

# Setup

µchan requires a few different software packages to run.

µchan is made with flask, uses postgres as the db, memcached to cache things, uses celery (and thus rabbitmq) for task processing. Varnish is used as a secondary cache in front of everyting.
Users first connect to nginx, where a media download is delivered by nginx, and other request are given to varnish. Varnish will then on a cache miss give the request to uwsgi and uwsgi will give the request to one of the instances.
Things that require a lot of time to process or need to be done in intervals are done with tasks and we use celery for that.


run.py and worker.py are for development, don't use them in production

This setup is based on my CentOS machine. This machine has 16GB of ram and most of the disk space allocated to /home. 


USER SETUP

First create a system user for uchan
useradd -r -m -s /bin/bash uchan
This will create a /home/uchan directory.

The uchan image processing library Pillow needs some packages, install zlib-devel and libjpeg-turbo-devel
Install python3.4 dev

Next, su into the uchan user and go to the home dir.

First create two directories, uchanmedia and uchanuploadqueue
uchanmedia will be used to put all the uploaded files in
uchanuploadqueue is a temp dir where all files that users upload will be temporarily placed

Clone the repo to /home/uchan/uchan and cd to it

Setup and activate a virtualenv:
`virtualenv --python /bin/python3.4 env`
`. env/bin/activate`
Test if you have the correct python version: `python --version`

Install the required python packages:
`pip install -r requirements`

Copy the configs:
cp config_sample.py config.py
cp config_celery_sample.py config_celery.py
cp uchan/static/robots.txt.sample uchan/static/robots.txt
Edit the configs to your liking.


You can test if the server starts with `python run.py`
We're now done with the uchan account.

Make the uchanmedia and uchanuploadqueue folders readable for nginx (www-data)
`chmod 755 /home/uchan`


POSTGRES
Install the postgres packages postgresql-server and postgresql-contrib

Change the db location to a folder in /home if you want, see the instructions of /usr/lib/systemd/system/postgresql.service on how to do that.
Initialize the db in this folder:
`sudo postgresql-setup initdb <location>`

Create a new user using the postgres account
`sudo -u postgres createuser --interactive`
Enter name of role to add: uchan
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

Create the uchan db.
`sudo -u postgres createdb uchan`

Comment everyting out in /var/lib/pgsql/data/pg_hba.conf and add the following:
```
local uchan uchan peer
local all postgres peer
```
This enables the postgres user to access everyting and the uchan user to only access the uchan database.

It's recommended to tweak the postgres configuration to maximise performance, as the default settings are rather conservative, especially RAM usage.
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

start and enable the postgres server


RABBIT-MQ AND CELERY
Uchan uses celery and rabbit-mq to do task processing.

install `rabbitmq-server`

start it

add uchan user: `sudo rabbitmqctl add_user uchan <password>`

`sudo rabbitmqctl add_user uchan password`
`sudo rabbitmqctl add_vhost uchanvhost`
`sudo rabbitmqctl set_user_tags uchan uchantag`
`sudo rabbitmqctl set_permissions -p uchanvhost uchan ".*" ".*" ".*"`
delete guest user: `rabbitmqctl delete_user guest`

for more in info for setting up the rabbitmq system see http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html

change the amqp password in config_celery.py

Create a systemd service for the worker:
/etc/systemd/system/uchanworker.service:
```
[Unit]
Description=Celery uchan workers
After=network.target

[Service]
Type=forking
User=uchan
Group=uchan
WorkingDirectory=/home/uchan/uchan
RuntimeDirectory=uchan

Environment=CELERYD_PID_FILE=/var/run/uchan/%%n.pid
Environment=CELERYD_LOG_FILE=/var/log/uchanworker/%%n.log
Environment=CELERYD_LOG_LEVEL=INFO
Environment=CELERYD_CONCURRENCY=8

ExecStart=/home/uchan/uchan/env/bin/celery multi start uchanworker -c ${CELERYD_CONCURRENCY} -A uchan:celery --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} --pidfile=${CELERYD_PID_FILE}
ExecStop=/home/uchan/uchan/env/bin/celery multi stopwait uchanworker --pidfile=${CELERYD_PID_FILE}
ExecReload=/home/uchan/uchan/env/bin/celery multi restart uchanworker -c ${CELERYD_CONCURRENCY} -A uchan:celery --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} --pidfile=${CELERYD_PID_FILE}

[Install]
WantedBy=multi-user.target
```

Start and enable the worker.


MEMCACHED

Install the memcached package

In /etc/sysconfig/memcached change the following:
```
# 4GB
CACHESIZE="4096"
# Only listen on localhost
OPTIONS="-l 127.0.0.1"
```

This will make it to use 4GB of ram for caching, and to listen on localhost. This is required, we don't want the memcache to be open to the world.


NGINX

Nginx server config
/etc/nginx/conf.d/uchan.conf:
```
upstream uchan {
  server 127.0.0.1:6081;
}

server {
  listen 443 ssl;
  server_name your.virtual.host;
  # ssl params here

  add_header X-XSS-Protection "1; mode=block";
  add_header X-Content-Type-Options "nosniff";
  add_header X-Frame-Options "DENY";

  location / {
    # Keep this the same as your config.py MAX_CONTENT_LENGTH config
    client_max_body_size 5M;
    gzip off;

    proxy_http_version 1.1;

    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-Ssl     on;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_pass http://uchan;
  }

  location /favicon.ico {
    alias /home/uchan/uchan/uchan/static/favicon.ico;
    expires 1y;
  }

  location /robots.txt {
    alias /home/uchan/uchan/uchan/static/robots.txt;
  }

  location /static/ {
    alias /home/uchan/uchan/uchan/static/;
    expires 1y;
  }

  location ~ ^/static/(.*)(\.[\d]+)\.(css|js)$ {
    alias /home/uchan/uchan/uchan/static/$1.$3;
    expires 1y;
  }

  location /media/ {
    alias /home/uchan/uchanmedia/;
    expires 1y;
  }

  error_page 404 /static/404.html;
  error_page 500 501 502 503 504 /static/500.html;
}
```

Configure the media and uploadqueue dirs you set in config.py

Start and enable nginx.


VARNISH
Install varnish

Change from storage to memory
/etc/varnish/varnish.params
VARNISH_LISTEN_ADDRESS=127.0.0.1
VARNISH_STORAGE="malloc,3G"

connect varnish to uwsgi:
/etc/varnish/default.vcl
```
backend default {
    .host = "127.0.0.1";
    .port = "5723";
}
```
Port might by any random port that uwsgi listens on.

We want to cache all GET non-cookie requests for 15s, except for the /banned/ page,
because the banned page can be different even without cookies.
```
sub vcl_recv {
  if (req.url ~ "^/banned/") {
    return (pass);
  }
}

```
In varnish.params:
VARNISH_TTL=15

UWSGI
install uwsgi and the python 3.4 plugin for uwsgi

The default CentOS config configures uwsgi to use an emperor and runs the emperor as uwsgi.

/etc/uwsgi/uwsgi.ini:
```
[uwsgi]
uid = uwsgi
gid = uwsgi
pidfile = /run/uwsgi/uwsgi.pid
emperor = /etc/uwsgi.d
stats = /run/uwsgi/stats.sock
emperor-tyrant = true
cap = setgid,setuid
```

/etc/uwsgi.d/uchan.ini:
```
[uwsgi]
chdir = /home/uchan/uchan
disable-loggin

plugin = python3
virtualenv = /home/uchan/uchan/env
module = uchan
callable = app

processes = 8
threads = 4

http-socket = 127.0.0.1:5723
uid = uchan
gid = uchan
```

Tweak processes and threads to your liking. This configuration will launch 8 * 4 threads. The port must be the same as configured in varnish.

setup models script
enable all with systemd


Go to /mod/ and log in with the admin account. Check [mod site] and make sure the ip is correct.
Check if everything works by posting some things and uploading some images.

Tips:
use systemctl status <service> to look for any errors
check /var/log/nginx/error.log for the nginx log
check the log in /home/uchan/uchan/log/uchan.log for other errors

bad gateway? nginx can't connect to the backend
internal server error? uwsgi problem

Make sure (at least on a single machine setup) that you don't open the services to the outside world. Use netstat to see what services are listening.
netstat -lntup
The entries with the local address as 0.0.0.0 are open