Skip to content

Latest commit

 

History

History
279 lines (215 loc) · 7.83 KB

install-manager.rst

File metadata and controls

279 lines (215 loc) · 7.83 KB
Raw

Install Manager

We assume that your system is configured with a sudoable admin user named devops. Also, you should already have a running etcd cluster, PostgreSQL and Redis servers as guided in previous step guides.

Guide variables

warning

Prepare the values of the following variables before working with this page and replace their occurrences with the values when you follow the guide.

Name Meaning
{NS} The etcd namespace (just create a unique string like domain names)
{ETCDADDR} The etcd cluster address ({ETCDHOST}:{ETCDPORT}, localhost:2379 for development setup)
{REDISADDR} The Redis server address ({REDISHOST}:{REDISPORT}, localhost:6389 for development setup)
{DBADDR} The PostgreSQL server address ({DBHOST}:{DBPORT}, localhost:5442 for development setup)
{DBUSER} The database username (e.g., postgres for development setup)
{DBPASS} The database password (e.g., develove for development setup)
{ENDPOINT} The DNS hostname of the API server (depending on your environment, this may be either a publicly registered domain or a local private domain)

Optional variables

Name Meaning
{SSLCERT} The path to your SSL certificate (bundled with CA chain certificates)
{SSLPKEY} The path to your SSL private key
{S3AKEY} The access key for AWS S3 or compatible services1
{S3SKEY} The secret key for AWS S3 or compatible services
{DDAPIKEY} The Datadog API key
{DDAPPKEY} The Datadog application key
{SENTRYURL} The private Sentry report URL

Install dependencies for daemonization

Ubuntu

$ sudo apt-get -y update
$ sudo apt-get -y dist-upgrade
$ sudo apt-get install -y ca-certificates git-core nginx supervisor libsnappy-dev

Here are some optional but useful packages:

$ sudo apt-get install -y vim tmux htop

CentOS / RHEL

(TODO)

Prepare Python 3.6+

Check out Install Python via pyenv </install/install-python-via-pyenv> for instructions.:raw-html-m2r:<br> Create a virtualenv named "venv-manager".

Install Backend.AI Manager from Source

$ pyenv shell venv-manager
$ git clone --branch=master "https://github.com/lablup/backend.ai-manager" "backend.ai-manager"
$ cd backend.ai-manager
$ pip install -U pip setuptools
$ pip install -U -r requirements.txt

Configure NGINX

$ sudo vi /etc/nginx/sites-available/gateway

(For testing) Without SSL

map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    server_name {ENDPOINT};
    charset utf-8;
    client_max_body_size 32M;
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

(For production) With SSL

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;

map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 443 ssl;
    server_name {ENDPOINT};
    charset utf-8;
    client_max_body_size 32M;
    ssl_certificate {SSLCERT};
    ssl_certificate_key {SSLPKEY};
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Monitoring and Logging (optional)

Check out the [[Install Monitoring and Logging Tools]] guide.

Configure supervisord

supervisord eases the management of daemonization lifecycles of user-defined programs.

supervisord application config

$ sudo vi /etc/supervisor/conf.d/apps.conf
[program:backendai-manager]
user = devops
stopsignal = TERM
stopasgroup = true
command = /home/devops/run-manager.sh

pyenv + venv initialization script for non-login shells

$ vi /home/devops/init-venv.sh
#!/bin/bash
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv shell venv-manager

The main program managed by supervisord

$ vi /home/devops/run-manager.sh
#!/bin/bash
source /home/devops/init-venv.sh
umask 0002
export AWS_ACCESS_KEY_ID="{S3AKEY}"
export AWS_SECRET_ACCESS_KEY="{S3SKEY}"
export DATADOG_API_KEY={DDAPIKEY}
export DATADOG_APP_KEY={DDAPPKEY}
export RAVEN_URI="{SENTRYURL}"
exec python -m ai.backend.gateway.server \
            --db-addr={DBHOST}:{DBPORT} \
            --db-user={DBUSER} \
            --db-password='{DBPASS}' \
            --db-name=backend \
            --etcd-addr {ETCDHOST}:{ETCDPORT} \
            --namespace NS \
            --redis-addr {REDISHOST}:{REDISPORT} \
            --events-port 5002 \
            --service-ip 127.0.0.1 \
            --service-port 8081

Initialize database and load intial fixtures

See the Prepare Databases for Manager </install/prepare-database-for-manager> guide.

Restart nginx

$ sudo cp /etc/nginx/sites-enabled/default ../nginx.site-enabled.default
$ sudo rm /etc/nginx/sites-enabled/default
$ ln -s ../sites-available/gateway /etc/nginx/sites-enabled/gateway
$ sudo systemctl restart nginx

Finally, Run!

$ sudo supervisorctl reread
$ sudo supervisorctl start backendai-manager
$ sudo service nginx restart

  1. AWS S3 is used to store the output files generated by the user code in kernels' /home/work/.output directory. If not specified, Backend.AI will just skip uploading generated files.↩