-
Notifications
You must be signed in to change notification settings - Fork 120
Setup
Sketchy uses:
Flask + Gunicorn + Nginx + Celery + Redis + PhantomJS
- Python2.7
- pip
- git
The scripted setup is tailored for Ubuntu only.
First clone Sketchy:
git clone https://github.com/Netflix/sketchy.git
Run setup Script:
cd sketchy/
bash ubuntu_install.sh
If you do not want to perform an automatic setup, you can use the following steps:
On Ubuntu
sudo apt-get install libmysqlclient-dev libxslt-dev libxml2-dev mysql-client
On OSX with Brew
brew update && brew install phantomjs
On OSX with Brew
brew update && brew install mysql
On Ubuntu 64 bit
sudo apt-get install mysql-client
On Ubuntu 32 bit
sudo apt-get install mysql-client
On OSX
brew install redis
On Ubuntu
sudo apt-get -y install redis-server
git clone https://github.com/Netflix/sketchy.git
sudo pip install virtualenv
cd sketchy/
virtualenv sketchenv
source sketchenv/bin/activate
python setup.py install
python manage.py create_db
If you are just testing out Sketchy, use the following to startup the environment via command line.
cd sketchy/
source sketchenv/bin/activate
On OSX
redis-server /usr/local/etc/redis.conf
On Ubuntu
redis-server &
cd /your/directory/sketchy
celery -A sketchy.celery worker &
Sketchy will set the capture object urls based on what is specified in the config-default.py directives.
You can set the host:port in the environment variable 'host' or you will need to set the following directives in config-default.py to match your environment:
# Set hostname:port of your server or IP address if running in test setup (default is 127.0.0.1:8000)
# If you are using Nginx with SSL, only specify IP or Hostname
# Alternatively, you can export the 'host' variable on your system to set this as well
HOST = os.getenv('host', '127.0.0.1:8000')
# Set to true if you are serving Sketchy over SSL with Nginx (default is False)
# Alternatively, you can export the 'use_ssl' variable on your system as well
SSL = os.getenv('use_ssl', False)Note: In the test startup, you must specify the port. In the production setup, Nginx will proxy to Gunicorn, so you only need to specify the IP or FQDN.
python manage.py runserver --host 0.0.0.0 --port 8000
Now that Sketchy is running, give it a try by creating your first capture. #Production Setup# After you have completed either the scripted setup or manual setup, you can configure Sketchy for a more production environment.
The production setup includes Nginx, Gunicorn, and SSL configuration. Gunicorn should have been installed when you ran the setup.py file. The following steps assume you are running an Ubuntu system. You will want to use a certificate that has been signed by a trusted certificate authority.
For this guide, we will use a self-signed SSL certificate.
There are some great instructions for generating a certificate on the Ubuntu website:
Ubuntu - Create a Self Signed SSL Certificate
The last commands you need to run from that tutorial are in the “Installing the Certificate” section:
sudo cp server.crt /etc/ssl/certs
sudo cp server.key /etc/ssl/privateOnce you have finished the instructions at the link above, and these two files are in your /etc/ssl/certs and /etc/ssl/private, you are ready to move on in this guide.
Sketchy uses Gunicorn to serve up content on its internal 127.0.0.1 address. Nginx listens on 0.0.0.0 and proxies connections to Gunicorn for processing.
sudo apt-get install nginx
sudo mkdir -p /var/log/nginx/
sudo touch /var/log/nginx/access.log
sudo touch /var/log/nginx/error.logYou will need to modify the following directive to the path where you have pulled sketchy in the example sketchy.conf file below:
root /apps/sketchy;
Next you can save the config file below to:
/etc/nginx/sites-available/sketchy.conf
server {
listen 0.0.0.0:443 ssl;
ssl_certificate /etc/ssl/certs/server.crt;
ssl_certificate_key /etc/ssl/private/server.key;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
root /apps/Sketchy;
location / {
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://127.0.0.1:8000;
proxy_connect_timeout 30;
proxy_read_timeout 40;
}
}Symlink the sites-available file to the sites-enabled folder:
sudo ln -s /etc/nginx/sites-available/sketchy.conf /etc/nginx/sites-enabled/sketchy.conf
Delete the default configuration:
sudo rm /etc/nginx/sites-enabled/default
Restart nginx:
sudo service nginx restart
You will need to export the 'host' and 'use_ssl' environment variables or you will need to set the following directives in config-default.py to match your environment:
# Set hostname:port of your server or IP address if running in test setup (default is 127.0.0.1:8000)
# If you are using Nginx with SSL, only specify IP or Hostname
# Alternatively, you can export the 'host' variable on your system to set this as well
HOST = os.getenv('host', '127.0.0.1')
# Set to True if you are serving Sketchy over SSL with Nginx (default is False)
# Alternatively, you can export the 'use_ssl' variable on your system as well
SSL = os.getenv('use_ssl', False)The config-default.py file is located in /path/to/sketchy/config-default.py
You will need to change the user and directory settings for each program defined in the supervisor.ini file according to your environment.
You will need to modify the directory and user options for both celeryd and gunicorn:
[program:celeryd]
command=celery worker -A sketchy.celery
directory=/home/sbehrens/dev/stash/sketchy
user=sbehrens
[program:gunicorn]
command=python gunicorn sketchy:app -b 0.0.0.0:8000
directory=/home/sbehrens/dev/stash/sketchy
user=sbehrens
You can start supervisor and use the configuration file included with Sketchy (after modifying command paths and users):
sudo -s
cd /path/to/sketchy
source sketchenv/bin/activate
cd /path/to/sketchy/supervisor
supervisord -c supervisord.ini
An example config is below:
[unix_http_server]
file=/tmp/supervisor.sock
chmod=0700
[supervisord]
logfile = /var/log/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = False
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
[program:redis]
command=redis-server
[program:celeryd]
command=celery worker -A sketchy.celery
directory=/Users/sbehrens/dev/stash/sketchy
user=sbehrens
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
[program:gunicorn]
command=gunicorn sketchy:app -b 0.0.0.0:8000
directory=/Users/sbehrens/dev/stash/sketchy
user=sbehrens
autostart=true
autorestart=true
redirect_stderr=true
Supervisor will start three jobs and make sure they are running. The first job is Redis, which will act as the broker for Celery. The second job is the Celery task management system. The third job is Gunicorn. If something is not starting correctly, you can view the Supervisor log file located at /var/log/supervisord.log.
You can double check everything is working by visiting:
https://yoursite.com/api/v1.0/capture
If you receive a blank JSON object, you are set!
Now that your production environment is setup you can create your first capture.