ec2_setup.notes.txt

################################################################################################
 Setting up Network Portal on EC2
 notes by: cbare@systemsbiology.org
################################################################################################

Start w/ alestic Ubuntu 11.04 or latest 64 bit.
instance type: m1 large, 7.5GB
Firewall: SSH, HTTP

The basic stack is Python (2.7.x) and Apache2. Python PIP (a package management
system) is used to install several Python libraries, including numpy which has
several binary dependencies. The Solr search engine runs on the Jetty app
server and requires an installation of Java. So far, we've used Java 6. Solr is
reverse proxied through apache so we can hit it from javascript running in web
pages. The backend is PostgreSQL version 9, and some fiddling with it's security
policies is needed so that local user accounts can use the psql command line
console.

################################################################################################
 install
################################################################################################

sudo apt-get -y update
sudo apt-get -y install emacs
sudo apt-get -y install git
sudo apt-get -y install apache2
sudo apt-get -y install python
sudo apt-get -y install python-pip python-dev build-essential
sudo apt-get -y install libapache2-mod-wsgi

sudo pip install --upgrade pip
sudo pip install django

PostgreSQL 9.x on Ubuntu 11.10+
-------------------------------
sudo apt-get -y install postgresql-9.1 postgresql-client-9.1
sudo apt-get -y install libpq-dev

Python postgres client
----------------------
sudo pip install psycopg2

NetworkX
--------
sudo pip install networkx

Numpy
--------
sudo pip install numpy

Java
----
sudo apt-get -y install openjdk-6-jdk

Altervatively, use the sun JDK
------------------------------
sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get -y install sun-java6-jdk
sudo update-alternatives --config java

Jetty
-----
// alternatively, just get a tgz from codehaus
sudo apt-get -y install jetty
sudo apt-get -y install libjetty-extra


Solr
----
// don't do this: sudo apt-get -y install solr-jetty (old version 1.4.1)
curl -O http://mirror.metrocast.net/apache/lucene/solr/3.5.0/apache-solr-3.5.0.tgz

// extract into /home/ubuntu/network_portal/apache-solr-3.5.0/
tar -xzvf apache-solr-3.5.0.tgz

// install PostgreSQL JDBC driver
curl -O http://jdbc.postgresql.org/download/postgresql-9.1-901.jdbc4.jar


################################################################################################
 configure
################################################################################################


Setup postgres & populate DB
----------------------------
scp -i ~/ec2/cbare-work-2.pem network_portal.dump.2011.10.24.gz ubuntu@ec2-50-17-132-19.compute-1.amazonaws.com:/home/ubuntu/network_portal
sudo emacs /etc/postgresql/9.1/main/pg_hba.conf
 # change ident or peer to trust for postgres
 # change peer to md5 for local connections
sudo /etc/init.d/postgresql restart
psql -U postgres
create database network_portal
create user, grant privileges
create user XXXXX with password 'XXXXXX';
gunzip < network_portal.dump.2011.10.24.gz | psql -U postgres --dbname network_portal



Set up Apache WSGI
------------------

contents of the file wsgi.py=

import os
import sys
import django

# print >> sys.stderr,"\n"*4                                                                              
# print >> sys.stderr,"~"*100                                                                             
# print >> sys.stderr,"Django version = " + str(django.VERSION)                                           
# print >> sys.stderr,"Python version = " + str(sys.version_info)                                         
# print >> sys.stderr,"~"*100                                                                             

path = os.path.dirname(os.path.dirname(__file__)).replace('\\','/')
if path not in sys.path:
    sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = 'web_app.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()



Set up Apache conf
------------------
sudo emacs /etc/apache2/httpd.conf

WSGIScriptAlias / /home/ubuntu/network_portal/network_portal/web_app/wsgi.py
WSGIPythonPath /home/ubuntu/network_portal/network_portal/web_app

<Directory /home/ubuntu/network_portal/network_portal/web_app>
 <Files wsgi.py>
 Order deny,allow
 Allow from all
 </Files>
</Directory>

Alias /robots.txt /home/ubuntu/network_portal/network_portal/web_app/static/robots.txt
Alias /favicon.ico /home/ubuntu/network_portal/network_portal/web_app/static/favicon.ico

Alias /media/ /home/ubuntu/network_portal/network_portal/media/
Alias /static/ /home/ubuntu/network_portal/network_portal/web_app/static/

<Directory /home/ubuntu/network_portal/network_portal/web_app/static>
Order deny,allow
Allow from all
</Directory>

<Directory /home/ubuntu/network_portal/network_portal/media>
Order deny,allow
Allow from all
</Directory>

<IfModule mod_proxy.c>
  ProxyRequests Off

  <Proxy *>
  Order deny,allow
  Allow from all
  </Proxy>

  ProxyPass /solr/select http://localhost:8983/solr/select
  ProxyPassReverse /solr/select http://localhost:8983/solr/select

  ProxyPass /solr/suggest http://localhost:8983/solr/suggest
  ProxyPassReverse /solr/suggest http://localhost:8983/solr/suggest
</IfModule>


sudo /etc/init.d/apache2 restart



Solr-Jetty config
-----------------

** modify port in to standard solr port /etc/jetty/jetty.xml
<Set name="port"><SystemProperty name="jetty.port" default="8983"/></Set>

** add postgres jdbc driver to Solr's lib directory
sudo ln -s solr/lib/postgresql-9.1-901.jdbc4.jar /usr/share/solr/WEB-INF/lib/

/usr/share/jetty$ sudo -u jetty java -jar /usr/share/jetty/start.jar

// start Solr server (how best to do this in production??)
java -Dsolr.solr.home="/home/ubuntu/network_portal/network_portal/solr" -Dsolr.install.dir="/home/ubuntu/network_portal/apache-solr-3.5.0/" -jar start.jar

// a better alternative to using start.jar is to copy the solr.war to
// jetty's webapps directory and add the following to etc/jetty.xml
<Call class="java.lang.System" name="setProperty">
  <Arg>solr.solr.home</Arg>
  <Arg>/local/jetty/jetty_apps/solr</Arg>
</Call>
<Call class="java.lang.System" name="setProperty">
  <Arg>solr.install.dir</Arg>
  <Arg>/local/lib/apache-solr-3.5.0</Arg>
</Call>

// tell Solr to re-index
curl http://localhost:8983/solr/dataimport?command=full-import

Reverse Proxy Solr through Apache
---------------------------------
cd /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/proxy_http.load .
sudo ln -s ../mods-available/proxy.load .

Make sure proxy stuff is in /etc/apache2/httpd.conf




################################################################################################
 old junk, probably not useful anymore
################################################################################################

NOTE: don't install this stuff!

PostgreSQL 9.0 on Ubuntu 11.04
------------------------------
sudo add-apt-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get -y install postgresql-9.0 postgresql-client-9.0
sudo apt-get -y install libpq-dev


##### NOTE #####
Sunburnt is not used anymore, instead Network Portal search
is creating its own queries, sending them to Solr and uses
simplejson to get the results
######################################################
Sunburnt, python client for Solr
--------------------------------
sudo pip install sunburnt
sudo apt-get -y install libxml2 libxml2-dev
sudo apt-get -y install libxslt1.1 libxslt1-dev
sudo pip install httplib2
sudo pip install lxml
############################################