IPPC 4.0
HTML Python JavaScript CSS ActionScript Shell
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
calls
conf/locale
deploy
docs
forum
ippc
iyph
news
ocs
pce
requirements
schedule
static
t_eppo
.gitignore
.hgignore
__init__.py
fabfile.py
ippc_test.json
local_settings_example.py
manage.py
readme.md
settings.py
setup.py
urls.py
wsgi.py

readme.md

IPPC 4.0

Things to do

  • Install and use https://github.com/mbi/django-rosetta for sane translation management
  • Setup auto-restart of Nginx and Gunicorn in case server goes down
  • Setup daily auto backup of MySQL. Look at moving DB to dedicated DB servers.
  • Tag 4.0.0 release
  • About Secretariat page should have links to profiles
  • Finish FAQ update and send to translation
  • File upload field for forum comments
  • Make own gravatar_url tag to use instead of Mezzanine's in order to be able to serve over SSL
  • User registration open but behind login-required and staff-user required. The URL to add new users is /account/signup/
  • Setup Versioning of all page content
  • Remove the unused Blog app from admin
  • Eventually update to Mezzanine 3.1.5
    • Update to latest version of Mezzanine and make sure current functionality works
  • Document nginx/gunicorn/supervisor setup (currently running gunicorn with deprecated gunicorn_django -b 0.0.0.0:8000 command — get it running and working with recommended command instead)
  • Last modified date for pages
  • IRSS refactor
  • Phytosanitary.info refactor (use original code?)
  • Both phytosantiary.info and apppc.org will need to get SSL certificates for single sign-on to work securely: > Even if the authentication with CAS is made using a mechanism which makes it difficult to interfere with, all authorized communication will subsequentely use a cookie identifing the session which can be used to hijack the connection. So you need to encrypt the communication. There is just no way around that if you want to enforce some sort of security.
  • Order permission groups alphabetically in admin
  • Setup working fabric script for easy deployment that does the following after running fab deploy dev:
    1. Adds, commits and pushes files to Github (for future: if tests pass)
    2. Logs in to dev.ippc.int, activates application virtualenv and pulls changes from Github
    3. Collect static files to locations to be served on dev server
    4. Restart gunicorn and nginx
  • If no publication or agenda numbers exist, don't show header or cells
  • wiki.ippc.int

Installation / Setup

  1. Install Django development environment on your computer and follow instructions to get it running: http://wiki.bitnami.com/Infrastructure_Stacks/BitNami_Django_Stack

  2. Clone code repository (currently on GitHub), move into it and install third-party libraries for the project:

    git clone https://github.com/hypertexthero/ippcdj.git ippcdj_repo
    cd ippcdj_repo
    pip2.7 install -r requirements/project.txt
    mv local_settings_example.py local_settings.py 
    python manage.py createdb
    python manage.py runserver
    
    
  3. Go to 127.0.0.1:8000 to see the app running. go to 127.0.0.1:8000/admin to log in to the admin interface. To stop the server press Ctrl-C (Ctrl-Z and Enter in Windows) in the terminal.

Workflow

GitHub accounts are for codebase repository. Copies of codebase repository are also available in:

  1. Each developer's computer
  2. dev.ippc.int server
  3. www.ippc.int server

We're using a hared repository model and the main IPPC code is at

GitHub Flow (how to work on www.ippc.int code):

First, setup git (so you don't have to keep putting in your password every time).

Then, here's a basic guide (below is an example of working with this repository itself).

Git Workflow

  1. If you're working on the code for the first time, first clone repository from ippc repo (the first time you start working with it)

    cd ~/projects
    git clone https://github.com/ippc/ippcdj.git
  2. Before beginning the day's work, pull the latest changes from the online repo

    git pull
    
  3. Change to ippcdj_repo directory and open the directory with your text editor (in this case, mate == textmate editor)

    cd ippcdj_repo && mate .
  4. At end of day or more often as you prefer, add your changes to your local staging area and commit them to your local repo

    git add .
    git commit -m 'new work'
  5. Push your changes to main server at end of day (or more often)

    git push origin master

Branching

  1. Create a new branch called 'newfeature' and switch to it

    git checkout -b newfeature
  2. Create a new file for this new feature and add some text to it

    echo 'a text file!' >> newfile.txt
  3. At end of day or more often as you prefer, commit the changes to this branch

    git add .
    git commit -m 'new feature'
  4. Push your branch to main repo at end of day (or more often)

    git push origin newfeature

More information

IPPC Repository: https://github.com/hypertexthero/ippcdj - The master branch that should eventually be the same code that is live at production website. Another, likely better, option is to create an IPPC Organization page and move this there.

Translation updates for non-user-generated site content

https://docs.djangoproject.com/en/dev/topics/i18n/translation/

Edit the django.po file for each language in ippcdj_repo/conf/locale/ and then run the following commands in the terminal to compile the translation files:

python manage.py makemessages --all  
python manage.py compilemessages

Deployment & Data Migrations

Dev server exlqaippc2.ext.fao.org setup and configuration for IPPC 4.0 prototype at http://dev.ippc.int/en/ (only available within FAO network). To update code (eventually this will all be done with one command which fires a fabric script such as fab deploy dev):

  1. ssh root@hqldvippc2.hq.un.fao.org

     ssh root@hqldvippc2.hq.un.fao.org
    
  2. Change directory to the project and activate virtualenv

     cd /work/projects/ippcdj-env && . bin/activate
    
  3. Change to repository directory

     cd	ippcdj_repo/
    
  4. Pull latest changes. If you get a warning about overwriting existing changes, do git stash save --keep-index then git stash drop http://stackoverflow.com/a/14318266

     git pull
    
  5. move any static media to proper serving location python manage.py collectstatic Not necessary anymore as we have configured nginx to look in the right places for static media.

  6. Run any data migrations on the database: Right now we're just transfering the whole dump from dev to the MySQL server when models are updated.

     python manage.py schemamigration ippc --auto
     python  manage.py migrate ippc
    
  7. Compile and make translations

     python manage.py makemessages --all  
     python manage.py compilemessages
    

    If some translations, especially ones in blocktrans don't show up, check http://stackoverflow.com/questions/1377372/django-fuzzy-string-translation-not-showing-up

  8. Stop and restart Gunicorn application server:

    This command shows you the master PID of Gunicorn if it is running (http://stackoverflow.com/a/26926130):

     pstree -ap|grep gunicorn
    

    Restart Gunicorn gracefully with the following (replace with the process number, for example, kill -HUP 10745):

     kill -HUP <pid>   
    

    If the above doesn't work and it is an emergency, just do the following (NOTE: THE SITE WILL GO DOWN):

     pkill gunicorn  
     gunicorn_django -b 0.0.0.0:8000 --daemon --log-file /var/log/nginx/gunicorn-beta-ippc-error.log  
    

    Note: If you are starting Gunicorn for the first time, the command is the last one above beginning with gunicorn_django

  9. Restart Nginx reverse-proxy server (web-facing) server

     service nginx restart
    

Example Nginx Configuration

	# http://stackoverflow.com/a/24820722
	map $http_user_agent $limit_bots {
	     default 0;
	     ~*(google|bing|yandex|msnbot) 1;
	     ~*(AltaVista|Googlebot|Slurp|BlackWidow|Bot|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker) 1;
	     ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|Go!Zilla|Go-Ahead-Got-It|GrabNet|Grafula|HMView|Go!Zilla|Go-Ahead-Got-It) 1;
	     ~*(rafula|HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|tool|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE) 1;
	     ~*(GrabNet|NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider) 1;
	     ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus) 1;
	     ~*(Twengabot|htmlparser|libwww|Python|perl|urllib|scan|Curl|email|PycURL|Pyth|PyQ|WebCollector|WebCopy|webcraw) 1;
	 } 
	 
	# http://stackoverflow.com/a/18338015
	ssl_certificate      /etc/pki/tls/certs/name.of.your.cert.chained.crt;
	ssl_certificate_key  /etc/pki/tls/private/name_of_your_key.key;
	ssl_session_timeout  5m;
	ssl_protocols  SSLv3 TLSv1;
	ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+EXP;
	ssl_prefer_server_ciphers   on;
	
	index index.php index.htm index.html;
	
	## www.domain.tld ####################
	
	# redirect http://domain.tld and https://domain.tld to https://www.domain.tld
	server {
	       listen xxx.xx.xx.xxx:80;
				 listen xxx.xx.xx.xxx:443 ssl;
				 server_name domain.tld;
	       return 301 https://www.domain.tld$request_uri;
	}
	
	# redirect http://www.domain.tld to https://www.domain.tld
	server {
	       listen xxx.xx.xx.xxx:80;
				 server_name www.domain.tld;
	       return 301 https://$server_name$request_uri;
	}
	
	server {
		listen xxx.xx.xx.xxx:443 ssl;
		server_name www.domain.tld;
		root /work/projects/ippcdj-env/public;
		
		location /admin/media/ {
			alias /work/projects/ippcdj-env/lib/python2.7/site-packages/django/contrib;
		}
		
		location /static/media {
			alias /work/projects/ippcdj-env/public/static/media;
		}
		
		location /static {
			alias /work/projects/ippcdj-env/ippcdj_repo/static;
		}
		
		location /largefiles {
			alias /work/projects/ippcdj-env/public/largefiles;
		}
		
		location /robots.txt {
			alias /work/projects/ippcdj-env/ippcdj_repo/static/robots.txt;
		}
		
		location / {
			proxy_set_header X-Real-IP  $remote_addr;
			proxy_set_header X-Forwarded-For $remote_addr;
			proxy_set_header Host $host;
			proxy_pass http://127.0.0.1:8000;
		}

		# Deny illegal Host headers
		# http://stackoverflow.com/a/17477436
		 if ($host !~* ^(domain.tld|www.domain.tld)$ ) {
		   return 444;
		 }
		 
		# Deny annoying bots (see mapping at top of file)
		# http://stackoverflow.com/a/24820722
		if ($limit_bots = 1) {
		 	return 403;
		}
		
		# what to serve if upstream is not available or crashes
		error_page 500 502 503 504 /var/www/html/index.html;
		
		# don't display .htaccess files
		location ~ /\.ht {
			deny all;
		}	
	}
	
	## html.staticsitedomain.tld ####################
	server {
		listen xxx.xx.xx.xxx:80;
		server_name html.staticsitedomain.tld;
		root /var/www/html/html.staticsitedomain.tld;
		index index.php index.htm index.html;

		location / {
			try_files $uri $uri/ /index.php?q=$uri&$args;
		}
		
		location ~ /\.ht {
			deny all;
		}
	}
	
	## www.phpsitedomain.tld ####################
	server {
		listen xxx.xx.xx.xxx:80;
		server_name phpsitedomain.tld www.phpsitedomain.tld;
		root /opt/lampp/htdocs/prodsite;
		
		location / {
			try_files $uri $uri/ /index.php?q=$uri&$args;
	  }
		
	  location ~ \.php$ {
	    expires    off;
	    include /etc/nginx/fastcgi_params;
	    fastcgi_pass    127.0.0.1:9000;
	    fastcgi_index   index.php;
	    fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
	  }
		
		location ~ /\.ht {
			deny all;
		}
		error_page 500 502 503 504 /var/www/html/index.html;
	}

MariaDB (MySQL)

MySQL, php and phpmyadmin

Start:

/etc/init.d/mysql start

Permissions System

Server Architecture

Request ----> Reverse-Proxy Server (Nginx)
                 |
                  \                           
                   `-> App Server (Gunicorn). 127.0.0.1:8081 --> Django app

php-fpm

The Django site runs behing a Nginx reverse-proxy server which routes dynamic (non-static requests such as img, pdf, css, js, etc) requests to Gunicorn application server, which routes them to the Django app. Other Drupal/PHP sites on the same server receiving requests from Nginx use php-fpm, which needs to be running:

service php-fpm status
service php-fpm start
# Sometimes php-fpm might have broken instances running, preventing a restart. This command is a clean way to clear them out and restart php-fpm
killall -9 php-fpm; service php-fpm restart

Current Software Stack

The main web application is built with Django and Mezzanine.