Skip to content
Browse files

Reorganise some package structure, and move installer to use setuptoo…

…ls (with implicit dependencies)
  • Loading branch information...
1 parent 24a5c41 commit 46debaf5913be11d12473e2d8817865615be067c @cnorthwood cnorthwood committed Nov 12, 2010
View
4 .gitignore
@@ -2,4 +2,6 @@
*.DS_Store
build/
docs/build/
-local/
+local/
+molly.egg-info/
+dist/
View
96 INSTALL
@@ -21,39 +21,83 @@ Debian packages or their equivalents for your distribution or OS:
* libldap2-dev
* libsasl2-dev
-postgis python-gdal proj libgeos-3.0.0 binutils libgdal1-1.5.0 postgresql-8.3 postgresql-8.3-postgis postgresql-server-dev-8.3 python-virtualenv python-setuptools python-dev libxslt-dev libldap2-dev libsasl2-dev
-
Unfortunately, pip isn't yet in Debian stable, so you'll need to install it
using easy_install:
$ aptitude install python-setuptools
$ easy_install pip
-Next, set up and activate a virtualenv. This has been automated as
-scripts/install_virtualenv.py. If you want to do this by hand, do the following:
-
- $ virtualenv --no-site-packages ~/mollyenv
- $ cd ~/mollyenv
- $ source bin/activate
-
- Next, clone the git repository to a temporary directory:
-
- $ git clone git://mollyproject.git.sourceforge.net/gitroot/mollyproject/mollyproject
-
- Install the Molly dependencies and Molly itself:
-
- $ cd ~/mollyenv/mollyproject/branches/extracting_oxford/molly/
- $ pip -E ~/mollyenv/ install `cat requirements/core.txt` --upgrade
- $ python setup.py install
+On Fedora, this list is:
-Once you've set up the database as below, you'll be able to run the development
-server as follows:
+ * python-virtualenv
+ * python-pip
+ * libxml-devel
+ * libxslt-devel
+ * python-devel
+ * postgresql-devel
+ * openldap-devel
+ * openssl-devel
+ * postgis
+ * gdal-python
+ * proj
+ * postgresql-server
+ * geos
+ * httpd
+ * libjpeg-devel
+
+Next, you'll want to deploy Molly, this can be done in the usual way. This uses
+setuptools to install all your dependencies. If you just want to play with
+Molly, then it may make more sense to set up a virtualenv[1] and install Molly
+into that.
+
+Unfortunately, one of the dependencies on PyPI is actually broken - you will
+need to install a fixed version from http://github.com/alexdutton/PyZ3950/
+BEFORE running the setup script, otherwise the broken version will get
+installed:
+
+ $ pip install git+http://github.com/alexdutton/PyZ3950.git
+
+ $ python setup.py install
+
+If you don't have network access, you will need to install the following Python
+packages manually (from PyPI unless otherwise listed):
+
+ * Levenshtein (SVN at http://pylevenshtein.googlecode.com/svn/trunk)
+ * pywurfl
+ * ply
+ * feedparser
+ * simplejson
+ * rdflib
+ * pytz
+ * python-dateutil
+ * Django
+ * oauth (version 1.0.1)
+ * psycopg2
+ * PIL
+ * lxml
+ * python-ldap
+ * django-compress
+ * python-memcached
+ * django-staticfiles
+
+You'll then need to set up the database as below, and configure a deployment
+of Molly. A simple deployment is provided in demo/. You'll need to give
+settings.py a once over, and then build the media for the site:
+
+ $ python manage.py build_static --noinput
+ $ python manage.py synccompress
+ $ python manage.py generate_markers
+
+You will also need to set up the cron jobs required for batch processing to
+work. If you don't have a crontab at present, simply running:
+
+ $ PYTHONPATH=.. python manage.py create_crontab | crontab
+
+will do the job, if not, don't pipe it to crontab and then add the required
+lines by hand. Once this is done, you can finally start a test server.
- $ cd ~/mollyenv/mollyproject/branches/extracting_oxford/molly/demos/molly_oxford
$ python manage.py runserver
-
-
Database
--------
@@ -74,8 +118,7 @@ Once this is done, you should create the molly database as a DB superuser
As a DB superuser, create the molly user:
-
-(Make sure the password is the one given in settings.py in the demos folder)
+(Make sure the password is the one given in settings.py in the demo folder)
$ createuser molly -P
Enter password for new role:
@@ -118,7 +161,8 @@ database:
It should ask you to create a new superuser. Say yes and set one up.
-The site also requires a cache directory, currently /var/cache/molly .
+The site also requires a cache directory, currently /var/cache/molly.
Create this directory and give user www-data and/or your developer users/group
all permissions to do things to it.
+[1] http://pypi.python.org/pypi/virtualenv
View
0 __init__.py
No changes.
View
284 ez_setup.py
@@ -0,0 +1,284 @@
+#!python
+"""Bootstrap setuptools installation
+
+If you want to use setuptools in your package's setup.py, just include this
+file in the same directory with it, and add this to the top of your setup.py::
+
+ from ez_setup import use_setuptools
+ use_setuptools()
+
+If you want to require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, you can do so by supplying
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import sys
+DEFAULT_VERSION = "0.6c11"
+DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
+
+md5_data = {
+ 'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca',
+ 'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb',
+ 'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b',
+ 'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a',
+ 'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618',
+ 'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac',
+ 'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5',
+ 'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4',
+ 'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c',
+ 'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b',
+ 'setuptools-0.6c10-py2.3.egg': 'ce1e2ab5d3a0256456d9fc13800a7090',
+ 'setuptools-0.6c10-py2.4.egg': '57d6d9d6e9b80772c59a53a8433a5dd4',
+ 'setuptools-0.6c10-py2.5.egg': 'de46ac8b1c97c895572e5e8596aeb8c7',
+ 'setuptools-0.6c10-py2.6.egg': '58ea40aef06da02ce641495523a0b7f5',
+ 'setuptools-0.6c11-py2.3.egg': '2baeac6e13d414a9d28e7ba5b5a596de',
+ 'setuptools-0.6c11-py2.4.egg': 'bd639f9b0eac4c42497034dec2ec0c2b',
+ 'setuptools-0.6c11-py2.5.egg': '64c94f3bf7a72a13ec83e0b24f2749b2',
+ 'setuptools-0.6c11-py2.6.egg': 'bfa92100bd772d5a213eedd356d64086',
+ 'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27',
+ 'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277',
+ 'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa',
+ 'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e',
+ 'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e',
+ 'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f',
+ 'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2',
+ 'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc',
+ 'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167',
+ 'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64',
+ 'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d',
+ 'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20',
+ 'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab',
+ 'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
+ 'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2',
+ 'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e',
+ 'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372',
+ 'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902',
+ 'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de',
+ 'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b',
+ 'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03',
+ 'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a',
+ 'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6',
+ 'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a',
+}
+
+import sys, os
+try: from hashlib import md5
+except ImportError: from md5 import md5
+
+def _validate_md5(egg_name, data):
+ if egg_name in md5_data:
+ digest = md5(data).hexdigest()
+ if digest != md5_data[egg_name]:
+ print >>sys.stderr, (
+ "md5 validation of %s failed! (Possible download problem?)"
+ % egg_name
+ )
+ sys.exit(2)
+ return data
+
+def use_setuptools(
+ version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+ download_delay=15
+):
+ """Automatically find/download setuptools and make it available on sys.path
+
+ `version` should be a valid setuptools version number that is available
+ as an egg for download under the `download_base` URL (which should end with
+ a '/'). `to_dir` is the directory where setuptools will be downloaded, if
+ it is not already available. If `download_delay` is specified, it should
+ be the number of seconds that will be paused before initiating a download,
+ should one be required. If an older version of setuptools is installed,
+ this routine will print a message to ``sys.stderr`` and raise SystemExit in
+ an attempt to abort the calling script.
+ """
+ was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules
+ def do_download():
+ egg = download_setuptools(version, download_base, to_dir, download_delay)
+ sys.path.insert(0, egg)
+ import setuptools; setuptools.bootstrap_install_from = egg
+ try:
+ import pkg_resources
+ except ImportError:
+ return do_download()
+ try:
+ pkg_resources.require("setuptools>="+version); return
+ except pkg_resources.VersionConflict, e:
+ if was_imported:
+ print >>sys.stderr, (
+ "The required version of setuptools (>=%s) is not available, and\n"
+ "can't be installed while this script is running. Please install\n"
+ " a more recent version first, using 'easy_install -U setuptools'."
+ "\n\n(Currently using %r)"
+ ) % (version, e.args[0])
+ sys.exit(2)
+ else:
+ del pkg_resources, sys.modules['pkg_resources'] # reload ok
+ return do_download()
+ except pkg_resources.DistributionNotFound:
+ return do_download()
+
+def download_setuptools(
+ version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+ delay = 15
+):
+ """Download setuptools from a specified location and return its filename
+
+ `version` should be a valid setuptools version number that is available
+ as an egg for download under the `download_base` URL (which should end
+ with a '/'). `to_dir` is the directory where the egg will be downloaded.
+ `delay` is the number of seconds to pause before an actual download attempt.
+ """
+ import urllib2, shutil
+ egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3])
+ url = download_base + egg_name
+ saveto = os.path.join(to_dir, egg_name)
+ src = dst = None
+ if not os.path.exists(saveto): # Avoid repeated downloads
+ try:
+ from distutils import log
+ if delay:
+ log.warn("""
+---------------------------------------------------------------------------
+This script requires setuptools version %s to run (even to display
+help). I will attempt to download it for you (from
+%s), but
+you may need to enable firewall access for this script first.
+I will start the download in %d seconds.
+
+(Note: if this machine does not have network access, please obtain the file
+
+ %s
+
+and place it in this directory before rerunning this script.)
+---------------------------------------------------------------------------""",
+ version, download_base, delay, url
+ ); from time import sleep; sleep(delay)
+ log.warn("Downloading %s", url)
+ src = urllib2.urlopen(url)
+ # Read/write all in one block, so we don't create a corrupt file
+ # if the download is interrupted.
+ data = _validate_md5(egg_name, src.read())
+ dst = open(saveto,"wb"); dst.write(data)
+ finally:
+ if src: src.close()
+ if dst: dst.close()
+ return os.path.realpath(saveto)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+def main(argv, version=DEFAULT_VERSION):
+ """Install or upgrade setuptools and EasyInstall"""
+ try:
+ import setuptools
+ except ImportError:
+ egg = None
+ try:
+ egg = download_setuptools(version, delay=0)
+ sys.path.insert(0,egg)
+ from setuptools.command.easy_install import main
+ return main(list(argv)+[egg]) # we're done here
+ finally:
+ if egg and os.path.exists(egg):
+ os.unlink(egg)
+ else:
+ if setuptools.__version__ == '0.0.1':
+ print >>sys.stderr, (
+ "You have an obsolete version of setuptools installed. Please\n"
+ "remove it from your system entirely before rerunning this script."
+ )
+ sys.exit(2)
+
+ req = "setuptools>="+version
+ import pkg_resources
+ try:
+ pkg_resources.require(req)
+ except pkg_resources.VersionConflict:
+ try:
+ from setuptools.command.easy_install import main
+ except ImportError:
+ from easy_install import main
+ main(list(argv)+[download_setuptools(delay=0)])
+ sys.exit(0) # try to force an exit
+ else:
+ if argv:
+ from setuptools.command.easy_install import main
+ main(argv)
+ else:
+ print "Setuptools version",version,"or greater has been installed."
+ print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)'
+
+def update_md5(filenames):
+ """Update our built-in md5 registry"""
+
+ import re
+
+ for name in filenames:
+ base = os.path.basename(name)
+ f = open(name,'rb')
+ md5_data[base] = md5(f.read()).hexdigest()
+ f.close()
+
+ data = [" %r: %r,\n" % it for it in md5_data.items()]
+ data.sort()
+ repl = "".join(data)
+
+ import inspect
+ srcfile = inspect.getsourcefile(sys.modules[__name__])
+ f = open(srcfile, 'rb'); src = f.read(); f.close()
+
+ match = re.search("\nmd5_data = {\n([^}]+)}", src)
+ if not match:
+ print >>sys.stderr, "Internal error!"
+ sys.exit(2)
+
+ src = src[:match.start(1)] + repl + src[match.end(1):]
+ f = open(srcfile,'w')
+ f.write(src)
+ f.close()
+
+
+if __name__=='__main__':
+ if len(sys.argv)>2 and sys.argv[1]=='--md5update':
+ update_md5(sys.argv[2:])
+ else:
+ main(sys.argv[1:])
+
+
+
+
+
+
View
18 requirements/core.txt
@@ -1,18 +0,0 @@
-svn+http://pylevenshtein.googlecode.com/svn/trunk@2#egg=Levenshtein
-pywurfl
-ply
-git+http://github.com/alexdutton/PyZ3950.git#egg=PyZ3950
-feedparser
-simplejson
-rdflib
-pytz
-python-dateutil
-Django
-oauth==1.0.1
-psycopg2
-PIL
-lxml
-python-ldap
-django-compress
-python-memcached
-django-staticfiles
View
1 requirements/fedora-package-list
@@ -1 +0,0 @@
-python-virtualenv python-pip libxml-devel libxslt-devel python-devel postgresql-devel openldap-devel openssl-devel postgis gdal-python proj postgresql-server geos httpd libjpeg-devel
View
0 scripts/create_template_postgis-debian.sh 100644 → 100755
File mode changed.
View
0 scripts/create_template_postgis-fedora.sh 100644 → 100755
File mode changed.
View
0 scripts/create_template_postgis-ubuntu.sh 100644 → 100755
File mode changed.
View
5 scripts/deploy-molly.sh
@@ -6,14 +6,17 @@ if [ -n "$1" ] ; then
DIR="$( cd "$( dirname "$0" )" && pwd )"
# Set up the virtual environment
- python $DIR/install_virtualenv.py $1
+ virtualenv --distribute --no-site-packages $1
# An empty directory is needed here otherwise things error later
mkdir $1/lib/python`python -V 2>&1 | cut -d" " -f2`/site-packages/molly/media
# Go inside the Python virtual environment
source $1/bin/activate
+ # Install our PyZ3950, because the PyPI one is broken
+ pip install git+http://github.com/alexdutton/PyZ3950.git
+
# Install Molly
python $DIR/../setup.py install
View
43 scripts/findtemplates.py
@@ -0,0 +1,43 @@
+import re, sys, os, os.path
+
+templates = ''
+temps = set()
+for (dirpath, dirnames, filenames) in os.walk('.'):
+ path, dir = os.path.split(os.path.normpath(dirpath))
+ path, tdir = os.path.split(os.path.normpath(os.path.join(dirpath, '..')))
+ path, t2dir = os.path.split(os.path.normpath(os.path.join(dirpath, '..', '..')))
+ if tdir == 'templates':
+ for filename in filenames:
+ file, ext = os.path.splitext(filename)
+ temps.add(dir + '/' + file)
+ with open(os.path.join(dirpath, filename)) as fd:
+ templates += fd.read()
+ elif t2dir == 'templates' and tdir != 'site-media':
+ for filename in filenames:
+ file, ext = os.path.splitext(filename)
+ temps.add(tdir + '/' + dir + '/' + file)
+ with open(os.path.join(dirpath, filename)) as fd:
+ templates += fd.read()
+
+used_temps = set()
+for m in re.finditer(r"(render\(\s*.+?\s*,\s*.+?\s*,|get_template\()\s*'(.+?)'\s*\)", sys.stdin.read()):
+ used_temps.add(m.group(2))
+
+for m in re.finditer(r'{% (include|extends) "(.+?).x?html" %}', templates):
+ used_temps.add(m.group(2))
+
+
+print "The following templates exist are are used:"
+for t in sorted(temps & used_temps):
+ print t
+print
+
+print "The following templates exist, but are not used:"
+for t in sorted(temps - used_temps):
+ print t
+print
+
+print "The following templates are used, but do not exist:"
+for t in sorted(used_temps - temps):
+ print t
+print
View
75 scripts/install_virtualenv.py
@@ -1,75 +0,0 @@
-#!/usr/bin/env python
-
-import os, sys, os.path, subprocess, shutil, traceback
-
-def copy_demos(source_path, deploy_path):
- # Copy the demos across
- shutil.copytree(
- os.path.join(source_path, 'demos'),
- os.path.join(deploy_path, 'demos'),
- )
-
-def main(source_path, deploy_path):
- if os.path.exists(deploy_path):
- print "Cannot deploy to path - already exists"
- return 1
-
- commands = [
- ('Creating', 'virtual environment', ["virtualenv", "--distribute", "--no-site-packages", deploy_path]),
- ]
-
- requirements = [l[:-1] for l in open(os.path.join(source_path, "requirements", "core.txt")) if l[:-1]]
-
- for requirement in requirements:
- commands.append(
- ('Installing', requirement,
- [os.path.join(deploy_path, "bin", "pip"), "install", "-U", requirement])
- )
-
- commands += [
- ('Deploying', 'molly',
- [os.path.join(deploy_path, "bin", "python"), os.path.join(source_path, "setup.py"), "install"]),
- ('Copying', 'demos', copy_demos),
- ]
-
- stdout_log = open('molly.stdout.log', 'w')
- stderr_log = open('molly.stderr.log', 'w')
- succeeded = True
- for i, (action, item, command) in enumerate(commands):
- print "%s %s (%2d/%2d)" % (action[:12].ljust(12), item[:40].ljust(40), (i+1), len(commands)),
- if callable(command):
- try:
- return_code = command(source_path, deploy_path) or 0
- except Exception, e:
- return_code = 1
- traceback.print_exc(file=stderr_log)
- else:
- try:
- return_code = subprocess.call(command, stdout=stdout_log, stderr=stderr_log)
- except OSError, e:
- print "\n", ("No such shell command: %r" % command[0]).ljust(61),
- return_code = 1
- print "[%s]" % ('FAILED' if return_code else ' OK ')
- succeeded = succeeded and (return_code == 0)
-
- if succeeded:
- print """
-Molly was successfully installed to %(deploy_path)s.
-The following command will take you inside your virtualenv:
-
-$ source %(activate)s""" % {
- 'deploy_path': deploy_path,
- 'activate': os.path.join(deploy_path, "bin", "activate"),
- }
- else:
- print """
-The installation failed. You may find useful information in molly.stdout.log
-and molly.stderr.log. For assistance, please e-mail the mailing list at
-mollyproject-devel@lists.sourceforge.net or join the #molly IRC channel on
-irc.freenode.net."""
-
-
-if __name__ == '__main__':
- source_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
- deploy_path = os.path.abspath(sys.argv[1])
- exit(main(source_path, deploy_path) or 0)
View
32 setup.py 100644 → 100755
@@ -1,4 +1,7 @@
-from distutils.core import setup
+import ez_setup
+ez_setup.use_setuptools()
+
+from setuptools import setup
from distutils.command.install import INSTALL_SCHEMES
import os
@@ -51,7 +54,7 @@ def fullsplit(path, result=None):
setup(
name = 'molly',
- version = '0.3',
+ version = '0.4',
url = 'http://mollyproject.org/',
author = 'University of Oxford',
description ="A framework for building mobile information portals",
@@ -67,6 +70,31 @@ def fullsplit(path, result=None):
'Topic :: Education',
'Topic :: Internet',
],
+ install_requires = [
+ "python-Levenshtein",
+ "pywurfl",
+ "ply",
+ "PyZ3950", # The one in PyPI is broken! You should install the one from
+ # https://github.com/alexdutton/PyZ3950/ *BEFORE* running
+ # this script
+ "feedparser",
+ "simplejson",
+ "rdflib",
+ "pytz",
+ "python-dateutil",
+ "Django",
+ "oauth==1.0.1",
+ "psycopg2",
+ "PIL",
+ "lxml",
+ "python-ldap",
+ "django-compress",
+ "python-memcached",
+ "django-staticfiles",
+ ],
+ dependency_links = [
+ 'http://pylevenshtein.googlecode.com/files/python-Levenshtein-0.10.1.tar.bz2#egg=python-Levenshtein'
+ ]
)

0 comments on commit 46debaf

Please sign in to comment.
Something went wrong with that request. Please try again.