-
Notifications
You must be signed in to change notification settings - Fork 1
Oxford Podcast Management System
This code is part of ongoing work to bring together all the various management and feedback functions related to media publishing at the University of Oxford. This work has been ongoing as low priority task since 2010 and received a boost during the JISC funded Listening for Impact Project.
This documentation is rather lagging behind current development, and is mostly ported from an existing (internal to OUCS) wiki site. Blame most of everything you see here on Carl Marshall ;-)
Install NOTES Here...
Schema Notes?
Webauth notes?
- Analytics app (Stats) - developed initial for the LfI project to aide analysis of Apple and Apache log data, this app is expected to form the core of the analysis for the Podcasting service. (Feedback functionality to be put into an as yet unnamed app) ...More on Stats
- File and Feed Manager app (FFM) - developed initially to allow Stats data to be queried in relation to OxItems information, this application is intended to hold a working catalogue of publishing paths and media assets.
- Core app (Core) - holder of models that should be generic and central to the overall OPMS. Mostly a way of cataloging people/users and some institutional entities.
- Client Relations Manager app (crm) - conceived as a way of filling the gap between people and assets. This may yet be outsourced to another application and in which case this will act as an interface to that application so that we can have all elements of a podcast's management in one location.
- Monitoring app (monitor) - split from the initial Stats app to handle periodic scanning tasks that monitor other systems and sites and report back for podcasting purposes. The first sub function is to monitor the speed of various URLs. The next stage is to scan the iTunes Store periodically to record chart and marketing information.
- Oxitems app (oxitems) - This is an interface app that links to a copy of the legacy metadata system OxItems allowing the OPMS apps to query the data structure directly. The key work here is to map this into a more usable fashion within the FFM.
This is a Python/Django based project, using Twitter's Bootstrap for the UI basics. The production version of this site is run on an Ubuntu 10.04 virtual server, whilst the development is done from a range of Apple Macs. Thus we can say this has been tested on OS X 10.7, and Linux. Those of you wanting to try this on Windows are welcome to experiment and do please let us know of any issues.
THIS STUFF IS ALL MOSTLY OUT OF DATE!!!
== Stats and Feedback app (Stats) ==
'''Aims / Todos''' - Incomplete listing
- DONE - Create summary page for Apple Supplied User Actions
- IN PROGRESS (Non-trivial, see FFM) - Integrate RSS Data from Oxitems
- IN PROGRESS - Link RSS data to Apple Stats data
- ON HOLD (pending FFM work)- Produce reports on a per feed basis using Apple data
- ON HOLD (see Github issue list) - Refactor Apache data model to handle more than 3 months of data
=== Future Development Plans ===
This list has no particular order, and needs updating again... See also the Github Issue list.
'''Import and admin:'''
- Add web upload for import
- Debug issues with browse duplicate data import (apple?)
- Enhance to allow batch processing within a directory and with compressed files
- Determine what other stats can be shown
- Add graphs for longitudinal stats
- Add data overlays to graphics (tooltips showing values under mouse? Drilldown to more data?)
'''Individual stats:'''
- WIP - requires feed info from oxitems to work for both pre/post aug data (post aug could attempt to group via split of path)
- requires some form of access control? Or just limit to Oxford sso access? PR has determined (April 2011) that the work done to show individual feed stats should not be migrated to the production site without access control in place. Consequently this is what is stopping any merging of Master to Production branches in the repository.
'''Refactoring the Apache Data''' Look into performance gains likely offered by MongoDB and then whether some form of mixed solution can be found that can benefit the OPMS. Useful links:
- http://nosql.mypopescu.com/post/408753658/python-django-and-mongodb
- http://blog.montylounge.com/2010/02/11/integrating-mongodb-and-django/
- https://github.com/hmarr/mongoengine
- http://hmarr.com/projects/
- http://mongoengine.org/
- http://mongoengine.org/docs/v0.4/ Related:
- http://bret.appspot.com/entry/how-friendfeed-uses-mysql - alternative approach to dealing with large datasets staying with MySQL
- https://github.com/jamesgolick/degrade - a possible way of monitoring failures and degrading gracefully. Something that we shouldn't be suffering from for a long while yet.
== File and Feed Manager app (FFM) == '''Aims / Todos''' - Incomplete listing
- DONE - Import and remodel OxItems Podcasting data
- ON HOLD (pending workload related to Mediahub project) - Produce templates to allow browsing of data
== Miscellaneous ==
Useful links collected along the way:
'''Setting up the Server'''
- http://www.gyford.com/phil/writing/2010/09/29/django-environment.php - forms the template of the development setup
- http://bryanhelmig.com/setting-up-ubuntu-10-04-with-apache-memcached-ufw-mysql-and-django-1-2-on-linode/ - used a little of this to help start me do the ubuntu setup, but not all of it is applicable.
- https://help.ubuntu.com/10.04/serverguide/C/firewall.html - how to configure the firewall on the vm
- http://stackoverflow.com/questions/3555551/why-does-pip-install-matplotlib-version-0-91-1-when-pypi-shows-version-1-0-0 - reminder of how to install matplotlib from a tarball, as I have had to do to get 1.0.0 on the vm
- http://matplotlib.sourceforge.net/faq/howto_faq.html#generate-images-without-having-a-window-popup - matplotlib does indeed work and can generate images. Here is a reminder of how to do just that.
- Nice walk through of ''Setting up Django using Apache/mod_wsgi on Ubuntu 8.10'' at https://gist.github.com/106077
'''Code Management'''
- https://github.com/oucs/OPMS - where to find the OPMS code!
'''Apache Log Parsing'''
- http://www.the-art-of-web.com/system/logs/ - handy note on how to skim through log files visually using awk. Helpful for basic analysis and structure.
- http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html - http status codes. Need to refer to this to more fully understand what our logs are telling us.
- Apache Log Parser: http://pypi.python.org/pypi/apachelog/1.0 - doing much of the regex grunt work
'''User Agent Analysis'''
- http://user-agent-string.info/download/UASparser-for-Python - for the User Agent String parsing in the apache import function.
'''Geo IP Analysis'''
- http://www.dnspython.org/ - reverse DNS lookups
- https://github.com/haypo/python-ipy - IPy module, handling some of my IP address verification duties
- Database from Maxmind: http://www.maxmind.com/app/geolitecountry
- Client Library: http://code.google.com/p/pygeoip/