Skip to content

scan_itunes Documentation

Jump to bottom
sjlegg edited this page Aug 3, 2012 · 7 revisions

'scan_itunes' is a command that may be executed using Django's manage.py. It updates parts of the iTunes U monitoring app database, and may be run either as a series of cron jobs (to enable nightly/hourly scanning) or directly from the command line.

Notes regarding database structure

Periodicity of records

Version control of historical records

Scanlogs

How to use

There are currently four scan modes available:

1. Institutional Scan

From the command line, run: ./manage.py --mode 1 "Institution Name"

"Institution Name" must be the name of an institution, as listed in the table of institutions in the database. The name must be in quotes if it contains any spaces, and non-ascii names are not yet supported. If the name of the institution is not specified, the script will default to "Oxford University".

This will do the following:

  • Create any new historical records of all the collections owned by the specified institution.
  • Create any new historical records of all the items in these collections.
  • Create records of any new comments found attached to these collections.
  • Create new absolute records of collections and items as necessary.
  • Create new records of genres as necessary.

2. Top Collections Scan

From the command line, run: ./manage.py --mode 2

This will do the following:

  • Create new records of the positions of historical collection records in the iTunes U Top Collections chart.
  • If an historical collection record in the top collections chart cannot be found in the database, then an institutional scan of this collection's institution will be run.
  • Before an institutional scan is run by the top collections scan script for the first time (and only for the first time for each top collections scan) a scan of the institutions page will be run. This means that institutions new to the iTunes U chart will be picked up.
  • If the institution still can't be found (there are a few unlisted institutions on iTunes U), the script will attempt to look it up directly and then scan it.
  • As a last resort, the script will record nothing in a row of the table for which it cannot obtain an historical collection record. In theory, this should never happen...

3. Top Downloads Scan

From the command line, run: ./manage.py --mode 3

This will do the following:

  • Create new records of the positions of historical item records in the iTunes U Top Downloads chart.
  • If an historical item record in the top downloads chart cannot be found in the database, then an institutional scan of this item's institution will be run.
  • Before an institutional scan is run by the top downloads scan script for the first time (and only for the first time for each top downloads scan) a scan of the institutions page will be run. This means that institutions new to the iTunes U chart will be picked up.
  • If the institution still can't be found (there are a few unlisted institutions on iTunes U), the script will attempt to look it up directly and then scan it.
  • As a last resort, the script will record nothing in a row of the table for which it cannot obtain an historical item record. In theory, this should never happen...

4. Institutions Scan

From the command line, run: ./manage.py --mode 4

This scans the following three pages on iTunes U for institutions, and creates/updates their records in the database:

Note that some institutions, for unknown reasons, are not listed on these pages. They will not be detected by an institutions scan, although they will be dealt with should their collections or items show up in a top collections or top downloads scan.

Clone this wiki locally