Skip to content

scan_itunes Documentation

Jump to bottom
sjlegg edited this page Aug 6, 2012 · 7 revisions

'scan_itunes' is a command that may be executed using Django's manage.py. It updates parts of the iTunes U monitoring app database, and may be run either as a series of cron jobs (to enable nightly/hourly scanning) or directly from the command line.

Notes regarding database structure

Collections and Items

The database is built around the two key concepts Apple uses in iTunes - collections and items. Items correspond to individual podcasts/videos and their metadata. Collections are groups of items. Both are also grouped by institution and genre.

Periodicity of records

There are three types of records of both collections and items. This is motivated by the need for efficiency when processing large numbers of data, and by the need for a version control system (since the presentation of collections and items on iTunes U changes over time). The types of record are as follows:

1. Absolute records

Each item and collection should have one, and only one, absolute record. This record contains very little information, as most of the information describing any given collection or item can and does change over time. In fact, only the institution of the absolute record is stored for convenience when scanning; this should be permanent.

Multiple absolute records will only exist if an institutional scan has failed to pick up that two historical records in fact refer to the same entity. This could be due to a bug (which should, of course, be fixed!), or could be due to a change so radical that only an human intelligence could pick up that the two historical records do indeed refer to that same entity. If so, the two absolute records should be merged. Automatic merging has not yet been implemented, so this will need to be done manually.

Absolute records are created when a brand new collection or item is first spotted, and should not be changed thereafter.

2. Historical records

Historical records contain the bulk of the metadata for a given collection or item. They represent this metadata as seen on a particular scan, thus allowing the presentation of collections and items to be tracked through time.

Ratings (ie. how many people rated a collection with how many stars) are stored in a separate table, but this is really just an extension of an historical collection record. Comments are also attached to historical collection records, but a change in a comment does not constitute a change in the historical collection record. Comments will only ever be stored once, and will be linked to the historical collection record which was made during the scan on which the comment was first sighted. Note that ratings and comments do not exist for historical item records, since these do not exist on iTunes.

New historical records are created during institutional scans. They are never updated. Instead, a version control system is used to link them together through time (see below).

3. Chart Scan records

Whenever a top collections scan or top downloads scan is executed, one chart scan record is created for each row in the chart scanned at that time. These records contain a position in the chart, a link to the historical record at that position, and a link to the scanlog.

New sets of records should be created regularly (potentially hourly), and so there will be a lot of them. So, in order to improve database efficiency (when fetching data for a graph, for example), the datetime of the scan, and a link to the relevant absolute record are stored in the chart scan tables themselves, even though this is duplicate information.

Version control of historical records

Whenever a new historical record is created, it is given a version number, starting at 1. This version number is then incremented by 1 for each new historical record for a given absolute record. An historical record also contains a link to the previous historical record for its absolute record.

Historical records have three methods which may be used to navigate from one such record to another:

  • next(self) finds the next-most-recent historical record. If there is no such record (ie. the historical record is the latest record), next(self) returns None.
  • original(self) returns the first historical record in the chain. This could be itself.
  • latest(self) returns the last historical record in the chain. This could be itself. This method should not be used without good reason - using self.itucollection.latest or self.ituitem.latest is more efficient.

To improve efficiency, whenever a new historical record is created, the 'latest' field in the corresponding absolute record is updated to contain the new historical record. Absolute records do have the method find_latest(self), but this should not be used without good reason for efficiency's sake. However, the method find_original(self) remains the most efficient way of finding the original historical record for an absolute record.

Historical records also have a 'missing' field. This is only relevant if an historical record is the latest available record for a given absolute record. If this record was visible last time an institutional scan of its institution was carried out, 'missing' will be null. If it was missing, 'missing' will contain a link to the institutional scanlog on which it was first discovered to be missing.

Scanlogs

Historical records and chart scan records are all linked to the scanlog on which they were created. This scanlog describes the scan, including an institution field which is non-null only if the scan was an institutional scan.

How to use

There are currently four scan modes available:

1. Institutional Scan

From the command line, run: ./manage.py scan_itunes --mode 1 "Institution Name"

"Institution Name" must be the name of an institution, as listed in the table of institutions in the database. The name must be in quotes if it contains any spaces, and non-ascii names are not yet supported. If the name of the institution is not specified, the script will default to "Oxford University".

This will do the following:

  • Create any new historical records of all the collections owned by the specified institution.
  • Create any new historical records of all the items in these collections.
  • Create records of any new comments found attached to these collections.
  • Create new absolute records of collections and items as necessary.
  • Create new records of genres as necessary.
  • Check to see if any collections or items have gone missing or reappeared, and update the 'missing' fields of historical records as appropriate.

2. Top Collections Scan

From the command line, run: ./manage.py scan_itunes --mode 2

This will do the following:

  • Create new records of the positions of historical collection records in the iTunes U Top Collections chart.
  • If an historical collection record in the top collections chart cannot be found in the database, then an institutional scan of this collection's institution will be run.
  • Before an institutional scan is run by the top collections scan script for the first time (and only for the first time for each top collections scan) a scan of the institutions page will be run. This means that institutions new to the iTunes U chart will be picked up.
  • If the institution still can't be found (there are a few unlisted institutions on iTunes U), the script will attempt to look it up directly and then scan it.
  • As a last resort, the script will record nothing in a row of the table for which it cannot obtain an historical collection record. In theory, this should never happen...

3. Top Downloads Scan

From the command line, run: ./manage.py scan_itunes --mode 3

This will do the following:

  • Create new records of the positions of historical item records in the iTunes U Top Downloads chart.
  • If an historical item record in the top downloads chart cannot be found in the database, then an institutional scan of this item's institution will be run.
  • Before an institutional scan is run by the top downloads scan script for the first time (and only for the first time for each top downloads scan) a scan of the institutions page will be run. This means that institutions new to the iTunes U chart will be picked up.
  • If the institution still can't be found (there are a few unlisted institutions on iTunes U), the script will attempt to look it up directly and then scan it.
  • As a last resort, the script will record nothing in a row of the table for which it cannot obtain an historical item record. In theory, this should never happen...

4. Institutions Scan

From the command line, run: ./manage.py scan_itunes --mode 4

This scans the following three pages on iTunes U for institutions, and creates/updates their records in the database:

Note that some institutions, for unknown reasons, are not listed on these pages. They will not be detected by an institutions scan, although they will be dealt with should their collections or items show up in a top collections or top downloads scan.

Initial setup

There is one fixture which should be imported automatically when ./manage.py syncdb monitors is executed. To populate the database initially, it should be sufficient to run a top collections scan and a top downloads scan from the command line.

Things to change in settings.py

You will of course need to configure your database, settings.py and settings_default.py correctly to get the OPMS app as a whole to run correctly. However, there are two changes you may wish to make to settings_default.py:

  • Change YOUR_INSTITUTION from "Oxford University" to your institution's name, as stored in the institutions table.
  • Set WAIT_GET to False if you do not want chart scans to wait between page gets (spreads the load on Apple's servers out).

Recommended cron configuration

Assuming you are an academic institution wishing to track the presentation of your podcasts in iTunes U, the recommended cron jobs to run are as follows (with the right path to manage.py!):

  • Hourly: manage.py scan_itunes --mode 2; manage.py scan_itunes --mode 3
  • Daily: manage.py scan_itunes --mode 1 "Your Institution Name"

This will mean that the charts are tracked hourly, doing institutional scans as necessary. This will also mean that your institution's records are kept up-to-date, and in particular that any new comments regarding your podcasts are found each day.

Clone this wiki locally