Multi-level, progressive data collection

[pombredanne]

Rather than having everything all the time in the PurlDB, I would like to have a way where:

1. New versions of packages are kept current across all supported ecosystems, and possibly all watched PURLs ( as with Continuously watch a PURL of interest, aka. purlwatch #244)

2. On demand or on watch, we can keep current with metadata (fetched from ecosystem repo APIs) for new versions.

3. On demand or on watch, we can keep current with scans (running a full ScanCode) on these PURLs.
   We would need some property stored on the Package model to track the status may be with a date for 2. and 3.

• to track the date of last metadata fetch for this PURL.
• to track the date and ScanCode version of the last ScanCode scan for this PURL.

Could we store a a PURL without version where we would store "control" information across the whole package, ignoring versions? This could a way track the last time we checked for versions, and also where we store register interest to "watch", and possibly store the latest and greatest data about a package and a few statistics. It could be derived from the Package model with a few extra fields. Something like a "MetaPackage". See also #186 for this.

[pombredanne]

See also:

• For level 2, aka. metadata, see Collect core package metadata on demand, live, aka. purl2meta #242
• For level 3, aka. scancode scan, see Endpoint to run scanning tool(s) on demand, live, aka. purl2scancode #243