Scrapetopia

Preparation

Install Python 2.7, pip and a C compiler. Windows users would be far better off using cygwin, though it may work with native Python and exactly MSVC++ 2008.

If you don't have pip, install it with easy_install pip.

Then install dependencies with pip install -r requirements.txt.

Finally, run setup.py to create the database and other directories.

Fetching lecture metadata

Run getallmeta.py to fetch lecture information.

There is no official list of units, and no indication of the highest ID, so by default it tries 0 to 9999, which appears to be sufficient because the highest ID that exists within this range is 5151.

Extracting the media file list

Run getlist.py to dump all media file URLs from the database, except for those which have a corresponding file saved in data/media.

The list of files will be at data/list.txt. Don't try to feed it into a typical download manager like JDownloader, it won't handle it nicely because there can be over 100000 files.

Downloading media files

Run getmedia.py to download media files.

It supports retrying and continuing partial downloads, so feel free to ^C at any time. If you restart the downloader, it'll go through and ping all of the files to make sure they exist and are the right size.

Files that are definitely done will automatically go into data/done.txt, and you can remove these from the main list by running scalpel.sh. Running this is optional but makes restarts faster because the list is shortened.

If you have lost data/done.txt and want to remove completed files in data/media from your current data/list.txt, run redo.py and a new list.txt will be generated without the existing media files.

Backing up media files

You can run backup.py with the main and backup directories supplied as arguments to copy the media files to another location. The need to copy a particular file is determined by a quick check of file sizes, as matching file contents exactly would be very slow.

Hosting the web-based browser

Run web.py to start the application.

Install nginx and use the provided nginx.conf, changing the alias path on line 15 to point to the data/media directory. nginx proxies to the application but is still needed because it is responsible for serving the media files.

Start nginx and the interface will be available on port 8653.

Screenshots