Skip to content

A. Trapper Keeper Overview

Bill Fitzgerald edited this page Jan 29, 2022 · 1 revision

The Short Version

If you want to archive information from the web and work with it offline, Trapper Keeper can help you do that.

The (Slightly) Longer Version

Trapper Keeper can be used by anyone that wants to track data and/or information stored online at a specific URL. So, if you have list of pages or documents that can be accessed over the internet, and you want to use the information stored on those pages, Treapper Keeper might help you out.

When you use Trapper Keeper to track urls, you will be able to:

  • Create a local copy of what is stored at that url today;
  • See if the content stored at the url changes over time;
  • Create a clean text version and download any images stored at these locations.

Currently Trapper Keeper processes web pages and pdfs.

Archive URLs

Archiving urls is managed via archive.py.

Start with a csv file with urls you want to archive. The initial intake creates several records.

See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/archive_collect_sample.csv

For every url, archive.py creates:

  • a json file that stores information about the url
  • a txt file that contains the text of the document retrieved from the url

If the url points to a web page, archive.py creates:

  • a copy of the full html of the web page
  • a cleaned snippet that just contains the relevant text from within the web page

If the url points to a pdf, archive.py creates:

  • a downloaded copy of the original file

If you want to add urls to the archive, create a new csv with the additional urls you want to archive.

To check all stored urls for updates, run archive.py -p update. This will check all urls for new versions.

For each url archived by the system:

  • if the data has not changed, then the json file associated with the url has "accessed_on" value updated to the current date/time.
  • if the data has changed, then:
    • the existing json file associated with the url has the "current" value set to "no";
    • a new json file is created for the url with "current" set to "yes", and "first_saved" and "accessed_on" values set to the current date/time;
    • new copies of the text and original source data are archived locally

Now that you have started archiving information stored at various urls, we can begin to use that information.

Track changes in URLs

The show_diffs.py script highlights changes that occur in content over time.

To see changes in content that has occurred since a specific date, create a csv with 4 columns: source_urls, yyyy, mm, dd.

See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/diff_sample.csv

  • source_urls - the url to examine
  • yyyy, mm, and dd should define the date you want to check. yyyy should contain the 4 digit year; mm should contain the two digit month; dd should contain the two digit day: January 3, 2022 would be defined as 2022,01,03

When running the script, it checks for changes that have occurred after the "check" date. Using January 3, 2022 (or 2022,01,03) as an example, if a document had changed on January 4th, the script would surface the difference. If the document had changed on January 2, it would not surface the difference.

If a document has changed multiple times between the "check" date and the present, the script shows a diff between the oldest version after the "check" date and the current version of the document.

To see all versions that have been stored, see the section on "Exporting URL Information" below.

Collect data from specific urls for more detailed analysis

collect_texts.py

The collect_texts.py script groups urls, and exports the data from those urls into a clean directory structure.

For example, if you were creating a series of presentations, you could group urls as Unit 1, Unit 2, and Unit 3, and export content sorted by unit. Then, you could work with that content using whatever tools you wanted.

The collect_texts.py script requires a csv with two columns: collection and source_urls. This csv allows you to select the specific urls you want.

See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/collect_sample.csv

The script also takes two arguments:

  • collect_texts.py -c text exports cleaned text;
  • collect_texts.py -c html exports cleaned text and the original html/pdf file.

Exporting URL Information

To see a complete list of all urls tracked in Trapper Keeper, use export.py.

Current functionality supports exporting information about the most current version of every url, or exporting information about all versions (current and past versions) of urls. If you want to see what urls have changed over time, export information about all versions.

  • export.py -e current exports the latest urls;
  • export.py-e all exports all records.

Maintaining the Data

The housekeeping.py script cleans up any unneeded or untracked files from the data archive.

When you run the housekeeping.py script, all files in the "archive" directory (the directory where content from urls are stored) will be checked, and files that are not connected with any current url will be moved to a "manual_review" directory. In general, all files moved to "manual_review" aren't needed and can be deleted, but because deletion is forever, files should be reviewed on the off chance that data in one of these files contains useful information.