Repository navigation
A. Trapper Keeper Overview
If you want to archive information from the web and work with it offline, Trapper Keeper can help you do that.
Trapper Keeper can be used by anyone that wants to track data and/or information stored online at a specific URL. So, if you have list of pages or documents that can be accessed over the internet, and you want to use the information stored on those pages, Treapper Keeper might help you out.
When you use Trapper Keeper to track urls, you will be able to:
- Create a local copy of what is stored at that url today;
- See if the content stored at the url changes over time;
- Create a clean text version and download any images stored at these locations.
Currently Trapper Keeper processes web pages and pdfs.
Archiving urls is managed via archive.py.
Start with a csv file with urls you want to archive. The initial intake creates several records.
See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/archive_collect_sample.csv
For every url, archive.py creates:
- a json file that stores information about the url
- a txt file that contains the text of the document retrieved from the url
If the url points to a web page, archive.py creates:
- a copy of the full html of the web page
- a cleaned snippet that just contains the relevant text from within the web page
If the url points to a pdf, archive.py creates:
- a downloaded copy of the original file
If you want to add urls to the archive, create a new csv with the additional urls you want to archive.
To check all stored urls for updates, run archive.py -p update. This will check all urls for new versions.
For each url archived by the system:
- if the data has not changed, then the json file associated with the url has "accessed_on" value updated to the current date/time.
- if the data has changed, then:
- the existing json file associated with the url has the "current" value set to "no";
- a new json file is created for the url with "current" set to "yes", and "first_saved" and "accessed_on" values set to the current date/time;
- new copies of the text and original source data are archived locally
Now that you have started archiving information stored at various urls, we can begin to use that information.
The show_diffs.py script highlights changes that occur in content over time.
To see changes in content that has occurred since a specific date, create a csv with 4 columns: source_urls, yyyy, mm, dd.
See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/diff_sample.csv
-
source_urls- the url to examine -
yyyy,mm, andddshould define the date you want to check.yyyyshould contain the 4 digit year;mmshould contain the two digit month;ddshould contain the two digit day: January 3, 2022 would be defined as2022,01,03
When running the script, it checks for changes that have occurred after the "check" date. Using January 3, 2022 (or 2022,01,03) as an example, if a document had changed on January 4th, the script would surface the difference. If the document had changed on January 2, it would not surface the difference.
If a document has changed multiple times between the "check" date and the present, the script shows a diff between the oldest version after the "check" date and the current version of the document.
To see all versions that have been stored, see the section on "Exporting URL Information" below.
collect_texts.py
The collect_texts.py script groups urls, and exports the data from those urls into a clean directory structure.
For example, if you were creating a series of presentations, you could group urls as Unit 1, Unit 2, and Unit 3, and export content sorted by unit. Then, you could work with that content using whatever tools you wanted.
The collect_texts.py script requires a csv with two columns: collection and source_urls. This csv allows you to select the specific urls you want.
See the sample file: https://github.com/billfitzgerald/trapper-keeper/blob/main/samples/collect_sample.csv
The script also takes two arguments:
-
collect_texts.py -c textexports cleaned text; -
collect_texts.py -c htmlexports cleaned text and the original html/pdf file.
To see a complete list of all urls tracked in Trapper Keeper, use export.py.
Current functionality supports exporting information about the most current version of every url, or exporting information about all versions (current and past versions) of urls. If you want to see what urls have changed over time, export information about all versions.
-
export.py -e currentexports the latest urls; -
export.py-e allexports all records.
The housekeeping.py script cleans up any unneeded or untracked files from the data archive.
When you run the housekeeping.py script, all files in the "archive" directory (the directory where content from urls are stored) will be checked, and files that are not connected with any current url will be moved to a "manual_review" directory. In general, all files moved to "manual_review" aren't needed and can be deleted, but because deletion is forever, files should be reviewed on the off chance that data in one of these files contains useful information.