Skip to content
This repository has been archived by the owner on Nov 7, 2018. It is now read-only.

Downloader for FOIAonline #23

Merged
merged 32 commits into from
Aug 28, 2014
Merged

Downloader for FOIAonline #23

merged 32 commits into from
Aug 28, 2014

Commits on Aug 13, 2014

  1. renaming data/ to documents/

    @konklone
    konklone committed Aug 13, 2014
    Configuration menu
    Copy the full SHA
    cd93d78 View commit details
    Browse the repository at this point in the history

  2. shell

    @konklone
    konklone committed Aug 13, 2014
    Configuration menu
    Copy the full SHA
    592b617 View commit details
    Browse the repository at this point in the history

  3. allow import if needed

    @konklone
    konklone committed Aug 13, 2014
    Configuration menu
    Copy the full SHA
    87f573a View commit details
    Browse the repository at this point in the history

  4. ignore documents/ now

    @konklone
    konklone committed Aug 13, 2014
    Configuration menu
    Copy the full SHA
    9d5746d View commit details
    Browse the repository at this point in the history

  5. 3.4.1

    @konklone
    konklone committed Aug 13, 2014
    Configuration menu
    Copy the full SHA
    c23c4cc View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2014

  1. Merge branch 'master' into foiaonline

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    5e5d285 View commit details
    Browse the repository at this point in the history

  2. just to suppress a warning

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    b375414 View commit details
    Browse the repository at this point in the history

  3. moving stuff around

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    8853e39 View commit details
    Browse the repository at this point in the history

  4. use urlretrieve, update scrapelib

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    f4cd20b View commit details
    Browse the repository at this point in the history

  5. one more shuffle

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    1c81de2 View commit details
    Browse the repository at this point in the history

  6. ignoring things

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    6394447 View commit details
    Browse the repository at this point in the history

  7. session preservation and posting works

    @konklone
    konklone committed Aug 19, 2014
    Configuration menu
    Copy the full SHA
    05beedd View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2014

  1. okay we are saving metadata to disk now this is good

    @konklone
    konklone committed Aug 20, 2014
    Configuration menu
    Copy the full SHA
    13be975 View commit details
    Browse the repository at this point in the history

  2. now it can download all the pages

    @konklone
    konklone committed Aug 20, 2014
    Configuration menu
    Copy the full SHA
    108574e View commit details
    Browse the repository at this point in the history

  3. add appeals/referrals, document agency slugs, don't overwrite meta

    @konklone
    konklone committed Aug 20, 2014
    Configuration menu
    Copy the full SHA
    e5a6001 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2014

  1. dang - assumption that tracking number was unique per-document very w… 

    …rong.
    @konklone
    konklone committed Aug 21, 2014
    Configuration menu
    Copy the full SHA
    fcf44d7 View commit details
    Browse the repository at this point in the history

  2. helper script to archive metadata for foiaonline

    @konklone
    konklone committed Aug 21, 2014
    Configuration menu
    Copy the full SHA
    9d6c1e3 View commit details
    Browse the repository at this point in the history

  3. downloading record PDFs and extracting text

    @konklone
    konklone committed Aug 21, 2014
    Configuration menu
    Copy the full SHA
    cfe4ca6 View commit details
    Browse the repository at this point in the history

  4. some basic dir iteration over downloaded metadata

    @konklone
    konklone committed Aug 21, 2014
    Configuration menu
    Copy the full SHA
    fe20c71 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2014

  1. scraping more metadata, handling other file types, not caching landin… 

    …g pages
    @konklone
    konklone committed Aug 22, 2014
    Configuration menu
    Copy the full SHA
    f8d5833 View commit details
    Browse the repository at this point in the history

  2. ignore exemption 5 subtypes explicitly for now

    @konklone
    konklone committed Aug 22, 2014
    Configuration menu
    Copy the full SHA
    b416eaf View commit details
    Browse the repository at this point in the history

  3. allow scoping of data fetching

    @konklone
    konklone committed Aug 22, 2014
    Configuration menu
    Copy the full SHA
    252ed8c View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2014

  1. added resume mode to data download for records

    @konklone
    konklone committed Aug 25, 2014
    Configuration menu
    Copy the full SHA
    4f96a67 View commit details
    Browse the repository at this point in the history

  2. comment more

    @konklone
    konklone committed Aug 25, 2014
    Configuration menu
    Copy the full SHA
    a9a116e View commit details
    Browse the repository at this point in the history

  3. Handle unreleased docs

    @konklone
    konklone committed Aug 25, 2014
    Configuration menu
    Copy the full SHA
    e44bd00 View commit details
    Browse the repository at this point in the history

  4. comments on file types

    @konklone
    konklone committed Aug 25, 2014
    Configuration menu
    Copy the full SHA
    ded048f View commit details
    Browse the repository at this point in the history

  5. update resume language

    @konklone
    konklone committed Aug 25, 2014
    Configuration menu
    Copy the full SHA
    2e43ab8 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2014

  1. always download docs as binary from foiaonline

    @konklone
    konklone committed Aug 26, 2014
    Configuration menu
    Copy the full SHA
    4f2abd5 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2014

  1. okay ignore misc/

    @konklone
    konklone committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    eb1aaac View commit details
    Browse the repository at this point in the history

  2. capture original release date, handle unexpected dates gracefully

    @konklone
    konklone committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    5434f14 View commit details
    Browse the repository at this point in the history

  3. not in use yet, but wrote a function to get header details for binaries 

    would be annoying to re-download everything, so may end up using this in
a batch task, but in theory this should run on its own, and be saved as
a separate file-headers.json or something, and cached alongside the rest,
to be used in determine file type and extension for the downloaded
binary file.
    @konklone
    konklone committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    b9b4c43 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2014

  1. Merge branch 'master' into foiaonline 

    Conflicts:
	.gitignore
    @konklone
    konklone committed Aug 28, 2014
    Configuration menu
    Copy the full SHA
    43af6a7 View commit details
    Browse the repository at this point in the history