Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Back fill the Raw Inmate Data to August 17th, 2013 #448

Open
nwinklareth opened this issue Jun 15, 2014 · 7 comments

Comments

@nwinklareth
Copy link
Contributor

commented Jun 15, 2014

This is part of doing #395.

To make the raw Inmate data useful and to support need to populate the V2.0 API database, the data must be backed filled, to Auguts 17th, 2013.

The rational for the starting date and one design, not the design that has to be used, on how to do this is located on the Populating-the-Raw-Inmate-Data-from-the-V1.0-API-Database page.

@wilbertom

This comment has been minimized.

Copy link
Member

commented Jun 30, 2014

I should finish this tonight or tomorrow night.

@bepetersn

This comment has been minimized.

Copy link
Member

commented Jun 30, 2014

This whole issue?! That would be great!

On Mon, Jun 30, 2014 at 9:42 AM, Wilberto Morales notifications@github.com
wrote:

I should finish this tonight or tomorrow night.


Reply to this email directly or view it on GitHub
#448 (comment).

Brian Peterson

@bepetersn

This comment has been minimized.

Copy link
Member

commented Jun 30, 2014

Something weird happened on the 27th of June, that resulted in no data dump being made for that date of the scraping run. (See future issue...) However, the scraper didn't totally fail, and the v1 database contains this data.

So I would request that this data be recreated in the same fashion as this issue requires--whatever program is made to fulfill this issue should be kept around, certainly until we know what caused this failure, and how to ensure it doesn't happen again.

@wilbertom

This comment has been minimized.

Copy link
Member

commented Jul 1, 2014

I made some progress on this but it's really really slow. Tonight I'll reread it to make sure it's right and work on making it faster.

https://github.com/wilbertom/cookcountyjail/tree/backfill

@bepetersn do we have a traceback of the error?

@wilbertom

This comment has been minimized.

Copy link
Member

commented Jul 1, 2014

Never mind I know how to make it faster. So much for sleeping lol

@wilbertom

This comment has been minimized.

Copy link
Member

commented Jul 11, 2014

This is harder than I thought. It's taking too much time to dump. I'm starting a new branch.

https://github.com/wilbertom/cookcountyjail/blob/backfill-b/countyapi/management/commands/dump_to_csv.py

@wilbertom

This comment has been minimized.

Copy link
Member

commented Jul 11, 2014

I'll make a writeup on the google groups to get some feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.