Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100755 61 lines (39 sloc) 1.395 kb
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
1 #!/usr/bin/env python
2
3 import sys
4
5 from csvkit.unicsv import UnicodeCSVReader
70eb7a3 @JoeGermuska push mongo collections back into dataprocessing/utils
JoeGermuska authored
6 from pymongo import objectid
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
7
8 import config
9 import utils
10
11 if len(sys.argv) < 2:
12 sys.exit('You must provide the filename of a CSV as an argument to this script.')
13
14 FILENAME = sys.argv[1]
15
16 YEAR = '2010'
17
70eb7a3 @JoeGermuska push mongo collections back into dataprocessing/utils
JoeGermuska authored
18 collection = utils.get_geography_collection()
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
19
20 with open(FILENAME) as f:
fe3b6d0 Revert "Use latin-1 encoding."
cgroskopf authored
21 rows = UnicodeCSVReader(f)
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
22 headers = rows.next()
23
47b6737 Load all census files.
cgroskopf authored
24 updates = 0
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
25 row_count = 0
26
27 for row in rows:
28 row_count += 1
29 row_dict = dict(zip(headers, row))
30
31 xref = utils.xref_from_row_dict(row_dict)
32
47b6737 Load all census files.
cgroskopf authored
33 geography = utils.find_geography_by_xref(collection, xref, fields=['data'])
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
34 if not geography:
35 continue
36
37 if YEAR not in geography['data']:
38 geography['data'][YEAR] = {}
39
40 tables = {}
41
42 for k, v in row_dict.items():
86c5800 SF1 labels mapped to SF data.
cgroskopf authored
43 # Format table names to match labels
bf42e3e Implemented SF field mapping.
cgroskopf authored
44 t = utils.parse_table_from_key(k)
640e9ab Tons of minor fixes to crosswalk and delta computations. New tests.
cgroskopf authored
45
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
46 if t not in tables:
47 tables[t] = {}
48
47b6737 Load all census files.
cgroskopf authored
49 tables[t][k] = v
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
50
51 for k, v in tables.items():
52 geography['data'][YEAR][k] = v
53
65d1949 @onyxfish Added safe checks to all database writes. Closes #46.
onyxfish authored
54 collection.update({ '_id': objectid.ObjectId(geography['_id']) }, { '$set': { 'data': geography['data'] } }, safe=True)
47b6737 Load all census files.
cgroskopf authored
55 updates += 1
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
56
97a5fb5 @JoeGermuska minor tweaks to make console spew slightly more informative
JoeGermuska authored
57 print "File: %s" % FILENAME
58 print ' Row count: %i' % row_count
59 print ' Updated: %i' % updates
021477f Parametized sf loader, added FAKE option for using 2000 data as 2010.
cgroskopf authored
60
Something went wrong with that request. Please try again.