Sqlite example #51

derekeder · 2012-09-20T20:46:10Z

create an example usage of dedupe with a sqlite database using campaign contribution data

create tables and loads in raw data
create pattern for sampling, training and creating blocking rules
run rules and dedupe functions to create table of unique donors

relates to #49

… on sqlite needs

…ordDistances

…e overhead

+ sampling using multiple small buckets + passing record_distances back from active learning to save an expensive call + changed predicateCoverage so empty predicates don't match

+ added cosineSimilarity, createCanopies and coverage functions + added basic test cases (not in testing framework) + wired up TF-IDF to trainBlocking in blocking.py + BUG: TF-IDF not covering blocked data the way we expect + TODO: investigate TF-IDF blocking logic

…ment tfidf blocking on original dataset

…s, continued working on blocking sqlite data

derekeder · 2012-12-05T21:27:58Z

Done:

implemented non-compound tf-idf predicates

To do:

compound predicates
use learned tf-idf thresholds to define blocking
read/write settings is broken with addition of tf-idf

derekeder · 2012-12-05T22:33:01Z

Todo:

compound predicates
test against sqlite_example

…e can cut the memormy usage in half

…ersII function

… unique

…on of 70000 rows :

write entity_map table in illinois_contributions db

…@andreasvc

…@andreasvc

…of blocks of records

derekeder · 2013-01-28T21:12:15Z

OMG this feature branch is ready to be merged in to master!

Accomplished all above stated goals using a sqlite database with IL campaign contributions

Sqlite example

derekeder and others added 15 commits September 20, 2012 15:39

good progress on sqlite_example, updates to blocking and dedupe based…

bddb3b9

… on sqlite needs

changed names of saved settings files to match example name

e307378

sqlite_example runs.... slowly

ddc42be

refactored recordDistances and removed recordDistancesII

e1b1197

cleanup and refactoring of addTrainingData and recordDistances

052ef79

profiling and improving performance of calculateDistance and buildRec…

98c59f2

…ordDistances

fixed scorePairs to handle field order

fe7bd62

more optimizing of record distance calculations

18bb463

optimized buildRecordDistances and removed calculateDistance to reduc…

d4cae47

…e overhead

commented out interaction-terms from canonical example

75d726c

optimized blocking

6a6c792

+ sampling using multiple small buckets + passing record_distances back from active learning to save an expensive call + changed predicateCoverage so empty predicates don't match

working on blocking

0235f2f

fast semi-supervised distinct duplicates, closes #55

f9cf23b

semi-supervised duplicates now samples without replacement

3e47689

speed optimizations

52a0b1a

derekeder mentioned this pull request Oct 24, 2012

Better Blocking #52

Closed

derekeder and others added 7 commits October 24, 2012 17:34

started on TF-IDF blocking predicate

c28caf4

some minor performance tweaks for blocking

bacf952

got learn blocking working with tfidf

aa5d954

cleaned up coverage function

6fb9a40

worked through sqlite_example, saving tfidf predicates, need to imple…

418264d

…ment tfidf blocking on original dataset

created Blocker class for abstracting away tfidf and simple predicate…

20b5d7c

…s, continued working on blocking sqlite data

derekeder added 4 commits December 5, 2012 15:29

implemented non-compound tf-idf predicates

772e9d8

canonical test using tf-idf predicates for blocking

dc7824d

handling reading/writing settings with tfidf predicate types

276f1c6

using learned thresholds of tfidf predicates

ad0c4a0

removed print statements from blocking

f67074a

fgregg and others added 26 commits January 15, 2013 07:14

optimizing condensed distances

7a2d194

fastcluster 1.8 has MIT compatible license

8e76186

by not maintianing the input when we perform the clustring linkage, w…

7710da5

…e can cut the memormy usage in half

fixed canonical test after sqlite refactoring, removed duplicateClust…

a136744

…ersII function

removed reference to 'values' in trainig data (not needed)

c7b5807

continuing memory profiling, added numpy_test_memory file

e35a4b2

tracking down memory leak in affinegap

17bd1a7

possible solution to memory leak

8e81747

tracked havey memory usage to the set we were using to make canidates…

5e8cb78

… unique

caching is not worth overhead in speed or memory

507531c

fixes bug in distance of condensed distance

1ba40ec

full data FTW

9f0ffc6

only clustering connected components allows completion of deduplicati…

91de0d3

…on of 70000 rows :

added requirements

500434f

subclassed sqlite connection to return lower cased and striped rows

f57b84f

write entity_map table in illinois_contributions db

significant optimization of affinegap, now using pure C arrays, h/t to …

b877069

…@andreasvc

typo

f021ecd

factored calculations in initialization of C arrays

f7c0eb7

moved lower casing of strings into sql query saves about 30% overhead

44bca6e

factor out get donor select statements

e38456d

updated predicate assertions for near integers and n-grams

ef3ab4a

follow http://wiki.cython.org/DynamicMemoryAllocation in affinegap

a797620

fixed tests for Blocking

ff474ac

duplicateClusters builds list of candidates from generator

7d972fd

removed merge blocks, refactor duplicate clusters to take a iterable …

bf73c42

…of blocks of records

skipping invertIndex when no TFIDF predicates are used, resolves #63

51eb49f

derekeder added a commit that referenced this pull request Jan 28, 2013

Merge pull request #51 from open-city/sqlite_example

4381ee5

Sqlite example

derekeder merged commit 4381ee5 into master Jan 28, 2013

derekeder mentioned this pull request Jan 28, 2013

Create sqlite database example #49

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sqlite example #51

Sqlite example #51

derekeder commented Sep 20, 2012

derekeder commented Dec 5, 2012

derekeder commented Dec 5, 2012

derekeder commented Jan 28, 2013

Sqlite example #51

Sqlite example #51

Conversation

derekeder commented Sep 20, 2012

derekeder commented Dec 5, 2012

derekeder commented Dec 5, 2012

derekeder commented Jan 28, 2013