Design ETL pipeline

[dcwalk]

From @ambergman on February 11, 2017 22:44

What do we need from the incoming data?

Copied from original issue: edgi-govdata-archiving/filtration#1

[dcwalk]

From @DSLituiev on February 11, 2017 22:52

We need to build a pipeline that takes difference set from page freezer so that further we can produce categorization of text chunks.

[dcwalk]

From @aschn on February 11, 2017 23:59

step 0: incoming data format
step 1: naive deduplication
step 2: create separate pipeline from image changes
step 3: feature extraction
step 4: clustering
step 5: score cluster quality
step 6: rank clusters on importance
step 7: output

[janakrajchadha]

@dcwalk I think the wrong issues have been tagged with the steps here.
For example : - Naive deduplicaton is issue #14 and feature extraction is issue #12.

[dcwalk]

@janakrajchadha you are right, thanks for the catch. We merged our issues over to this repo and some of the old auto-lines are in the above. I'm going to remove the issue #'s for now

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.