Skip to content

Akvo ingest

Mark Westra edited this page Dec 16, 2013 · 6 revisions

####Tasks

  1. ingests zip files from s3
  2. geocodes incoming data
  3. creates and saves new instance if needed
  4. saves question-answers
  5. fires summarizers in queue

####Pipeline #####Data publishers

  1. Publish normalised data to incoming data queue. For example: a service listening to the device messages that there is data waiting on S3, a web form, changes made on the website. Each message in this queue has a type, which defines what type it is: "device-zip", "web-form", "dashboard-edit", etc. Each message that is published contains data about a single survey Instance. Depending on the type described above, different keys will be present, such as uuid, surveyedLocaleId, etc.

#####Data ingestion 2. Ingest data from incoming data queue. Depending on the need for geocoding, refer data to the geocoding queue, or to the persistence queue directly.

#####Geocode

Persistence
Summarizers

Clone this wiki locally