GOMNO: proxy external JSON and CSV ebola2014 outbreak data as OutbreakTSS API'd feeds

[JohnTigue]

There are very few JSON formatted sources of ebola outbreak data. They can be unified via the Outbreak APIs hopefully. At the very least, they must be proxied and made available via Outbreak API.

Relevant tool:Filtering JSON with pyjsonselect and jss

[JohnTigue]

#14 is related

[JohnTigue]

HOWTO read JSON wisely in node.js:
http://stackoverflow.com/questions/5726729/how-to-parse-json-using-node-js

[JohnTigue]

Really should tackle #8 first

[JohnTigue]

Early on the plan was to add this functionality to outbreak_time_series_reader but that is conflating things. So, now the plan is to have a site/service which does that messy stuff and "proxies" the non-OTSS JSON data into OTSS feeds. Also, by doing this experience/testing will be gained for outbreak_time_series_reader in node.js contexts

(Read the following for mini-TODOs)
From the early OTSS:
Nonetheless, in order to work with the existing data feeds (and confirm the Outbreak Time Series API would cover of the most important use cases), outbreak_time_series_reader can read from the pre-existing JSON feed.

outbreak_time_series_reader can currently read from a few non Outbreak API JSON feeds of Ebola2014 data that are on the Web. This was only simply to prove the reusability of the JavaScript object model and to hopefully encourage the adoption of the Outbreak Time Series APIs by the three known cases:

• Simon Johnson
• HDX
• ebolainliberia.org

Do not expect that to continue to work moving forward (read: this is deprecated and the functionality will be removed some time in 2015). For details of the above listed ebola data feeds, see [[JSON Ebola2014 Data Found on the Web]].

2014-11-20: until HDX have time series data via an API, still need to deal with keeping [[Outbreak Time Series Specification]] data for EbolaMapper fresh. See #9 and #18

outbreak_time_series_reader can read the CSV at:
https://github.com/cmrivers/ebola/

What about:
https://github.com/luiscape/hdx-datastorer-ebola-cases/blob/master/data/data.csv
That is not getting updated frequently. So, perhaps not important. That's just one guy versus the Caitlin Rivers' team of 27 updating daily.

[JohnTigue]

This is to be implemented with Apache Spark for two reasons.

1. Spark 1.1 has very nice JSON reading machinery which can detect implicit schemas in JSON JsonTable.fromRDD(sqlContext,,)
2. Currently, this isn't really a big data problem in terms of volume (but it will evolve to that) but it is in terms of variety (using IBMs 3Vs: volume, velocity, and variety). Actually this problem could involve all 3 Vs.

See:

• http://spark-summit.org/2014/talk/Easy-JSON-Data-Manipulation-in-Spark
• http://www.slideshare.net/Hadoop_Summit/building-a-unified-data-pipeline-in-apache-spark