Data processing chain

Initial thoughts on data processing:

• the two tables below would sit on the server nurtured by the API calls of independent micro services (see here)

• the algorithm processing the data would run on the main server.

Figure 1. Two tables playing a role in data processing

---

Procedure for computing a map of quality of life:

Iterate through the POIs in the db. For each POI:

1. Retrieve the associated classes (say: “health care facility” is retrieved for an hospital)

2. Apply the the behavior of the class. The behavior is simply some code contained in a method bearing the name of the behavior, such as “continuousRadiusDecrease”. The POI’s attributes are passed as parameters to the method. The code implements the logic suggested by the name of the method.

3. As a result, each geohash covered or influenced by a POI gets a score of “Quality of Life”. A geohash impacted by several POIs cumulates the scores.

Issues remaining:

[suggestion: each of these issues could take the form of a Github issue, to help track the conversation on each of them]

1. If the POI’s attributes are passed as arguments to the method applying the behavior, there should be a minimum of standardization of the attributes’ names for the method to recognize them. Solution: I hope this can be solved by maintaining a mapping of the most common attributes’ names to single denominations (a poor person’s ETL).

2. user preferences are still not taken into account. A start would be to maintain another mapping from ‘user preferences in natural language” to “Categories”. A POI’s behavior is computed only if the Class listed in its attributes has a Category corresponding to a preference expressed by the user.

c. How to store and manage data sources like: ‘average air quality index in Rhône Alpes region is 65”? *This issue corresponds to a broad category of data sources. I think it deserves a data processing chain of its own (different from the one described above), because the logic is quite different.*

→ The service retrieving the data source must return a bounding box (boundaries of the Rhône Alpes department, expressed as a set of long / lat coordinates). It passes this to a service converting this bounding box in a series of corresponding geohashes (either this service exists as a lib / api, or it needs to be created). The result of this conversion is then stored.

1. Most of data will be timeseries, at one point a date is needed.

e. Maybe a "Zoom" / "Scale" index on the data to limit the display or have different display from the same data at different map zoom scale ?

→ The zoom level of a given measurement and its display can be handled at computation time, not at data collection time.