South Africa Data - Sources

[RocketD0g]

Listed as a priority because we currently do not have any data from the entire continent of Africa.

Data: http://live.rbcaa.org.za/Default.htm
Coordinates: Click on station/station information for coordinates
('trs' in pollutant table is 'total reduced sulfur' - we won't want it

[olafveerman]

Quick note, the above site makes heavy use of iframes. The table with the data can be found on this url: http://live.rbcaa.org.za/DynamicTable.aspx?G_ID=1

[RocketD0g]

Another source recently mentioned to us: http://www.saaqis.org.za/RequestData.aspx

[sourabhtk37]

http://live.rbcaa.org.za/Default.htm
You can find the data here.

[dolugen]

Another source recently mentioned to us: http://www.saaqis.org.za/RequestData.aspx

This is a slow one. You submit your request and have to wait. Not a good channel for OpenAQ.

It aggregates data from all other South African data providers. It would be nice if we had a direct access to it. Even aqicn doesn't have much coverage of the country, let alone the continent.

http://live.rbcaa.org.za/Default.htm

This one only covers the Richards Bay town.

[RocketD0g]

Given @dolugen's comment, above, this one just focuses on the Richard's Bay data source (http://live.rbcaa.org.za/Default.htm) and organizes the data source for dev review.

0. These data are shared in a very similar way to the Israeli data, shared in this comment: Data from Israel #146 (comment)

1. Go to http://live.rbcaa.org.za/Default.htm

2. Click: Dynamic Table->Richard's Bay

3. Follow similar procedures as Number 5 in Data from Israel #146 (comment)

• Suggest 15 minute or higher averaged data from stations

4. Station name and coordinates are accessible by clicking on station name.

[dolugen]

Data addresses

• The /MenuSite.aspx file has a variable (search for var o or DynamicTable) that contains data for the Dynamic Table menu items (as well as the Envista system build version).
• Read the list of station IDs from page /DynamicTable.aspx?G_ID=X
• Read station coordinates from /StationDetails.aspx?ST_ID=X
• Read the AQ data from /StationReportFast.aspx?ST_ID=X

Underlying system

Looked around a bit, and found that the Israeli and South African sources both use the Envista Web system. Maybe they could share a single adapter that handles Envista sources.

Although they share similar "build numbers", their advertised version number differs:

Country	Build number in MenuSite.aspx	Version number in About.aspx
IL	2.0.20091.207	6.4.14
ZA	2.0.20063.108	4.0.8.9

(Turkey's system has the same build number as Israel's)

I think @maxgrossman (he just started working on the Israeli adapter) and I should attempt to make a unified Envista adapter. We'll see.

Edit: The build numbers are from a web component that's used in the system.

[dolugen]

I spent some excess amount of time debugging an export issue in the adapter, which at the end was an issue with the export page. In short: the export functionality was throwing the same files for different station exports.

This is worked around by giving each station requests their own cookie jar.

There are a few things that should be improved, but overall, the adapter is in working order.

Currently, a URLencoded XML is hardcoded in the adapter. The following is a note for dealing with that (moved it out of adapter to keep it cleaner):

const getLstMonitors = function () {
  /*
    TODO
    An XML tree that describes the checkedness of
    the monitors is URLencoded and passed as a string parameter
    in the form.

    It looks like below when decoded, going up to 20 something monitors:
    <WebTree><Nodes><lstMonitors_1 Checked="true"></lstMonitors_1><lstMonitors_2 Checked="true"></lstMonitors_2><lstMonitors_3 Checked="false"></lstMonitors_3></Nodes></WebTree>

    It cannot be queried with cheerio, as it's generated
    on the client. But as structure of the tree is known,
    it can be generated using the list of monitors and their tag ID.
  */
};

[dolugen]

With the report form properly customized, the adapter is dealing with smaller export file, and some useless stations are skipped early. This takes the adapter's running time from 4min down to about 1min.

I'm done with this source, but I'll be looking at adapting the adapter for other countries (e.g. Israel).