XML data sources

evasmolders edited this page Dec 2, 2016 · 11 revisions

##Introduction

For advanced home control like energy and climate management, the data on usage, production, and weather conditions are important. There are many free public wheather sources which deliver well structured hierarchical XML data. Also dedicated personal weather stations and other electronic equipment often expose their data in XML format. In general extraction of data from an XML source is simple. However weather forecast data, due to it's repetitive nature, is a bit harder to tackle. A basic understanding of the OpenRemote HTTP protocol is assumed. This protocol a.o. supports XPATH, a query language for XML data

###Selection

The most simple form of XML data is shown in the Solar Energy example below. The data is held within elements such as for example gauge_power or energy_today.

Solar Energy:

XML Data Sources - Solar Energy

Simple Weather:

XML Data Sources - Simple Weather

Forecast Weather:

XML Data Sources - Forecast Weather

The HTTP protocol supports the XPATH filter that allows to make certain selections from an XML source. So to read the actual energy sofar produced for the day use as XPATH expression:

//response/energy_today

XML not only allows to represent data in elements. This solar example can be transformed into a form where the data are stored in attributes:

<response gauge_power="423" gauge_temp="35.9" energy_today="1.600"/>

The Xpath expression to be used than takes the following form, where the @ stands for selection of an attribute)

//response/@energy_today

###Function

The selections described above are returned as strings. To display the daily energy production level on a dial, a range sensor is needed, which in OpenRemote requires an integer input. Next to element and attribute selection mechanisms, there is a wealth of functions to further manipulate the selection. With the expression below, a string value indicating kWh production is transformed into an integer Wh value suitable for a range sensor.

round((concat(//response/energy_today,""))*1000)

Note: It is as yet unclear why the concatenation with an empty string is needed to get this working in OpenRemote ###Examples ####Simple Weather Some of the weather sites give rather verbose observation time references. A few of them adhere to the RFC 822 representation standard dated 1982. This standard has a fixed length format, so it is safe to use the substring function to get the the required selections. For example an hour:minute:second selection from the original sting Fri, 17 May 2013 12:48:02 +0200

substring(//response/current_observation/observation_time_rfc822,18,8)

####Forecast weather Weather forecast data is often presented in multiple blocks that are valid for a fixed period of the day. Such blocks are marked with the repeating tag. To retrieve the wind speed in the second time frame use the following xpath expression, where the sequence number is enclosed in []:

//weatherdata/forecast/tabular/time[2]/windSpeed/@mps

##Miscellaneous

"JSON Data packaged in XML" This example comes from MYTHTV were it looks as if their old XML representation using attributes has been simplified to the usage of elements only. That new format has some JSON flavour:

<?xml version="1.0" encoding="UTF-8"?>

<FrontendStatus xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" serializerVersion="1.1">
<State>
<String>
<Key>currentlocation</Key>
<Value>library.xml</Value>`
</String>
<String>
<Key>menutheme</Key>
<Value>defaultmenu</Value>
</String>
<String>
<Key>state</Key>
<Value>idle</Value>
</String>
</State>
<ChapterTimes/>
<SubtitleTracks/>
<AudioTracks/>
</FrontendStatus>

You see a repetition of the element <Key> which cam in place of the attribute key. To extract data from this kind of representation do use the following example expression:

//FrontendStatus/State/String[Key='state']/Value/text()

####Yahoo RSS based Weather Services

Yahoo weather services (https://weather.yahooapis.com/forecastrss?w=24067405&u=c) does deliver a "styled" output. With -U in your browser you can see the tagged output:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
		<rss version="2.0" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
			<channel>
		
<title>Yahoo! Weather - Thorold, CA</title>
<link>http://us.rd.yahoo.com/dailynews/rss/weather/Thorold__CA/*http://weather.yahoo.com/forecast/CAXX0497_c.html</link>
<description>Yahoo! Weather for Thorold, CA</description>
<language>en-us</language>
<lastBuildDate>Wed, 25 Mar 2015 4:59 am EDT</lastBuildDate>
<ttl>60</ttl>
<yweather:location city="Thorold" region="ON"   country="Canada"/>
<yweather:units temperature="C" distance="km" pressure="mb" speed="km/h"/>
<yweather:wind chill="-5"   direction="140"   speed="3.22" />
<yweather:atmosphere humidity="81"  visibility=""  pressure="1022.4"  rising="0" />
<yweather:astronomy sunrise="7:09 am"   sunset="7:34 pm"/>
<image>
<title>Yahoo! Weather</title>
<width>142</width>
<height>18</height>
<link>http://weather.yahoo.com</link>
<url>http://l.yimg.com/a/i/brand/purplelogo//uh/us/news-wea.gif</url>
</image>
<item>
<title>Conditions for Thorold, CA at 4:59 am EDT</title>
<geo:lat>43.11</geo:lat>
<geo:long>-79.23</geo:long>
<link>http://us.rd.yahoo.com/dailynews/rss/weather/Thorold__CA/*http://weather.yahoo.com/forecast/CAXX0497_c.html</link>
<pubDate>Wed, 25 Mar 2015 4:59 am EDT</pubDate>
<yweather:condition  text="Unknown"  code="3200"  temp="-5"  date="Wed, 25 Mar 2015 4:59 am EDT" />
<description><![CDATA[
<img src="http://l.yimg.com/a/i/us/we/52/3200.gif"/><br />
<b>Current Conditions:</b><br />
Unknown, -5 C<BR />
<BR /><b>Forecast:</b><BR />
Wed - Rain/Wind. High: 9 Low: 2<br />
Thu - Rain. High: 4 Low: -2<br />
Fri - AM Snow Showers. High: -1 Low: -8<br />
Sat - Partly Cloudy. High: -2 Low: -8<br />
Sun - Partly Cloudy. High: 4 Low: 1<br />
<br />
<a href="http://us.rd.yahoo.com/dailynews/rss/weather/Thorold__CA/*http://weather.yahoo.com/forecast/CAXX0497_c.html">Full Forecast at Yahoo! Weather</a><BR/><BR/>
(provided by <a href="http://www.weather.com" >The Weather Channel</a>)<br/>
]]></description>
<yweather:forecast day="Wed" date="25 Mar 2015" low="2" high="9" text="Rain/Wind" code="12" />
<yweather:forecast day="Thu" date="26 Mar 2015" low="-2" high="4" text="Rain" code="12" />
<yweather:forecast day="Fri" date="27 Mar 2015" low="-8" high="-1" text="AM Snow Showers" code="14" />
<yweather:forecast day="Sat" date="28 Mar 2015" low="-8" high="-2" text="Partly Cloudy" code="30" />
<yweather:forecast day="Sun" date="29 Mar 2015" low="1" high="4" text="Partly Cloudy" code="30" />
<guid isPermaLink="false">CAXX0497_2015_03_29_7_00_EDT</guid>
</item>
</channel>
</rss>

<!-- api9.weather.bf1.yahoo.com Wed Mar 25 03:03:36 PDT 2015 -->

To get the present temperature, consider the following line

<yweather:condition  text="Unknown"  code="3200"  temp="-5"  date="Wed, 25 Mar 2015 4:59 am EDT" />

In the http call specify the following in the XPath field:

 //*[local-name()='condition']/@temp

###Availability of XML data

Both XML sources used have limited availability. The solar energy production logger is powered by solar energy, and hence not available during the night. The free version of Wunderground allows you 500 calls per day, which may seem a lot, but in practice is easily consumed. At night making calls to the solar logger produces many arrors in the logs. Therefore a shell script is used to download the xml source to a new directory webbapps/private, and let the HTTP protocol read the values there via the Tomcat webserver that came with OpenRemote.

#!/bin/sh
cd /volume1/@appstore/OpenRemote/webapps/private > /dev/null
 wget -q -N http://api.wunderground.com/api/<YourKey>/conditions/q/nl/<YourLocation>.xml

This script could be called every 10 minutes by the Shell execution protocol. That solution did prove to be very unstable. Scheduling it with Crontab (Linux) proved to be solid over a couple of months now. Similar results may be expected from Scheduler under Windows. Note You can download the present contions and the forecast in one single call to wunderground:

wget -q -N http://api.wunderground.com/api/<YourKey>/conditions/forecast/q/nl/<YourLocation>.xml

The highest forecasted temperature is retrieved with the following XPath:

//response/forecast/simpleforecast/forecastdays/forecastday/high/celsius

See wunderground documentation for more combinations.

###References

W3schools offers a good XPATH tutorial with a list of all implemented functions.

weather examples http://w1.weather.gov/xml/current_obs/KCMI.xml This is an XML source. Press U, to see the source! http://api.wunderground.com/weather/api/ You can sign up here for a free account. Wunderground supports both XML and JSON formats. http://www.yr.no/place/Sweden/Västra_Götaland/Torslanda/forecast.xml

##Useful links The W3 consortium maintains a Markup validation service at: http://validator.w3.org/#validate_by_input There you can test the validity of "web documents" using either URL's, uploaded files or just pasted text. Latest W3C formal XPath specification (1999)

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.